PlantOps BI

Routing Agent Work by Model Tier, and Red-Teaming My Own Automation

I run a router that sends each task to the smallest model tier that can finish it. It started as a third-party skill that I downloaded as a zip, not something I designed. I kept its core idea and added two units of my own, and one of them, a red team, earned its place quickly.

The Board

Each unit is defined as an agent file, and one skill file holds the routing rules:

  • The smallest tier does simple jobs, and also runs a read-only lookup unit and a read-only final check.
  • The middle tier does ordinary work.
  • The top tier takes single hard decisions, and it also runs the red team.
  • A larger pinned model takes the biggest multi-part operations.

I added the last unit and the red team. The large-operations unit exists because some jobs are too big for a single decision.

The red team has Read, Grep, Glob and Bash, and deliberately no Write or Edit. A lint check enforces that, so a later edit can’t quietly give the red team the ability to fix its own findings. The rule in the skill is that the red team goes after every top-tier or large-operations job and before anything hard to undo. Findings go back to the unit that wrote the work, and the read-only check signs off last.

One practical catch: new agents only load in a new session. Inside an existing one I used a top-tier stand-in carrying the red team’s instructions.

The Routine I Pointed It At

The board pins model names, and model lineups change. So a LaunchAgent runs every other Monday, reads the public models overview page, and keeps the pins current. It uses no LLM and no API key. It rejects the page unless the names, ids and aliases agree. It auto-applies only a same-family upgrade of at most two minor steps. Everything else, such as a major jump, a family change or a family leaving the lineup, is flagged as needing my call.

The edit itself is mechanical. Only the drifted model line and the package readme change. A separate backward line-by-line verify runs, then lint, then a check that the live board is unchanged. Files are replaced atomically, with a backup restored on failure, including on SIGTERM.

What the Red Team Found

The red team’s first two jobs were this routine.

Round one returned 17 findings. One was a real code-execution hole: text from the fetched page could reach an osascript notification. A routine that reads a web page and then builds a shell-adjacent command from it has handed the page’s author a way in. I fixed it with a redesign.

Round two confirmed 11 fixes and found 14 more. All 14 were fixed, each with a regression test. I did not send a round three. The suites now stand at 32 checker tests and 33 lint tests, and the built-in self-test passed against the live page.

The routine is built without a model on purpose, and it still had the hole. No LLM was involved, and it was still unsafe, because the problem was untrusted text flowing into something that executes.

Known Limits

I wrote these into the readme instead of pretending they don’t exist: a millisecond-scale race window, no way to acknowledge a review item, and breakage if a macOS update changes the system python3.

What I Would Copy

Give the red team a different tool set than the author, and make that difference something a check enforces. Run it on your own scheduled automation before you trust it, especially anything that reads from the internet and then acts on what it read. I would not have found a 17-item list by rereading my own code.