AI agent autonomy levels, and how we work

Our setup, as of November 2026

This is what we use today. It changes, so we date it.

• Claude Code writes most of our code.
• By default, a model from another family reviews that code. Today that means Codex on Claude-written work, with GLM 5.3 (run through melious) as the standing second reviewer.
• Some changes are written by Codex. Then a model from another family reviews those too.
• Kimi K3.1 (run through ollama) reviews only when Daniel asks, once per change.

The aim is simple: the model that wrote a change does not grade its own work. See who checks the AI for how a review runs and what it found.

Before we build

We start with a plan, not with code. Building starts after these steps:

1. A written plan. What we build, why, how, and how we will test it.
2. Architecture decisions, for anything new that connects several parts. When a new piece has to work with a backend, a new sign-in flow and AI access, we decide how the parts fit together before we build. We write each decision down with its reasons, and the plan review challenges it. The fit between parts is usually the costliest thing to change later. A small change inside one part needs no such step.
3. A plan review. Reviewers read the plan, round after round, until none of them finds a blocking problem.
4. A red team, for risky work. Risky means user data, logins or permissions, money, deletion, a schema change or our servers. One reviewer is told to attack the plan, not check it: how it fails, how it could be misused, how it could lose data, and whether we can roll it back. Example: the plan for a background job that caches user data. The red team found 3 bugs and 1 risk, and we fixed all four before building.
5. Requirements, sometimes. When a change is more complicated, we write down what it must do as concrete scenarios: given this, when that happens, then this. Anyone can check a scenario against the product. A small fix does not need them.
6. A test plan, with CI and automated testing. We automate tests and run them in CI as far as we can. On real iOS and Android devices we still test by hand.
7. Evals, for tasks where an AI agent does the work. We write them before we build: a set of tasks, a way to grade the answers and the result we accept. See how we write evals.

Then building starts.

In plain English

• Red team: a reviewer who plays the attacker and tries to break the plan before anyone builds it.
• CI (continuous integration): a computer automatically runs our automated checks on changes to the code, so a problem shows up early.
• Automated testing: small programs that check the product does what it should, so nobody has to click through everything by hand each time.
• Manual testing on real devices: some problems only show up on a real phone.
• Eval: a repeatable test of what an AI model produces.

In the loop and on the loop

You may have heard "human in the loop". There are two ways to be in it.

• In the loop: a person approves each step.
• On the loop: a person sets the goal and the checks, and steps in by exception.

At L2 you are in the loop. At L3 and L4 you are on it: you own the change or audit the result, and you step in where the risk calls for it.

We are not alone in this split. Anthropic's own Code Review for Claude Code (opens in a new tab) ranks bugs by severity and does not approve pull requests. That stays a human decision.