Understanding and Engineering Harness and Loops for Agentic Coding

Mid-July, I decided to build a small pet product entirely with a coding agent. The product was half the point. The other half was building the harness: the rules, skills, and gates that make the agent's output trustworthy - and the loops they run in. That means setting a system to force the agent build, maintain, and evolve the product in the long run without quietly wrecking it, while I keep enough understanding to stay in control at the key checkpoints.

What's in the article:

• A glossary - what all these terms actually mean
• A theory of AI-coding evolution, which helps make sense of what happens in the article and why
• A brief overview of the tools used
• How the agent's work is structured
• What the agent regularly invokes while working
• The elements of the environment the agent operates in that saves it from slopping
• How to improve the AI-coding rig itself
• What product development looks like when AI sweats instead of you
• Conclusions from the experiment, and what's next

I recommend keeping two repos open next to the article: the skills repo and the project repo. The article tells what and why - there you can see how exactly it looks.

Wait, should I even build my own harness?🔗

Depends on what you're after.

If you just want to try agentic coding, don't start with somebody else's pile of md files — start with what ships in the box. Claude Code alone has goal (the simplest loop there is), plan mode and lightweight sandboxes built in, and that covers more than people assume.

If you have an applied problem and want it solved, take a ready-made harness: from DeepSeek (docs and video), Mentiora (article) and disler (video).

If you want to own the system rather than just use it, build your own. Not because the ready-made ones are bad, but because the harness is now a bigger part of owning a system than the code is.

My goals were:

• to make sense of the new buzzwords AI-bros brought us;
• to try building toy model system;
• to decompose it and reflect on it before building an actual one.

This isn't an implementation of agentic engineering within a commercial development team. So there nearly won’t be any metrics or effectiveness analysis.

That said, off we go!

Glossary🔗

They nearly invented a separate AI-related language. Let's look at some important terms, anti-hype:

• Agent — a tool like Claude Code, Codex, Cursor, or something else built or plugged into your IDE. A wrapper above a large language model (LLM) with some additional features. I prefer Claude Code and might mention it, but mostly this means any agent. There might be some specific features; don’t mind them until mentioned.
• Prompt — an input message to the agent.
• Skill — a markdown file with a reusable prompt the agent loads to context on demand. Documents a part of a process. The agent picks one up itself when the task calls for it — Claude Code just prints /skill-name in the console when it does. You can call the same skill under same name as a command yourself. Might be yours or third-party.
• Guards or Gates or Guardrails — automated checks, AI-driven or not, that stop the agent at the moment of a wrong move: permission rules, hooks that parse commands before they run, CI gates, tests.
• CLAUDE.md or AGENTS.md — obviously markdown files the agent reads at the session startup with some instructions: how to write code in the project, what's forbidden, and what to do when caught making a mistake. At some point become split into several files and serve as a navigation catalogue among those.
• Workflow — a software development lifecycle (SDLC) the agent follows. Basically, an abstract diagram with all the connections, triggers and loops, where an element is a work step that produces some artifacts. Not the same as the harness: the workflow is the diagram, the harness is the files and checks that keep the agent on it. You draw the first and enforce it with the second.
• Harness — markdown files and scripts that force the agent to follow the workflow and run checks to prove it didn't wreck anything, kept the project readable for a human, and actually did what you asked. Basically skills, gates, settings and instructions from markdowns.
• Spec-driven development (SDD) — a framework, when the agent first creates a markdown file with its plan to execute something. A human reviews it — at least, is supposed to. The agent executes.

Theory🔗

Meaning of “I code with AI” varies:

• Vibe coding is prompting in a loop: ask, get code, poke, prompt a fix, tweak to fit codebase. You build a product with an agent, using it as a tool in a classical SDLC.
• Agentic coding is when the agent builds the product in a specific SDLC you build. You separate stages, invest upfront in research and specs, set up checks that tell the agent when it's wrong, and review at the level of systems and contracts. You build a system which builds the product — and occasionally you review its outputs.

The AI-adoption frontier has significantly moved between these stages:

1. First everyone competed at prompt engineering — who writes the smartest prompt. That didn't disappear, it rather moved into the skills section.
2. Then everyone competed at context engineering — filling the agent’s LLM context window with exactly what the task needs and nothing else. With 1M-token windows the stuffing problem faded. Though you still need to get rid of duplicates and contradictions.
3. Next the competition was around harness engineering: who builds the best rig around the agent — constitutions, permissions, gates, review skills, deterministic checks. Its unit of design is a rule or a gate. Its failure mode is a bad change slipping through. They engineer the environment the agent works in, while the agents or models might be swapped. Most of this article lives here.
4. Now, they compete in loop engineering: who builds the most autonomous process with the fewest, best-placed human checkpoints. And who minds a plan B: making sure every artifact the process produces (specs, docs, tests, plans) stays useful to a human, in case the specific model flakes out, can't handle the product domain, or simply gets too expensive. They optimise the time and resources — design triggers, continuation conditions, stall detectors, budget ceilings, run journals.

Those shifts also reframe the job: a software engineer who moves to agentic coding stops being an operator of an agent and becomes the person who designs and uses an agentic SDLC: intake, planning, implementation, verification, release, the feedback loops between them — and the excellence at each of the steps.

Maturity levels🔗

Aleksei Litvinau proposed a nine-level “capability evolution” ladder for AI interaction:

1. writing code by hand
2. free chats
3. paid chats. The problem here and above: low trust in the output
4. CLI agents with manual confirmation of every step aka hamster-tapping
5. Fully autonomous repo access inside a sandbox
6. Parallel agents in different terminals for different tasks
7. Multi-agent system designed like an org: roles, hierarchy, shared memory
8. Orchestration — systems managing teams of agents
9. Ecosystem — governed by moral principles and semantic invariants

This project was built at L4. I wanted to watch the agent work — while constantly pushing it toward autonomy within that level: the goal was half-hour runs where it doesn't need me at all.

L5–L8 are where this goes next. L9 is a corporate fairytale: I doubt anyone actually operates there, as opposed to drawing slides about it.

Verification debt and evidence packages🔗

The deepest shift agents brought to SDLC: coding became cheap and fast, while understanding code and assuring quality became the bottleneck. It forces us into situations where an engineer puts an LGTM on a 3k-line PR without understanding it first. Though it doesn’t save time in the long run — it rather creates verification debt.

The terminal state of verification debt is cognitive surrender — the moment when a system is so heavily vibe-coded that nobody in the company understands how it works. Understanding can’t be delegated. At least if we’re not gaming the meaning of understanding. You cannot hand the agent responsibility for the actual reasoning.

The answer AI adepts come up with is not to read every line harder. It's evidence packages: instead of a wall of diff, the reviewer gets a proof on a higher level of abstraction. Those are the artefacts proving the system's behaviour changed exactly as planned: before/after screenshots, logs, deterministic checks, tests that cite the requirement they close.

A five-step roadmap from manual approval to systematic verification:

1. baseline — start measuring time-to-accept and rework share;
2. definition of ready/done (DoD or DoR) for AI tasks;
3. spec-driven development;
4. blast-radius classification of changes;
5. evidence packages.

Occasionally, my harness complies with this ladder: e.g., Easy Approach to Requirements Syntax (EARS) acceptance criteria are the DoR, OpenSpec implements SDD, /triage skill is a blast-radius classifier, and spec-coverage.ts plus the mutation floor are evidence that the tests mean something.

Tools🔗

If working with modern tooling and progressive engineers sounds like your kind of fun — we're hiring at Mojam. Check the vacancies and apply.

Every pick follows the same principle: the fewer moving parts, the fewer ways for the agent to screw up. So, the stack, briefly:

• Anthropic tools — the default choice. Claude Code is the most feature-rich agent around, the models are so-called state of the art, and Claude Design covered the UI drafting. An IDE turned out unnecessary: diffs read fine on GitHub when I want understanding, and on a sandboxed server running an IDE is far more hassle than running Code CLI.
• Bun — despite Anthropic scaring everyone with that rewrite-it-in-Rust attempt they actually did it while I was writing the article 🗿, I was working on 1.3.14. It is still a superb runtime: a test runner, a package manager, native TypeScript execution and a YAML parser out of the box. Every built-in is a dependency the agent doesn't get to drag in.
• TypeScript — strict types are one more constraint on the model, cutting the odds it silently breaks something. Mind that Bun executes TS by stripping types, not checking them — so tsc --noEmit or bunx tsgo --noEmit runs as a separate gate.
• Biome — a fast extensible linter-formatter instead of the classic pair of ESLint and Prettier. Might matter for CI minutes the agent burns on every push and general exec time satisfaction.
• Playwright — seems a good choice because it forces the agent to write good markup. And it also fits great into JS stack. Moreover, there’s @axe-core/playwright scanning accessibility on every page the smoke test visits. In AI coding, e2e tests play a crucial role: they prove that changes didn't break real user flows. The agent can run the suite while building UI or finishing changes — to verify itself. It also has an MCP server — useful when the agent has to poke at a site it has no code for, not my today’s case though.
• Stryker — mutation testing: it flips tiny pieces of the code and checks whether any test notices. Line coverage proves a test ran the code. A killed mutant proves the test would catch a bug. Provides a metric of test quality — and the floor it sets is agent-proof: you can't satisfy it with tests that assert nothing.
• simple-git-hooks — a thin wiring for the pre-commit/pre-push battery: a few lines in package.json, no framework, no config language.

Now, about an AI code review tool in a bit more detail.

CodeRabbit and alternatives🔗

CodeRabbit is a ready tool for AI code reviews. I rolled it out at work for all engineering teams and liked the results, so it earned its slot here too.

It could be replaced by a separate skill or a self-made review agent — but the point of this experiment was the harness, not reinventing a reviewer, so I took the ready-made tool to keep my focus.

Despite the tool’s known downsides:

• the CLI agent run takes 5–7 minutes,
• the GitHub CI run sometimes takes 15 minutes,
• the basic version costs $30, although for agent coding you'll have to pay $60 for extended limits

… imho it’s still worth it because:

• it actually works out-of-the-box even on default settings,
• is customisable enough via .coderabbit.yaml,
• makes meaningful findings

… and after all saves engineer’s time compared to a self-made code review agent. Especially when your company funds it for a private self-hosted GitLab repo.

For a public repo though it might trigger some cringe moments. I.e. my security analytics tool trial ended. And while support was reviewing my case, I got spammed with an alert saying it and offering me to go to the project admin, me as well, to activate that feature. Sometimes even twice in a single run 🤡

I also had to activate usage billing, as I've constantly ran into limits, even on highest Pro Plus plan. So for public repos or pet projects made with agentic coding you might want to consider tools with more friendly terms for individuals. I've heard about several, but haven't yet tried them: MergeStorm, Greptile, Qodo, Sourcery - would appreciate sharing your experience.

Workflow🔗

What’s worth commenting on:

• There are two loops here. First — is the change loop. It has a fork either it’s an explore-change or apply-change. It entirely covers the agent’s SLDC. Second is the pull request sub-loop inside the main one’s. It starts with the first push attempt and finishes when the PR is merged. Regretfully, CodeRabbit might require a few iterations to catch all the findings — and that’s something their developers warn about.
• Orchestration level — is something I took my part at. I run the commands there or do manual actions. Most of the optimisations to raise the system’s autonomy level lies here.

My advice is to draw such a block diagram beforehand: which gates are planned, what feeds what, where the human sits. Make agent read it: either to follow it strictly and flag deviations, or to update when the system changes. I didn’t, so I had to keep this picture in my mind while the agent kept reproducing it as a consequence of the loaded context. Not so convenient and makes the agent stick to it more randomly.

Most of the work in this workflow is done by skills. And the workflow exists in a kinda environment: the settings of the project. Skills, settings and gates together are the harness — the workflow is the shape it enforces. Let’s dig both deeper.

Skills🔗

For now, my skills live in a separate repo and get symlinked into projects with a small link.sh — one place to edit, every project picks up changes. A clone of the product repo gets them with one command: ./link.sh all <path-to-d2ass>.

Some of the skills are vendored. When you add them, the installer produces a skills-lock.json with source and content hash — free provenance. The policy fell out naturally: skills in the lock are vendored — re-download to update, never edit. Everything else is owned and evolves through fix & capture process described below. There's a skill-provenance.test.ts in the product repo enforcing it.

Three skills I forked from the public unlearndev/skills repo. All the other skills I wrote myself. I jumped between system improvements and development of my pet project, a few iterations per switch. Each session I ran skills and evaluated results. If something was poor, I prompted the fix (“group by feature, not by file”, “drop the paths, keep just the tiers”) and then asked the same agent in another directory to improve skills accordingly.

/triage went through four such rounds before I stopped touching it for some time, while I updated /coderabbit dozens of times, and keep doing it once per session.

Session commands🔗

/ponytail:ponytail is the first skill of every session — the ponytail plugin's kickoff, which pulls in the base settings and current plans by itself. I enjoyed the setup and can subjectively confirm the conclusions from the JetBrains' article:

Ponytail works. Across 80 paired tasks, it cut the typical bill by 10.3% and reduced code written by 15%, with no quality difference we could detect. It is the first tool in this series that clearly saved money. If you install it and forget about it, you should be modestly better off.

I also enjoy the resulting code and not verbose outputs, while they are much more readable compared to Caveman.

/session-wrapup is the last skill of every session. It checks and analyses the agent’s confidence in the outputs, and writes down a lessons-learned sweep, workflow state for the next kickoff, and an optional save-point doc. The lessons are suggested from repetitive mistakes made during skill calls and general session analysis. So the skill suggests skill, settings, and workflow corrections, as well as some domain or product-related memories.

Change commands🔗

Then comes a bunch of skills from OpenSpec. They sustain the main change loop both for product features and in-repo harness improvements.

/opsx:explore — takes raw material: a file from the spec inbox, a half-formed idea, a temp.md of audit findings — and grills it. It checks for viability and edge cases, and asks open questions one at a time. Produces in-context understanding, not files. If the content survives the grilling, it’s turned into a proposal.

/opsx:propose — describes an exact change in a pile of files: what and why, with EARS acceptance criteria and a non-goals section, design requirements, and a tasks checklist where each task cites the criterion it closes.

/opsx:apply — executes the tasks from a clean context, checking them off as it goes.

/opsx:archive — after the merge, folds the change into the living spec. The project accumulates its decision history, and the next propose starts on top of it instead of from zero.

Local review gate sequence🔗

/triage — reads a branch diff and groups it by feature area, not by file, into High/Medium/Low risk tiers. A map of where to spend review attention, explicitly forbidden from reviewing anything itself.

/zombies — turns a feature description or a diff into test ideas via the ZOMBIES heuristic: Zero, One, Many, Boundaries, Interface, Exceptions, Simple scenarios. Runs twice per feature: on the proposal text before any code exists, and in diff mode after implementation, cross-referencing existing tests.

/warm — evaluates every dependency a branch pulls in via another heuristic: Worth it, Alive, Right-sized, Maintained securely. Skipped when there are no new dependencies.

/ponytail:ponytail-review — proposes what might be removed according to the “you ain’t gonna need it” (YAGNI) principle.

Final validation steps are /coderabbit-local — to review the change locally, and /coderabbit — to pull and handle the code review findings from a CI run. At first I was copy-pasting code review findings from GitHub to Terminal, but then I recognised that dumb manual operation and made the skill.

Occasionally triggered🔗

/opsx:update — reworks a proposal when other changes affected one.

/playwright-cli — a skill that teaches the agent its browser CLI: run the e2e suite, read the trace of a failed test, poke at selectors on a live page. Gets invoked when e2e goes red, or whenever the question is “what does the browser actually see”.

/ponytail:ponytail-audit — a pass over the whole codebase, hunting for accumulated excess: dead code, speculative abstractions, dependencies that stopped earning their keep. A between-features ritual to catch a slow drift no single diff shows. Some prefer Claude Code’s /simplify for the same use-case.

Settings and guards🔗

Before any product code, I created the first version of the governing files.

CLAUDE.md started with 60 lines. There was a placeholder section for lessons learned at the end. Everything else was the meta-rules:

• The form: every rule must be checkable from a diff (“Validation errors return 4xx, never 500” — yes; “Be careful with auth” — no), one line (or closer to this form), imperative, non-duplicate.
• The fix & capture loop: whenever the same mistake is confirmed twice: a bug, a failed test, a review finding, a style correction mid-run — the agent fixes the code and proposes a rule, in the same turn, before treating the task as done. With a legitimate exit: “not capturing this” for one-offs, so the list doesn't bloat.
• The growth protocol: ~250 lines, whole sections move to docs/, CLAUDE.md becomes the index, docs never link to each other, no temporal language. I hit this trigger fast — the repo now has docs/code-style.md, docs/api-design.md, docs/testing.md, docs/verification.md.
• The knowledge-ownership table in the README file. Its main rule is: “One fact lives in exactly one file; everything else links to it.” My favourite rows are the ones I didn’t plan upfront: a file that owns “which linter suppressions are approved, and how many”, another that owns “how many mutants survive in src/model.ts, and why that floor last moved”, a third that owns “what counts as evidence for a claim”.

Separately, I set some permissions and guards. For instance, the agent can't install packages or run bunx on anything unvetted without me. Some prohibitions can't be expressed as permission patterns, so a scripts/command-guard.ts hook parses shell lines and refuses what the patterns can't see. The harness has its own test suite: like agent-permissions.test.ts or rulebook.test.ts — the rules that guard the code are themselves guarded by tests.

After several workflow runs, the settings grew with some notable pieces:

• Basic supply-chain hardening: Bun's minimumReleaseAge gate (3 days), exact versions, lifecycle scripts blocked by default, a slopsquatting rule, pinned action SHAs, gitleaks in the pre-push hook, Renovate with the same 3-day cooldown and a security-fix exception.
• Fresh docs alignment: a rule like “never call an unfamiliar API from memory; models invent methods”. There are two ways to honour it. One is Context7, an MCP that serves current library docs and claims to optimise tokens along the way. The other is simpler: just make the agent search for fresh documentation — increasingly viable as many tools ship agent-oriented md docs like llms.txt for this.
• no-suppressions.ts — the agent discovered // biome-ignore and @ts-expect-error. Now every suppression is budgeted and approved.
• file-size.ts — finds files ballooned past reviewability.
• spec-coverage.ts — acceptance criteria no test cites.
• mutation-floor.ts — mutation testing on the model's arithmetic.

None were in the original plan. All of them are fix & capture working as designed — and all of them are elements of evidence-packages.

System improvements🔗

In the middle of the experiment, I decided to review the harness based on the usage experience, some Frontend Nation conference talks and Alexander Polomodov’s recent videos. I took both repos, references, and my ideas into a separate Claude web UI session for an outside view, and compiled the findings into a staged checklist: temp.md file of 9K words.

I’ve split the checklist into sections and fed the agent on a product directory. Then I marked the completion and iterated. I made it manually — but the same file could just as well be fed to /opsx:explore as raw material for a harness-improvement change. The inbox pattern doesn't care whether the spec is about the product or about the machinery.

What such an outside view and the references helped me to improve:

• A vendored skill contradicted the project's policy. The playwright-cli skill ships Bash(npx:*) in its allowed-tools — a standing grant of exactly what the constitution forbids. The provenance hash catches an edited skill, not a permission conflict in an untouched one.
• A skill with a command injection. The now-unused /code-review carried a bare Bash in allowed-tools and interpolated its argument into a shell line.
• A decorative gate. CodeRabbit's docstring-coverage check sat permanently yellow and was turned off. A gate that always warns and never means anything motivates you to skip the whole list, devaluing the gates that must block.
• A rule that violated its own quality bar. “Scope a proposal to one reviewable cycle” — and yet one phase still shipped 5 features in a single +3000-line PR. Because “reviewable” is an adjective, and the harness's own EARS principle demands measurable values.

Actual product development🔗

Time to give some context on the pet product. It is frankly silly and easy — a Dota 2 pick assistant. As player who joins ranked all-pick match, you open the site, enter the heroes picked and banned and get suggestions what heroes to pick to increase your winning probability. Then you see the final predict on what team’s pick is stronger and how much — to decide, whether it’s worth doubling your rank wager or not. So, how I developed it with this new harness?

Initial brief🔗

I specced the product in a web UI chat before setting up the repo. In a couple of sessions I turned “a site you open next to a ranked Dota 2 All-pick mode game that tells you what to pick” into a stack of artifacts: 43 user stories with a v1/v2 split and recorded assumptions, a full scoring-model, a shared types.ts contract and a fixture generator producing a snapshot.

The artifacts landed in the repo through a staging area: spec-inbox/ — gitignored except its own README, because the repo is public and the raw specs are not. There are only two related rules:

1. A spec leaves this directory only by becoming an OpenSpec artifact or a source file.
2. If a referenced file is missing, ask the user — do not reconstruct a spec from memory.

Design🔗

The UI was drafted in Claude Design and then frozen into a screens spec before implementation. I prepared the design spec by prompting the web UI as well:

• single screen PWA
• no navigation
• no save buttons — the in-game timer is ticking for the user
• every input costs at most two actions on desktop
• screen states are derived from the session: an empty session shows setup, a partial one shows the board, a full one shows win probability
• refresh restores everything
• hotkeys are printed on the buttons that they trigger

Plan🔗

Specs and design from the inbox also go through OpenSpec: /opsx:explore reads the directory, gathers missing info and finishes the content. /opsx:propose then cuts the work into exact sub-tasks, which makes the plan two-level: first level lives in PLAN.md, the second — the details of every queue item — lives in the related proposal. PLAN.md answers “what's next and why”. The proposal answers “what exactly and how we'll know it's done”.

I prefer to create a pack of proposals through exploration first, then switch to a new session each time I want to apply and archive them.

Run🔗

The usual run just looks like /opsx:apply, then several /coderabbit, then /archive, /session-wrapup, /clear — then back to exploration, proposition, or application.

Let’s go through a log of a single feature as an example — that’s how I was integrating STRATZ and OpenDota data providers.

Session 1 — explore and probe. Started at 20% Claude’s weekly token budget. 279k tokens this session.

23:38 /opsx:explore. Spotted that the change reaches into the previous stage and offered a fork: probe the API first, or design against the docs. I picked probing. Despite the instructions printed the API key into the transcript — hah, classics.

23:52 The agent tried to call real API as a part of exploration. Figured out details on User-Agent and real rate limits. Schema introspection killed two assumptions from the spec. The most significant — is that a pick phase was missing in the win rate statistics.

00:05 /opsx:update to put those findings into the specs.

00:18 The agent branched off a name whose PR had already been squash-merged, trusted merge-base --is-ancestor over the list of PRs, and pushed into a mess. Cleaned up, new rule in docs/git-and-prs.md. Opened a PR.

00:35 /coderabbit to handle the findings, second push.

00:40 CodeRabbit limit refreshes in 43 minutes. Continue tomorrow.

Session 2  — propose

20:02 Resumed and /compact — to safe a few tokens.

20:04 /opsx:propose for the next sub-change. Twenty minutes later it handed the proposal back and asked for a /zombies pass. Added it, plus a rule not to pass the control to me before the gates pass.

20:31 Opened a PR for review. Went shopping.

22:21 The proposal came back oversize. Split it, and added a rule for specs that outgrow one review. Handle the finding through /coderabbit.

23:02 One more review iteration.

23:20 CodeRabbit bot didn’t appear for 20 minutes. Opened a support ticket, continue tomorrow.

Session 3  — apply. Ended at 31% of the weekly budget, 5 minutes before reset.

19:27 Started with compacting again: the previous session had ended at 400k tokens.

19:30–21:20 Dealing with two side branches before reaching the actual work with /opsx:apply.

21:56 35 minutes and 120k tokens later — folding in review results, taking ownership of a data schema from another feature to unblock this one, and splitting the change into pieces small enough to actually read.

22:20 The follow-up ran another 15 minutes and 50k. Four branches out. Merged one at a time; the second collected a conflict and a pile of findings.

23:45 A working client for the API. /session-wrapup, then waiting for the CodeRabbit limit to reset.

Session 4  — schema

00:01 Database schema, data models, a Postgres connector, a CI job that runs against a real database.

00:50 Two major review findings. The bot spent 20 minutes on them: spinning up containers, mocking, dropping the database.

01:41 Merged, generated the schema diagram, /session-wrapup.

Session 5 — first data group. Interrupted by IRL chores, so not an optimal run.

15:20 Patch detection, plus adding a new hero — which happens about once a year.

15:55 CLAUDE.md plus PLAN.md went over budget. Another section moved to docs/.

16:10 The agent found that no task in the whole change fetches the hero list: two groups consume it, nothing produces it. Fixed the task lists on a separate branch.

16:50 found OpenDota in proposal. Figured out what for. Can’t get rid of it as a second source, but halved the data fetched from there.

17:25 CodeRabbit's best find: URL.pathname isn't decoded. Clone the repo into a directory with a space in the name and every icon 404s and the dev server won't start. Two files outside the diff.

17:40 The agent committed a symlink to its own node_modules. Caught not by review but by the file-length test, which saw a new “extension” called s.

18:13 CodeRabbit failed to re-review for 40 minutes. Second support ticket; merged without it, four commits unseen.

Session 6 — the rest of the data. 681.9k tokens this session only. Together sessions 4-6 took 10% of the weekly budget.

19:20 Patch metadata.

20:26 Hero-versus-hero statistics. One request returns “against” and “alongside” side by side, so both matrices go to the database and synergy becomes its own term in the formula.

22:23 Bans and contest rate. Three surprises in the endpoint, all measured before anything was written: a mandatory heroId that filters nothing, a day counted from the epoch while the neighbouring endpoint uses a timestamp, and the ban counter living in a field called matchCount.

23:18 Replacing the previous data.

00:20 The first run against a fresh database failed 31 tests. The official image brings up its temporary server on a unix socket only, so pg_isready reports ready before the real one is up, and then drops the connections opened in that window. The same bug was sitting in CI.

Completing the integrations enabled me to work on a snapshot — a prepared data loaded at the start of the user’s session and then used on a client, potentially fully offline.

Let's skip all the later sessions to a resulting product state.

Results🔗

Check them out yourself. An operational product with modern DevEx, 3^rd party API integrations, own DB and job, deployed on a simple infrastructure - entirely by the agent.

Conclusions🔗

I expected higher progress. I dedicated 3 — 4 hours daily. The resulting harness is, well, mediocre. But even though the agent, wired by high-effort Opus 5, was running constantly those hours, it never hit the per-session limit of a Claude Max (×5) subscription, and never consumed more than ~40% of the weekly one.

The progress with the harness has a saw effect. Tightening it first drops throughput, the agent fails, you tune the rules, throughput recovers slightly above where it was. Then again. The line tends to the ceiling, which is set by the capabilities of the current model.

At checkpoints you still steer the agent. It’s not just about approval or rejection; it’s about correcting the direction. The agent doesn't reliably capture everything it's told in passing, and derailing it mid-task with a side quest rarely ends well. So it’s still better to keep your side-notes for things the agent should do later — to steer between sessions.

The agent’s not only unreliable in capturing everything the way you want. For example, I set the rules to chat with me in Russian, but keep everything landing in the repo in English. Still, ponytail answers in Russian about half the time, OpenSpec is always English, my own forked skills — well, it depends. Or when the gate shows OPEN, the agent sometimes passes control to me — though it has an instruction to handle gate comments by itself. So remember that every rule is probabilistic, not a guarantee.

Seriously, this is a thing worth highlighting and repeating. Never forget, the LLM output is random. This random affects each session. Harness narrows the random range, but never eliminates it completely!

Agentic coding at L4 is still tiring — and so are the levels above. You context-switch while waiting, then try to figure out what exactly you're approving. At higher levels the agent gets more autonomous, but you start to run several — and become the orchestrator.

And there’s a funny thing about L4. While I was doing my experiment, Claude shipped auto mode as the default permission mode — the “hamster-tapping” is now deprecated. Means the article became partly outdates right when I was writing it. A few more patch notes from updates Anthropic made last month to demonstrate how fast things are moving:

• cross-session messaging, where sessions discover and message each other
• /code-review moving to a background subagent with its own context window
• security plugin running multi-agent vulnerability scans
• worktree isolation hardening
• self-hosted cloud runners

As a consequence, the harness itself got outdated fast too. So just be mentally prepared to rework it often. Once widely hyped Ryan Carson’s Ralph was thrown away by a single Claude Code’s /goal command: repo with 21K stars is abandoned for half year already.

There's an old startup tale about guys who opened a tire shop and ended up building a B2B SaaS analytics platform for tire shops. That’s kinda true about the harness. When you start, it catches you more than the product you’re trying to build with it. Though it’s fine as you have an opportunity to reuse it on the next product — the thing I plan to do.

What's next🔗

A few things out of ambitious plans.

• Unlearn. The repo I forked a few skills from is material for an ongoing AI coding course I’m going to take.
• Separate harness repo. Otherwise, it’s hard to reuse and improve it in isolation from the product.
• Sandbox. I’d like to set up a dedicated VM with Claude Code's built-in sandbox inside it, a few commands to recover it easily, and remote control to drive it from my phone — and to combine overseeing agent sessions with gym sessions.
• Swarm. An orchestrator agent plus worker agents, each change in its own git worktree, several PRs in parallel with CodeRabbit reviewing them concurrently — so the 5–15-minute review latency stops being a tax. Cross-session messaging looks like a feature making the piping dramatically simpler.
• Trigger-driven work. Today, every loop starts with me. The next step is wiring external events to the agent — with n8n or something similar. First candidate: a finished GitHub CI run triggers the agent to read the gates’ outputs and handle them.

A lot to do. Hope there will be something more worth sharing.

If you found this article helpful, please show your support by sharing it with your friends or colleagues.

If you want to talk on managing engineering teams, designing systems, improving your tech product, and — I guess — applicable AI now too, send an email or message me.

Glad to connect on LinkedIn. Check out Mojam’s open positions.

Peace!

Share:

Updated: Thursday, August 27, 2026