Rafmer Internal Development Agent

Rafmer · Domain: Agentic Engineering, DevOps, Observability · Skills: Agent Design, Agent Orchestration, LLM/AI, Kubernetes, Docker, Python, OOP, Prompt Engineering, openai-agents, OpenCode, GitHub, Git, Project Management, Context Management

TLDR

What?

Built two iterations of an agent-driven coding system powered by local LLM infrastructure:

Version 1

Pure Python job dispatcher and workflow, interacting with OpenCode exposed tools and repo:

Version 2

Hermes profile with flow-step specific skills and dedicated OpenCode slots:

Why?

Inside Rafmer, I was doing more and more agent-driven development requiring me to sit in a continuous dialogue with OpenCode, often lasting several hours. Rafmer had also moved to tracking tasks in tickets through several GitHub project Kanban boards. The Rafmer systems comprised several smaller repos, so I worked manually where necessary and context-switched to and from several active OpenCode instances doing vastly different work across different repos.

As Rafmer’s agent-side DevOps tooling became more mature, with added Prometheus, UI-Debug, and Kubernetes scripts/skills, my role as the human in the loop became more redundant. I was essentially answering mostly redundant questions and guiding OpenCode through the ticket, implement, validate, correct, validate loop. To preserve my own time and sanity, I wanted to at least automate part of this process so I could simply do code reviews remotely and only periodically validate the whole repo, without having to sit at my computer waiting for OpenCode to ask me random questions.

Important caveat:

Rafmer’s infra was internal, and the apps were non-critical (for now). The infra (including LLM and serving) was strictly Rafmer’s own on a completely private network, so the leak surface was minimal. With the infra being Rafmer’s own, token usage could also be maximized, which would have been a significant rolling cost with an external provider. Periodic backups of both code and data were taken, and as a two-person startup with minimal capital, getting other devs onboard was not an option. This was an option under Rafmer’s conditions and not necessarily the right choice for a larger org with external infra.

How?

Version 1

Version 1 was a two-stage Python system:

  1. Dispatcher: handled sorting and chain building of tickets stored on GitHub via LLM calls and dispatched them to the worker
  2. Worker: ran the implementation loop on the connected repo

The dispatcher pulled ready tickets from GitHub projects and sorted them into dependency chains, assigning dependencies to tickets and dumping the sorted state to disk. The dispatcher had a series of workers to which it delegated each chain. The worker picked up the next ticket in the chain.

On a new ticket, the worker reset the repo state using basic git actions and checked out whatever dev branch it was responsible for. It then moved on to the dev flow, done in part by a Python library wrapping around OpenCode and subagents using the openai-agents framework:

  1. Repo Validation: determine whether anything had to be set up before starting. Yes/Correct/Hard Failure
  2. Enrich ticket with repo context: assess whether the ticket could be done with the current repo. Yes/Soft failure (ticket comment), next ticket
  3. Plan: break down the enriched ticket into concrete steps
  4. Implementation - Review: OpenCode implemented, the agent reviewed, committed and moved to the next step / returned to implement (4 iterations) / soft failure (ticket comment)
  5. Review: checked the acceptance criteria, compared the actual code against the plan, pushed PR / global correct (3 iterations) / soft failure + pushed partial PR + ticket comment

Architecturally, each step was its own class, subclassed from an abstract step with the per-step state management handled within the class parameters. The flow came together in a top-level caller (i.e. the worker) as a step-by-step flow iterating over each step instance. Each PR ended up in a PR-stack per chain to simplify the review, and was reviewed by me on GitHub.

The LLM-stack for this version was Qwen3.5-35B-A3B and Laguna-S 2.1 running on a PoC version of the Rafmer Edge Node. Laguna was used in the planning and enrichment part to compensate for the Qwen MoE’s lack of hard intelligence. It was hosted on an experimental node with 128 GB due to its size, with roughly 100k context going at 15 tps. The Qwen was responsible for the implementation and ticket sorting. It ran through the internal Rafmer cluster, primarily chosen for being the best LLM for throughput at the time, doing roughly 130 tps with 300k context per deployment.

Version 2

Version 2 was a Hermes profile hooked up to a dedicated gateway, with isolated OpenCode slots that could work in parallel on separate tasks in the same repo. Hermes communicated with each OpenCode instance through the sloctl-mcp, and each instance had its own checkout of the repo that was fully reset as needed.

The Hermes dispatcher decomposed tasks, either by reading tickets with a cron job, through a connected UI to the gateway, or by the user messaging it from Telegram or Signal. The dispatcher had the previous flow tasks decomposed as specific skills and sent queries directly to OpenCode and validated the results. The usage of skills and the scope of responsibility + heuristics were managed through the profile’s SOUL.md.

The whole system made use of the Rafmer cluster running only Qwen3.8-27B, on deployments running at the dense-model throughput documented in the Rafmer LLM Infrastructure throughput table. If context, model quality and token speeds had been much lower, the Hermes approach would have been much less feasible. Hermes was also built to self-audit and improve, which meant I could interrogate the instance as a means of observability and that its traces were preserved.

In Common

Both designs were made to be GitHub-project, repo, LLM and coding-language agnostic, configurable from a simple .env. Both systems were deployable through Docker Compose and Kubernetes, and runnable on host.

Results

Version 1

Version 1 was a good start for problem scoping, but was bogged down by sub-agent overspecialization, large model bottlenecking, MoE output quality and too much rigidity. Some findings:

  1. Flow worked very well for pure Python repos
  2. Very hard to keep track of end results of prompt orchestration
  3. Minimal and flaky observability through the OpenCode Python submodule (std.out being eaten)
  4. Whole flow bottlenecked by the single-client larger model
  5. Flow broke down for simple tickets like docs, requiring layers of special casing
  6. Contradictory system-prompts across subagents and flow-steps
  7. Pure code, big maintenance surface, especially for large prompt templates
  8. Dumb MoE model requiring tool definitions and example use cases in each system prompt, further widening the maintenance surface
  9. Spotty repo-state management by pure git

Overall, Version 1 was a good case study in harness vs agent-flow engineering and a good throughput test for Rafmer’s agent infra.

Version 2

Version 2 was a significant improvement over Version 1, primarily due to the choice of harness and model capability:

  1. Could communicate asynchronously across multiple channels and continue across them
  2. Actual observability through Hermes’ built-in capabilities
  3. Much larger degree of freedom through Hermes and tools, allowing more dynamic task-solutions and ticket breadth
  4. Ability to separate scheduled work and ad-hoc tasks, and multitask at different levels
  5. Homogeneous models meant no single one was bottlenecking, both quality and performance-wise
  6. Ability to communicate with Hermes about background tasks
  7. A lot simpler to maintain, debug and audit
  8. Hermes could modify tickets and answer questions about the project as well as the repo

Some of the results could be attributed to improved models, but UX, observability and overall flexibility were the winners here. All in all, a much better agent as opposed to the pre-defined workflow Version 1 was.