Introducing: Cohereny — Reviewed Context for Workflows Across Multiple Language Models

As of today, Cohereny is released as open source under the AGPLv3. The source code is on GitHub, and the project site is cohereny.com. This post covers what Cohereny is, why it exists and what of it works today. If you want to go deeper, the repository holds the architecture with every decision and its reasoning.

The problem

A prompt that works well on one model often does worse on another. Not because the other model is weaker, but because it expects a different structure, tone and format. Three effects add up, and you pay for them again with every model switch:

• Model variance: the same prompt behaves differently across models.
• Model drift: the same model behaves differently after an update.
• Prompt sensitivity: small changes in wording change the result.

In a workflow with several steps and several models, every step needs its own context to get right — and each one is exposed to all three at once.

How it started

An AI system can run flawlessly, be fully logged and still quietly get worse over weeks, because no control measures what actually comes out. I ran into this in a private project, and again when comparing notes with people I know.

It stuck with me, and I wanted to know whether it could be done better. So I tried — because I could, because I enjoy it, and because I wanted to find out in practice what works and what doesn’t. The question behind it is still the one Cohereny works on: how do you build a workflow out of several language models so that it stays traceable, even when the models underneath it change?

The approach: context doesn’t belong in the code

Cohereny keeps context outside the application code, in a repository every team shares: system prompts, domain knowledge, examples, tool schemas, routing rules. Each piece is a versioned block that only takes effect after review, and it belongs to a scope — an organisation, a team, a project or a single person.

At runtime, Cohereny assembles exactly the context a workflow step needs and hands it to the model the routing rules pick.

A few properties that mattered to me:

• Approval instead of direct edits. A block is written, submitted and approved. What one person learns about a model ends up in the repository, where everyone benefits — after someone has looked at it.
• The privacy ratchet. Further down the tree, a privacy rule can only get stricter, never looser. A project may be stricter than its organisation, but not more lenient.
• The data decides the model. Every scope carries a data class and every model a ceiling. Data that must not leave your network does not go to a cloud model.
• People at the points that matter. No workflow without an approval gate: Cohereny won’t activate a workflow in which a model call can end without human review, and a gate can work on the four-eyes principle.
• Nothing disappears. The audit log is append-only. It records who approved what, and when.
• Headless. Every function starts as an API endpoint, split into four API areas for serving, authoring, observing and administering, all protected by OpenID Connect.

What is built, and what isn’t yet

Cohereny is currently a pre-release and not meant for production use; the API will still change. What is built is on GitHub, the roadmap on cohereny.com.

Built and covered by tests:

• the context repository with versioned blocks, a review lifecycle and approval per scope
• composition along the scope tree, including the privacy ratchet
• routing rules as reviewed blocks, with data classes per scope and ceilings per model
• the workflow runner with branches, joins and approval gates
• the audit log and the API per area, protected by OpenID Connect

The benchmark is in progress. Calibration anchors and measurement tables are in place; runs against the configured models come next. The aim is publicly reproducible evidence of whether, and by how much, adapting context to the model pays off. It is also the piece that was missing in the experience above: a measurement of what actually comes out.

The adaptation itself is still planned: tailoring prompt and context to each model, derived from the benchmark results. That is the part Cohereny is ultimately about — and I will only call it done once the measurements show it works.

How Cohereny is built

Cohereny is a private project: built in my own time, on my own hardware, with my own accounts and licences, and at my own — and therefore very limited — expense. Development happens on a self-hosted GitLab; the GitHub repository is a snapshot of it.

Even so, it is built the way I think a proper software project should be, even though I work on it alone:

• Coding conventions that are checked automatically
• Tests with every change, including counter-checks: the test shows a safeguard works, the counter-check shows it would not work without the change
• No commit without a ticket, no merge without review
• Architecture decisions as ADRs, in arc42 documentation alongside a whitepaper, both in German and in the repository

The same goes for my coding buddy Claude Code, which writes much of the code with me. I looked at and checked every change it made, then approved it, or rejected it. How I work with Claude Code is in “Claude Code: Helpful, But Not a Free Pass”.

A few numbers as of today:

The pipeline fails if coverage drops by more than half a point.

Why open source, and why the AGPLv3

A tool that promises traceability should be traceable itself. That’s why the code is open, along with the architecture and the reasoning behind it.

The AGPLv3 keeps it that way: if you distribute a modified version or run it as a network service, you must make its source code available. For uses where that doesn’t fit, such as a proprietary integration, a commercial license is available on request.

What comes next

• Benchmark runs against the configured models, as evidence for adaptation — or against it
• A one-command setup with Docker. Today you still need to provide PostgreSQL, Redis, an OIDC provider and a LiteLLM gateway yourself
• Version 1.0, planned for the first quarter of 2027

The roadmap with the current state is on cohereny.com.

Getting involved

Issues are welcome. I can’t merge pull requests yet, because the GitHub repository is a snapshot — CONTRIBUTING.md explains why. Please report vulnerabilities to [email protected] rather than opening an issue. Questions and everything else go to [email protected].

And if you want to know which architecture questions sit behind Cohereny: I’m writing a book about them — more on that here.