Lane-first sweeps
Useful discovery runs, but model, effort, prompt, transport, planner, parser, and downstream changes could be entangled. Counts and contract validity often arrived before full atom and answer quality.
Mara · experiment cartographer · editorial guide persona
Who is Mara? A fictional editorial voice created for this explainer—not a person, model, sub-agent, or component in the extraction pipeline. She does not run experiments or make promotion decisions.
My read: Arranger is a strong fit because our failure is not “which model emits the most rows?” It is “where does a meaningful obligation disappear, which combination of stages prevents that loss, and is the lift worth its cost?” We have recurring model changes, interacting stages, expensive judges, recoverable provider drift, and multiple tenants. That is exactly when a retained map compounds.
Recipe-first experiment map
One frozen evidence release, exact factor assignments, content-addressed stage receipts, separate courts, explicit missingness, and a qualified recipe or portfolio at the end.
The experiment map, in one picture
A map is the frozen evidence, every executable recipe path through the pipeline, the receipts at each seam, and the separate courts that judge the terminal result. A model name alone is not a coordinate.
Raw sourceTranscript bytes, gold obligations, mention surfaces, known-at time.Frozen support
Census + boostOne exhaustive inventory. Never pre-filtered by a planner.Locked ceiling
Post-census plannerOptional decomposition or audit plan over the complete inventory.Optional
WriterTurns census obligations into high-quality candidate atoms.Factor seam
Auditor / repairFinds semantic gaps and local mechanical failures without erasing content.Factor seam
Entity custodyResolved, unresolved, or unbound—never silent false merge or drop.Factor seam
Retrieval mapRepresents atoms so later questions can recover every useful obligation.Factor seam
SynthesisReconstructs a faithful answer with no downstream obligation loss.Terminal value
Recipe coordinate
At each seam: implementation + model + prompt + effort + reasoning mode + transport + output contract + predecessor receipt hashes. Identical prefixes are reusable; changed coordinates are not.
One terminal recipe output
Semantic recall + atom usefulness
Evidence + provenance integrity
Entity fidelity + unresolved custody
Operations + contract validity
Cost + latency + capacity
Qualified Pareto frontier
Walk the map
Select a step. The important change is that each experiment becomes an executable, reusable object rather than a row in a temporary leaderboard.
01
We were comparing lanes, but several changes could move at once.
Model, effort, prompt, transport, planner behavior, parser recovery, and downstream representation were sometimes bundled together. Claim count was visible before atom usefulness, entity custody, retrieval, and answer recall were fully judged.
Mara's read
Useful discovery work, but not yet a causal map. A bigger inventory could look like a win while downstream quality remained unknown.
Why this is better for the failure we actually saw
The method follows the measured loss downstream instead of assuming a near-perfect census means a near-perfect knowledge system.
A ceiling is not delivery.
The v3 census made 175/178 gold facts visible, yet downstream atom and answer behavior still missed obligations. Arranger keeps census, stored recall, retrieval, and synthesis denominators distinct.
Good content survives mechanical drift.
The recovered nested wrapper restored 14 valid claims from immutable response bytes. The lane remains promotion-blocked, but semantic evidence is not thrown away because JSON shape drifted.
The winner can be a portfolio.
Luna can own cheap volume, Terra can earn an audit seam, and Sol can remain a bounded residual specialist—if paired quality evidence shows each role creates value.
How we play the first exact game
The first board is deliberately small: writer upgrade W × post-write auditor A × entity tri-state E. Three binary players produce all eight coalitions. Click the factors or a board cell to inspect the recipe coordinate.
Choose a coalition
A bit is on when that upgrade is present. This selector only explains the board; the real harness executes every cell.
Selected 000: baseline recipe.
1 · FreezeEvidence, obligations, splits, prompts, selectors, missingness law.
2 · Execute 2³Run all eight cells on identical support. Missing is not zero.
3 · ScorePrimary game is terminal meaningful-obligation recall and usefulness.
4 · AttributeSend the complete sealed value table to Precise exact Shapley / Owen.
5 · DecideUse floors and the Pareto frontier to choose; attribution explains, never promotes.
Do we need Shapley math?
Arranger works before attribution. Shapley becomes useful only after the harness has measured a complete, frozen game.
Short answerNo—not to run, compare, or promote recipes.
Yes only when we want to apportion measured lift among interacting upgrades. We should consume Precise's canonical sealed kernel rather than copy the math into Konstant.
Legitimate
- Players are binary version upgrades, not competing model names.
- Every one of the 2n coalitions was executed.
- Every cell uses identical evidence support and one frozen scalar value function.
- Recall, cost, and latency are separate games.
- Owen groups are real and declared in advance.
Not legitimate
- Missing or imputed coalitions.
- Different documents, gold, judges, prompts, or missingness policies by cell.
- Using raw atom count as the value function.
- Treating Luna, Terra, Sol, and Gemini as simultaneous “players” in one model slot.
- Calling attribution causal proof, winner selection, or production permission.
| Common model screen | Luna / Terra / Sol / Gemini × effort × reasoning modeRanks complete recipes on shared evidence. This is not a Shapley game. |
|---|---|
| Atom construction game | writer upgrade × auditor activation × entity tri-state = 2³A valid first exact game after every one of the eight recipes is measured. |
| Delivery game | atomization upgrade × retrieval representation × synthesis repair = 2³Separates atom quality from whether the final answer actually carries the obligations. |
Use the real Precise harness at the seam
This is not a proposal to reproduce Shapley in Konstant. Precise already exposes the executable experiment law and finite-game core; Konstant should emit sealed inputs and consume verified outputs.
Konstant owns the extraction map
Generate the frozen evidence manifest, recipe coordinates, stage DAG, terminal outputs, court scores, complete coalition value table, and cost/latency receipts. Raw customer content stays here.
Precise owns the attribution kernel
createResearchProgramHarness freezes and verifies the experiment. createStagedResearchAdapter preserves populations and stage receipts. createFiniteAttributionCore verifies the complete game and runs canonical exact Shapley, interactions, and Owen through 16 players; 17–20 is seeded sampled research-only.
Thin cross-company contract
Konstant emits a sealed complete value table + manifest hash + known-at time + value semantics + player order. Precise returns an attribution-only report bound to the game and implementation hashes. Boombox can host the worker and durable receipts without owning Precise math.
Optimizations for the mapping problem
The expensive part is learning a high-dimensional, mixed discrete map without corrupting the confirmation result. We can spend adaptively during discovery and still preserve an exact, interpretable final game.
Screen, then complete
Use a cheap common-support screen to eliminate obviously dominated model and effort lanes. Then run a complete exact microgame over only the surviving binary upgrades. Screening may be sparse; the attribution game may not be.
Cache the stage DAG
Content-address source, census, planner, writer, entity, retrieval, and synthesis outputs. Reuse an identical prefix when only a downstream factor changes, while retaining a receipt for the exact predecessor chain.
Spend judges by uncertainty
In discovery, use stratified audits, successive halving, and a surrogate acquisition rule to buy the next judgment where it can change the Pareto frontier or reduce uncertainty most. Keep inclusion probabilities and never adapt on the untouched confirmation split.
Pair every comparison
Use the same documents, obligations, census, judge contract, and seeded randomness across candidate cells. Paired deltas remove far more noise than simply buying more unpaired judgments.
Batch compatible work
Group calls by provider, model, prompt contract, and output schema; run independent batches asynchronously under explicit socket and spend ceilings. Batch measures discounted quality economics and provider capacity—not live latency. Every finalist must still pass a matched realtime canary before it can enter the production recipe.
Score once, keep courts separate
One materialized recipe output can feed semantic quality, evidence, entity, operations, cost, latency, retrieval, and synthesis courts. Reuse the bytes, but keep each value function and promotion floor distinct.
Use hierarchical games
Prefer small exact stage-local games and declared Owen groups—extraction, representation, delivery—over one enormous flat game. Only carry qualified assemblies into the next layer.
Shortcuts that break the map
Do not impute missing coalitions, adapt on confirmation, change denominators between cells, stop failed cells and call the game complete, use raw atom count as terminal value, or let one composite score hide evidence and entity failures. Those make the dashboard cheaper by making the conclusion unknowable.
What I would add next
Enough structure to make the next change cheaper and more trustworthy, without pausing the actual ablations to build an ornamental framework.
Final judgment: keep the idea, but keep it honest. The map must be generated from executable recipes and immutable receipts. The courts must stay separate. The confirmation set must remain untouched. And Shapley must be allowed to say “not computable” when the game is incomplete. If we do that, Arranger gives us a durable learning system—not just a prettier eval dashboard.