Konstant extraction ablation lab

Help steer this

These are the live choices where Adam's judgment can change strategy. The recommended option is an evidence-informed default, not a silent decision.

Which v3 downstream branch should get the next paid judge budget?
Direct low and medium are clean, while the planner lane has durable Batch work but still needs reconciliation. The comparison can now answer architecture rather than merely count atoms.
1. Finish and judge direct-low, direct-medium, and planner-low→low together (recommended).
2. Pause planner judging and spend first on the direct→gap-auditor→localized-repair challenger.
3. Cross-validate direct-medium on two fresh transcripts before any more planner work.
Where should the retrieval follow-up isolate loss first?
Three fixed-ID pairs show that full fact text preserves custody but does not automatically improve the final CEO answer.
1. Measure selector/reranker coverage before changing the answer prompt (recommended).
2. Hold the canonical packet fixed and add a coverage-aware synthesis checklist.
3. Run both as a 2×2 so selection and synthesis interaction is visible.
How strict should the first entity canary be?
The tri-state contract exists, but production still conflates unresolved valuable mentions with junk.
1. Shadow Precise first and require zero unsafe merges plus searchable unresolved mentions (recommended).
2. Start with the harder Blockdaemon recurring-meeting canary.
3. Prove projection parity only before adding retrieval-answer evaluation.

The map we use to make the next decision

Every model or orchestration idea is a stick on the same frozen evidence. Separate courts decide whether it advances, changes role, or stops; raw atom count never chooses the winner.

01 · Frozen map

Same 178 gold factsTwo held-out transcripts, 105 fixed census moments, exact source and receipt hashes.

One census ceiling175/178 facts visible. Downstream candidates cannot quietly change the source-discovery denominator.

02 · Candidate sticks

Luna-medium · control$0.2129 · 734 atoms · 148 exact / 28 partial / 2 missed.

Sol-high · challenger$2.4826 · 1,127 atoms · 154/23/1. +1.97 points, 11.66× writer cost.

Terra low / medium / medium-pro$0.4906 / $0.5115 / $2.0200. Candidate-independent semantic court running.

Gemini 3.1 Pro writer$1.0616, but the output contract failed. Content is recoverable evidence; the lane is not promotable.

03 · Independent courts

Mechanical custodyHashes, complete moments, exact model, valid rows. Failure stops promotion—not evidence recovery.

All-fact semantic coverageOne anonymous candidate per request. Full misses first; exact/partial and attribute loss second.

All-atom qualityLuna versus Sol: 1,861 canonical cells, 1,861 exact-packet cells, four duplicate units.

Terminal usefulnessRetrieval, synthesis, and tri-state entity custody determine whether good atoms survive to answers.

OperationsBatch cost and capacity are separate from a matched realtime critical-path canary.

04 · Route decision

Live bulk writerUnassigned. Requires terminal quality plus matched realtime latency.

Batch / reingestionUnassigned. Can favor slower discounted quality when queue time is acceptable.

Bounded specialistSol's likely alternative if its marginal atoms are excellent but bulk economics or latency lose.

Stop / diagnosticGemini writer and current contract-invalid planner remain informative, not deployable.

Hold the map fixed. One changed factor means a causal comparison rather than a model-name anecdote.
Advance on useful information. Closing meaningful misses matters; a larger inventory alone does not.
Assign a role, not only a winner. A costly model can earn a residual or Batch seam without owning live volume.
Confirm at the terminal. Retrieval, synthesis, entities, and realtime behavior can still erase an atom-level win.

Current read: Luna is the cheap control; Sol earned the all-atom court but not the live route; Terra is being screened for a cheaper quality frontier; the post-write auditor remains the highest-information orchestration challenger.

What happened

Newest first. A provider job, a passing contract test, and a measured quality result are deliberately different evidence states.

Proven result measured experimentContract only harness capability, not qualityInvalid / failed preserved but not rankedRunning no conclusion yetIdea untested

Jul 16, 2026, 2:15:48 AM ETrunning

Question

Is Sol-high's 1.97-point semantic lift made of faithful, self-contained, atomic, retrieval-worthy information, or costly duplication and fragmentation?

Candidate-independent canonical, exact-packet, and document-duplicate quality courts over all 734 Luna-medium and 1,127 Sol-high atoms heldout-v3-independent-all-atom-luna-medium-sol-high

What changed: Refactored the all-atom runner so each canonical and packet-quality request contains exactly one blinded candidate. Submitted the first 96-request canonical Gemini 3.1 Pro high Batch; packet and sparse duplicate courts follow from the same immutable manifests.
Result: The canonical all-atom Batch is running. No precision, fidelity, atomicity, retrieval-worthiness, or duplicate verdict has been declared.
Why it worked or failed: Sol-high closed one full miss and gained six exact facts, which is large enough to justify finalist judging despite its 11.66x writer cost. The court must now determine whether the extra 393 rows are useful atoms.
Decision: Keep both lanes provisional. A Sol bulk route needs material terminal quality; a smaller residual-specialist role remains possible if only a bounded subset creates value.
Next test: Reconcile canonical and packet cells, run one-candidate whole-document duplicate clustering, then compare end-to-end usable retrieval and synthesis.

QualityPending 1,861 canonical cells, 1,861 packet cells, and four duplicate units.Cost / speedGemini 3.1 Pro high native Batch; exact court cost pending. Writer cost remains Luna $0.2129 versus Sol $2.4826.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-independent-all-atom-luna-medium-v-sol-high-20260716/transcript-classifier-ablation

Jul 16, 2026, 2:08:13 AM ETproven result

Question

Does Sol-high's 53.5% larger atom inventory preserve materially more facts than Luna-medium without relying on side-by-side judge context?

Candidate-independent semantic court over the same 178 held-out facts: Luna-medium control versus Sol-high writer heldout-v3-independent-semantic-luna-medium-sol-high

What changed: Replaced the all-candidates-in-one-prompt judge with twenty single-candidate Gemini 3.1 Pro high Batch units. Each lane now sees the same immutable gold under an independent blind map and stable candidate-content identity.
Result: All 356 cells certified with zero unresolved or quarantined invalid rows. Sol-high scored 154 exact / 23 partial / 1 missed, or 92.98% partial-credit recall. Luna-medium scored 148/28/2, or 91.01%. Sol gained 1.97 points, six exact facts, and closed one full miss.
Why it worked or failed: Prior side-by-side verdicts moved when an unrelated third candidate changed. Single-candidate requests remove that comparison-context confound while keeping the gold, judge prompt, and scoring contract paired.
Decision: Advance both lanes to the candidate-independent all-atom court. Treat Sol's semantic gain as meaningful enough to test, not sufficient to overcome its 11.66x writer cost.
Next test: Judge every atom for canonical and exact-packet support plus whole-document duplication; run a matched realtime canary only if Sol-high remains quality-qualified.

QualitySol 154 exact / 23 partial / 1 missed versus Luna 148/28/2 across the same 178 facts.Cost / speed$1.5358 judge cost; 194.095s invocation wall. Batch wall is capacity evidence, not live latency.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-independent-semantic-luna-medium-v-sol-high-20260716/transcript-classifier-ablation

Jul 16, 2026, 2:06:34 AM ETproven result

Question

Can a stronger OpenAI writer extract materially more useful atoms than Luna-medium at an acceptable quality and cost?

Fixed v3 census; direct Sol-high atom-core writer over the same two held-out transcripts and 105 moments heldout-v3-sol-high-writer

What changed: Held census, source visibility, prompt, cap, and batching fixed while replacing Luna-medium with gpt-5.6-sol high.
Result: Sol-high returned 1,127 contract-valid claims with zero invalid rows across all 105 moments. Artifact cost is $2.4826 and provider Batch wall is 2,243.949s. Luna-medium returned 734 claims for $0.2129 on the identical input.
Why it worked or failed: Sol-high produced 393 additional claims, but it cost 11.66x as much and took 37.4 minutes of Batch wall. More rows are only valuable if they close meaningful misses without adding distortion, duplication, or retrieval noise.
Decision: Admit Sol-high to the repaired semantic screen; do not promote it and do not treat Batch wall as a live-latency measurement.
Next test: Compare Sol-high and Luna-medium under candidate-independent semantic judging, then judge all atoms only if Sol-high's marginal coverage is material.

QualityMechanical contract passes; semantic recall, faithfulness, atomicity, retrieval worthiness, and duplicate rate are pending.Cost / speed$2.4826 artifact cost versus $0.2129 for Luna-medium; 2,243.949s Batch wall is offline capacity evidence only.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-direct-writer-openai-ceiling-gemini-screen-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-sol__high__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-db0b4d734b2f__p-853202b7cee2__i-02528b44a1a9

Jul 16, 2026, 1:05:09 AM ETproven result

Question

Which v3 lane preserves the most meaningful facts, and does the planner's larger inventory actually close full misses?

Paired blinded semantic judge over v3 direct-low, direct-medium, and planner-low→low heldout-v3-threeway-semantic-judge

What changed: Submitted ten Gemini 3.1 Pro high native Batch judge units against the same 178-fact gold and exact candidate manifests.
Result: All 534 cells reconciled with two repair requests and three invalid rows quarantined. Luna-medium scored 156 exact / 20 partial / 2 missed, or 93.26% partial-credit recall and 94.12% importance-weighted partial credit. Planner-low→writer-low scored 157/17/4 and remained contract-invalid. Luna-low scored 146/27/5.
Why it worked or failed: The result established Luna-medium as the strongest promotable lane in this court, but comparison with the earlier candidate set changed 17/178 Luna-low and 9/178 Luna-medium verdicts. The partial-credit movement was small, yet the labels proved context-sensitive.
Decision: Keep these scores as measured historical evidence, not the final ranking court. Supersede the side-by-side design with candidate-independent requests and retain Luna-medium as the control.
Next test: Repeat the semantic court one candidate per prompt, then judge every atom only for the two quality-qualified finalists.

QualityLuna-medium 156 exact / 20 partial / 2 missed; planner output is unpromotable; candidate-independent confirmation required.Cost / speed$2.0413 for the recovered Gemini 3.1 Pro high Batch judge.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-threeway-semantic-luna-low-medium-planner-low-20260716/transcript-classifier-ablation
.harness/gary-local/stage2-heldout2-census-v3-threeway-semantic-luna-low-medium-planner-low-recovered-20260716/transcript-classifier-ablation

Jul 16, 2026, 1:03:11 AM ETproven result

Question

Can the planner preserve the zero-full-miss advantage without repeating its self-containedness and retrieval-quality losses?

Held-out v3 census; Luna-low post-census planner → Luna-low writer, native OpenAI Batch heldout-v3-planner-resume

What changed: Held the 175/178 v3 census fixed, reused certified planner packets, and localized omission/schema repair before writer fanout.
Result: The lane materialized 959 kept claims over all 105 moments, with one invalid row quarantined. Artifact cost is $0.4243: $0.2260 planner plus $0.1983 writer. The current resume invocation took 3.518s because it reused provider receipts; accumulated provider Batch wall including queues is 1,401.162s.
Why it worked or failed: Exact-manifest receipt reuse recovered paid work without repeating 48 planner/writer requests. The 959-row inventory is execution evidence only: the quarantined row and much larger output make paired semantic and all-atom quality judging essential.
Decision: Keep the completed lane in the v3 judge, retain the invalid row as an explicit promotion blocker, and do not infer a planner win from inventory size.
Next test: Finish the lane scorecard, then run one paired blinded v3 judge across direct-low, direct-medium, and planner-low→low with all-fact coverage and fully missed facts first.

QualityQuality unknown until paired semantic and all-atom judging finish.Cost / speed$0.4243 artifact cost; 1,401.162s provider Batch wall including queues; 3.518s local receipt-resume invocation.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-luna-planner-low-writer-low-cap32-batch-20260716/transcript-classifier-ablation

Jul 16, 2026, 1:03:11 AM ETcontract only

Question

Can a changed lane directory safely reuse an already-paid Batch receipt when the exact 22-request manifest is identical?

OpenAI Batch resume after a configuration-hash drift openai-batch-exact-manifest-resume-incident

What changed: A config-hash drift initially submitted a second identical 22-request writer manifest. It was canceled immediately; three requests completed before cancellation. Exact-manifest cross-lane reuse was then implemented and tested.
Result: The original 22/22 receipt was reused to materialize the writer output. No certified response was discarded, and the duplicate job incident remains visible instead of being folded into model quality.
Why it worked or failed: Lane-path identity was too strict while request-manifest identity was the real reuse boundary. Exact ordered request ids and hashes allow safe reuse across config-directory drift; merely similar prompts do not.
Decision: Use exact ordered manifest equality for cross-lane receipt reuse and preserve canceled/partial duplicate jobs as operational-cost evidence.
Next test: Add the incident to cost accounting and verify that any one-request hash change forbids reuse while an identical 22/22 manifest resumes idempotently.

QualityContract proof only; it changes resume safety and cost custody, not atom quality.Cost / speedThree duplicate-job requests completed before cancellation; their incremental provider charge must remain separately attributable when reported.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-luna-planner-low-writer-low-cap32-batch-20260716/transcript-classifier-ablation
lib/llm/batch-queue.ts
lib/llm/batch-queue.test.ts

Jul 16, 2026, 12:48:12 AM ETproven result

Question

Does preserving full atom text in an otherwise identical retrieval packet improve the final CEO answer?

Blockdaemon CEO synthesis; three fixed-ID paired runs comparing 120-character proposition text with canonical full fact text retrieval-fixed-id-three-pair

What changed: Kept the same 240 atom ids, order, and metadata; changed only 208 visible fact cells. Canonical packets used full fact_text and were about 1.95% larger.
Result: Prop120 won coverage quality in all three pairs (0.9089 vs 0.8967; 0.9785 vs 0.8918; 0.9953 vs 0.9141). Final-answer oracle recall favored prop120 in pairs 1 and 3 and canonical in pair 2. Neither serialization cleared the answer coverage gate.
Why it worked or failed: Full text restores evidence custody but does not force the synthesizer to select or express more obligations. The three-pair pattern points downstream to selection and synthesis variance rather than proving that truncation was harmless.
Decision: Keep full fact_text as the custody-correct live serialization. Do not claim an answer-quality lift; treat packet selection and synthesis as the next measured bottleneck.
Next test: Use the fixed canonical packet to isolate selector/reranker coverage from answer-model omission, then test a coverage-aware synthesis prompt on the exact same ids.

QualityCoverage quality varied materially by replication; exact fixed-ID repetition was necessary to avoid a false one-run conclusion.Cost / speedgpt-5.6-luna medium realtime; 15.385–21.940s per answer, 113,107–115,713 input tokens, and 2,117–3,066 output tokens.

Evidence paths

.harness/homebase-v2-packet-capture/blockdaemon-ceo-serialization-ablation-20260716/serialization-pair/2026-07-16T04-31-49-089Z/serialization-pair-summary.json
.harness/homebase-v2-packet-capture/blockdaemon-ceo-serialization-ablation-20260716/serialization-answer-ablation

Jul 16, 2026, 12:43:49 AM ETinvalid / failed

Question

Can a cheaper large-context Luna census match Gemini v3 visibility under the exact strict census contract?

OpenAI Luna high/medium as a direct census challenger on the same two held-out transcripts openai-luna-census-schema-400

What changed: Sent the exact v3 census prompt and nested JSON schema through native OpenAI Batch at explicit half-rate pricing.
Result: The provider job completed, but its primary rows returned HTTP 400 for the census schema and produced no valid census output or scorecard. This is a request-contract failure, not a Luna quality measurement.
Why it worked or failed: The strict nested schema accepted by the Gemini route was not accepted on this OpenAI Batch request surface. Model quality was never exercised.
Decision: Keep the failed receipt. Adapt the provider-specific schema mechanically or use JSON-object plus the same strict local validator; do not weaken semantic validation.
Next test: Run a no-spend request-shape preflight, then one bounded canary before resubmitting the two-document census challenger.

QualityNo model-quality conclusion is valid from this lane.Cost / speedBatch id and completion receipt are preserved; no valid semantic output was available for quality or cost comparison.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-openai-luna-high-medium-batch-20260716/transcript-extraction-ablation

Jul 16, 2026, 12:37:47 AM ETproven result

Question

Why does the cheap planner improve coverage while degrading self-contained, retrieval-worthy atom quality, and what is the highest-information next test?

Fable 5 low independent methodology critique of the downstream ablations fable-methodology-consult

What changed: Showed Fable the measured direct/planner recall and all-atom quality results and asked for a skeptical method critique rather than another extraction.
Result: Fable identified checklist-like planner fragments plus cover-at-all-costs low writing as the likely interaction. It proposed cover-or-flag custody and a cheaper challenger: direct writer first, then one post-census coverage audit and localized repair only for uncovered propositions.
Why it worked or failed: The recommendation explains both observed failures: low writing emits weak fragments, while medium writing silently exercises judgment and can drop planned facts. It remains a hypothesis until the bounded challenger is run.
Decision: Build the direct→gap-auditor→localized-repair lane as a post-census challenger; do not move a planner before census.
Next test: Compare that bounded repair lane with v3 direct-medium and planner-low→low on full misses, exact/partial coverage, self-containedness, retrieval-worthiness, redundancy, batch cost, and wall time.

QualityUseful causal hypothesis and test design, not proof that the proposed lane wins.Cost / speed1,494 input / 2,159 output tokens, 37.464s; estimated about $0.1229 at the recorded route rates.

Evidence paths

.harness/gary-local/fable-methodology-consult-20260716/receipt.json
.harness/gary-local/fable-methodology-consult-20260716/answer.md

Jul 16, 2026, 12:37:40 AM ETinvalid / failed

Question

Can native Gemini Batch reproduce the v3 realtime census more cheaply without weakening the output contract?

Gemini 3.1 Pro v3 census through native Batch with strict JSON schema gemini-batch-string-row-recovery

What changed: Replayed the same v3 prompt/schema in Batch and retried only invalid primary units.
Result: All three attempts returned moments as JSON-encoded strings inside moments[] instead of schema objects, so the strict parser marked the rows invalid. The raw strings and receipts are intact; no semantic content has been discarded and no batch census score exists yet.
Why it worked or failed: This is consistent provider transport drift: recoverable row content arrived in the wrong mechanical representation. Rejecting the entire content would violate the preserve-good-content ethos; silently accepting it would hide provenance drift.
Decision: Add deterministic string-row unwrapping followed by the unchanged strict schema/source validator. Mark recovered rows as transport-repaired and ineligible for production promotion until the provider contract is clean.
Next test: Reprocess the saved bytes offline; require every decoded row to validate exact keys, markers, roster, notes, and signals before scoring. No new paid call is needed.

QualityQuality is pending offline recovery and strict revalidation; the current artifact is invalid, not empty.Cost / speedThree provider Batch attempts are preserved. Recovery should add $0 because it operates on immutable saved responses.

Evidence paths

.harness/gary-local/stage2-heldout2-census-durable-eligibility-v3-batch-20260716/transcript-extraction-ablation

Jul 16, 2026, 12:35:01 AM ETproven result

Question

Does the cheaper low writer retain enough meaningful, self-contained atoms to displace medium before adding orchestration?

Direct Luna-low and Luna-medium writers after the exact held-out v3 census heldout-v3-direct-luna

What changed: Held source, 105 located moments, prompt, atom-core contract, cap 32, and native Batch constant; changed only Luna reasoning effort.
Result: Low produced 703 strict atoms for $0.1655 in 249.291s; medium produced 734 for $0.2129 in 220.164s. Both had zero invalid rows, zero cap drops, 100% moment location, and the same 175/178 source ceiling.
Why it worked or failed: The lane receipts prove clean execution and economics, but row count cannot tell whether low found more useful facts or merely wrote fewer details. Medium was unexpectedly faster in these Batch jobs, where queue time is part of wall time.
Decision: Keep both for the paired v3 semantic judge. Do not choose low from price or medium from atom count.
Next test: Judge all 178 facts and every atom; if quality is within 0.2 percentage points, prefer the cheaper/faster practical route.

QualitySemantic recall and atom-quality verdict pending the paired v3 judge.Cost / speedLow $0.1655 / 249.291s; medium $0.2129 / 220.164s. Native Batch queue time makes latency non-monotonic.

Evidence paths

.harness/gary-local/stage2-heldout2-census-v3-luna-low-direct-cap32-batch-20260716/transcript-classifier-ablation
.harness/gary-local/stage2-heldout2-census-v3-luna-medium-control-cap32-batch-20260716/transcript-classifier-ablation

Jul 16, 2026, 12:32:13 AM ETproven result

Question

Did the planner's recall gain produce good retrievable atoms, or only more rows?

All 2,576 emitted atoms from the old held-out census across direct-medium, planner-low→low, and planner-low→medium heldout-old-census-all-atom-quality

What changed: Judged every atom for grounding, precision, self-containedness, atomicity, retrieval worthiness, and duplicate burden instead of relying on counts.
Result: Direct-medium led self-containedness (99.19%) and retrieval-worthiness (97.42%). Planner-low→low reached zero full fact misses but fell to 95.59% self-contained and 94.16% retrieval-worthy. Planner-low→medium recovered quality but fully missed three facts.
Why it worked or failed: The planner creates exhaustive checklist items; low writing often literalizes dependent fragments, while medium writing applies judgment and can silently omit items. Coverage and atom quality are separate obligations.
Decision: Do not promote the zero-miss low-writer lane. Require cover-or-flag custody plus direct-vs-gap-repair comparison.
Next test: Repeat on v3 with all-fact recall first and explicit self-contained/retrieval quality floors; inspect each fully missed fact by stage.

QualityThe planner advantage is real on full misses but not yet acceptable as an end-to-end atom result.Cost / speedLane costs: $0.2053 direct, $0.3177 planner-low→low, $0.3705 planner-low→medium. Separate evaluator spend: $16.7681.

Evidence paths

.harness/gary-local/stage2-heldout2-threeway-all-atom-quality-control-luna-low-luna-medium-20260716/transcript-classifier-ablation

Jul 16, 2026, 12:23:49 AM ETproven result

Question

Did the revised census prompt raise the source ceiling before downstream work?

Gemini 3.1 Pro v3 realtime census on held-out DTCC Investor Flows and JPM Testing Sync heldout-v3-census-realtime

What changed: Used the durable-eligibility v3 prompt and stage-local retry while keeping one primary census plus its recall boost.
Result: The valid result exposed 175/178 meaningful facts (98.31%) and 138/140 load-bearing facts (98.57%) across 105 moments. Three facts remained invisible: DTCC onboarding precondition g8, DTCC flow-detail question g59, and JPM assistance request g35.
Why it worked or failed: The prompt improved visibility over the earlier 169/178 held-out census, but two initially invalid calls were billed before localized repair, inflating artifact cost.
Decision: Use this exact v3 census for downstream comparisons. Repair the three true census misses at the census prompt/boundary layer, never with a pre-census planner.
Next test: Score direct and post-census treatments on the same 175-visible-fact ceiling, while still counting the three census misses in end-to-end recall.

QualityThis is a source-visibility ceiling, not a downstream atom-quality score.Cost / speed$0.8101 artifact cost across six calls; about $0.4136 steady-state successful-call cost. Resumed repair wall: 122.887s.

Evidence paths

.harness/gary-local/stage2-heldout2-census-durable-eligibility-v3-realtime-20260716/transcript-extraction-ablation

Jul 16, 2026, 12:21:28 AM ETcontract only

Question

How do we avoid choosing between unsafe identity merges and deleting valuable-but-unresolved entity mentions?

Entity mention disposition and reversible merge custody entity-tristate-contract

What changed: Added explicit retained_bound, retained_unresolved, and discarded_junk states with exact claim/source hashes, mandatory reasons, reversible merge records, and conservative legacy-null adaptation.
Result: The reusable harness contract and tests pass. Legacy null now maps to retained_unresolved, never silently to junk. Per-row and aggregate custody counts are emitted.
Why it worked or failed: The earlier two-state canonical_map null collapsed ambiguity and junk into one value, making precision gains indistinguishable from recall destruction.
Decision: Adopt tri-state as the target entity contract, but label it contract-only until live AtomDocument, atom_entities, graph, and retrieval paths preserve unresolved mentions.
Next test: Thread an additive unresolved searchable field through a shadow generation, prove projection parity, then compare dangerous merges, valuable retention, junk retrieval, and answer support.

QualityContract proof only; no held-out retrieval lift or tenant reingestion claim yet.Cost / speed$0 provider spend; local contract tests only.

Evidence paths

lib/extraction/eval/transcript-entity-ablation.ts
lib/extraction/eval/transcript-entity-ablation.test.ts
scripts/gary-harness/run-transcript-entity-ablation.ts

Historical result · exact event time not capturedproven result

Question

What did 10 Blockdaemon transcripts, 594 adjudicated facts tell us?

10 Blockdaemon transcripts, 594 adjudicated facts stored-v9-ten-doc-gold

What changed: 10 Blockdaemon transcripts, 594 adjudicated facts
Result: Stored v9 atoms are precise but porous: 0.717 semantic recall, 0.762 load-bearing recall, 0.970 precision, zero hallucinations, and 0.122 duplicate rate.
Why it worked or failed: The dominant opportunity is recovery of omitted load-bearing topics without giving back grounding or fidelity.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Keep as the standing baseline and held-out comparison anchor.

QualityThe dominant opportunity is recovery of omitted load-bearing topics without giving back grounding or fidelity.Cost / speedHistorical baseline; no new provider spend in this session.

Evidence paths

docs/reviews/2026-07-14-atomization-quality-eval.md

Historical result · exact event time not capturedproven result

Question

What did Full old Gemini 3.1 Pro census visibility over the 10-document gold tell us?

Full old Gemini 3.1 Pro census visibility over the 10-document gold ten-doc-census-visibility

What changed: Full old Gemini 3.1 Pro census visibility over the 10-document gold
Result: The census exposes 0.9512 of all gold and 0.9510 of load-bearing gold, while stored downstream recall is 0.717/0.762.
Why it worked or failed: This is a visibility ceiling, not a same-run model-lift estimate; it places the largest headroom downstream of census.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Prioritize proposition custody, classification, repair, and whole-document verification before changing the census default.

QualityThis is a visibility ceiling, not a same-run model-lift estimate; it places the largest headroom downstream of census.Cost / speedReused saved census evidence.

Evidence paths

docs/reviews/2026-07-14-atomization-quality-eval.md

Historical result · exact event time not capturedproven result

Question

What did Three hardest transcripts, fixed Gemini 3.1 Pro census tell us?

Three hardest transcripts, fixed Gemini 3.1 Pro census three-doc-fixed-census-classifiers

What changed: Three hardest transcripts, fixed Gemini 3.1 Pro census
Result: The prior best saved downstream lane reached 0.7573 semantic recall; Fable planning led load-bearing recall at 0.8012.
Why it worked or failed: Useful diagnostic evidence, but insufficient for promotion: the sample is small and the older stage contract did not prove full downstream custody.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Reopen the provisional Gemini 3.5 two-pass staging decision under the new reconstruction and terminal-receipt gates.

QualityUseful diagnostic evidence, but insufficient for promotion: the sample is small and the older stage contract did not prove full downstream custody.Cost / speedSaved realtime lane costs ranged from $0.244 to $2.962 on the three-document corpus.

Evidence paths

docs/reviews/2026-07-14-atomization-quality-eval.md

Historical result · exact event time not capturedcontract only

Question

What did Census, direct providers, review panel, fanout, quality, and reconstruction contracts tell us?

Census, direct providers, review panel, fanout, quality, and reconstruction contracts contract-hardening

What changed: Census, direct providers, review panel, fanout, quality, and reconstruction contracts
Result: Evidence-integrity gates now fail closed on mismatched source/gold/candidate hashes, wrong models, and forged or stale bindings. Repairable transport, parser, and judge-cell failures remain visible for targeted retry instead of erasing a lane.
Why it worked or failed: Execution health, evaluation-ranking eligibility, and production-promotion eligibility are separate decisions. A fully bound experimental lane can be ranked without being shippable.
Decision: Use this as harness capability, not as evidence that a model or production path wins.
Next test: Keep evidence corruption as the true no-go boundary; use paired coverage bounds and explicit promotion blockers everywhere else.

QualityExecution health, evaluation-ranking eligibility, and production-promotion eligibility are separate decisions. A fully bound experimental lane can be ranked without being shippable.Cost / speed$0 provider spend; local tests only.

Evidence paths

lib/llm/response-receipt.ts
scripts/gary-harness/extraction-review-panel.ts
lib/extraction/eval/transcript-classifier-fanout-request-plan.ts

Historical result · exact event time not capturedrunning

Question

What did Two hardest transcripts reviewed by Gemini 3.1 Pro, Sonnet 5, and Fable 5 tell us?

Two hardest transcripts reviewed by Gemini 3.1 Pro, Sonnet 5, and Fable 5 independent-review-panel

What changed: Two hardest transcripts reviewed by Gemini 3.1 Pro, Sonnet 5, and Fable 5
Result: The first paid panel spent at most $2.3017 and produced no contract-valid reviewer result: Gemini returned a useful downstream-loss diagnosis that failed citation and receipt checks, Sonnet exhausted its 16K reasoning budget before a visible answer, and Cloudflare lacked upstream authorization for Fable.
Why it worked or failed: This is harness evidence, not a model-quality ranking. The failures expose three distinct contract, output-budget, and route-authentication defects; no lane is promotable from this run.
Decision: No decision until the running evidence is reconciled.
Next test: Fix receipt propagation and citation coverage, retry Sonnet with a larger output budget, and route Fable through the direct Anthropic API before another bounded panel.

QualityThis is harness evidence, not a model-quality ranking. The failures expose three distinct contract, output-budget, and route-authentication defects; no lane is promotable from this run.Cost / speed$2.3017 accounted upper bound against a $27.2929 authorization ceiling; Gemini $1.1603, Sonnet $1.1414, Fable $0.

Evidence paths

.harness/gary-local/review-hard2-20260715-v6/extraction-review-panel/scorecard.json
scripts/gary-harness/run-extraction-review-panel.ts

Historical result · exact event time not capturedproven result

Question

What did Fresh Gemini 3.1 Pro census on the 122-fact LSEG workshop gold tell us?

Fresh Gemini 3.1 Pro census on the 122-fact LSEG workshop gold fresh-one-doc-census

What changed: Fresh Gemini 3.1 Pro census on the 122-fact LSEG workshop gold
Result: The fixed census exposed 119/122 facts (97.54%) and 73/76 load-bearing facts (96.05%).
Why it worked or failed: This confirms that the urgent loss is downstream on this corpus while preserving the three census misses as real upper-bound constraints.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Hold the census fixed while classifier, entity, retrieval, and synthesis treatments are compared.

QualityThis confirms that the urgent loss is downstream on this corpus while preserving the three census misses as real upper-bound constraints.Cost / speed94,031 input and 12,324 output tokens; no fabricated dollar estimate.

Evidence paths

.harness/gary-local/stage2-fixed-census-20260715/transcript-extraction-ablation

Historical result · exact event time not capturedcontract only

Question

What did Extraction architecture ordering tell us?

Extraction architecture ordering pipeline-order-lock

What changed: Extraction architecture ordering
Result: The architecture is locked to raw source → one exhaustive Gemini 3.1 Pro census with its existing recall boost → deterministic locate/boundary completion → optional downstream proposition planner → writer and verifier. A planner before census would duplicate census rather than clarify downstream custody.
Why it worked or failed: Census misses are repaired at the census prompt, model, or source-boundary layer. The planner under ablation owns decomposition, hedging, attribution, and proposition custody only after census.
Decision: Use this as harness capability, not as evidence that a model or production path wins.
Next test: Keep the census fixed while the leading downstream planner receives packet-local omission repair and is tested on held-out meetings and claims-only reconstruction.

QualityCensus misses are repaired at the census prompt, model, or source-boundary layer. The planner under ablation owns decomposition, hedging, attribution, and proposition custody only after census.Cost / speed$0 provider spend; this is an evidence-driven architecture boundary from the 97.54% LSEG census visibility result.

Evidence paths

lib/extraction/passes/mine-gemini-census.ts
lib/extraction/post-locate-prompts.ts
.harness/gary-local/stage2-fixed-census-20260715/transcript-extraction-ablation

Historical result · exact event time not capturedrunning

Question

What did Fresh LSEG fixed-census atom-core lanes with paired semantic, all-atom quality, and cap-32 replay evidence tell us?

Fresh LSEG fixed-census atom-core lanes with paired semantic, all-atom quality, and cap-32 replay evidence atom-core-uncapped-planner-provisional

What changed: Fresh LSEG fixed-census atom-core lanes with paired semantic, all-atom quality, and cap-32 replay evidence
Result: The final three-way blinded semantic judge scored Luna-medium direct at 103 exact / 18 partial / 1 missed, revised Luna-low planning plus Luna-low writing at 106/16/0, and Terra-low planning plus Luna-low writing at 105/15/2 across the same 122 LSEG facts. The planner result is a real downstream recall lead, not an inference from claim count.
Why it worked or failed: All 2,188 claims were judged. Luna planning produced 755 claims / 713 eligible with 100% grounding, 99.205% precision, 98.808% atomicity, 98.278% retrieval-worthiness, and 12.185% redundant extras. Direct Luna produced 623/579, 100%, 99.518%, 95.345%, 98.555%, and 13.162%; Terra produced 810/692, 100%, 99.259%, 99.383%, 91.728%, and 16.543%. Luna planning is the clean LSEG leader, but remains provisional.
Decision: No decision until the running evidence is reconciled.
Next test: Finish the v4 packet-local planner retry, then compare the repaired Luna finalist against the direct control on the strict held-out census and claims-only reconstruction. Do not promote from the LSEG result alone.

QualityAll 2,188 claims were judged. Luna planning produced 755 claims / 713 eligible with 100% grounding, 99.205% precision, 98.808% atomicity, 98.278% retrieval-worthiness, and 12.185% redundant extras. Direct Luna produced 623/579, 100%, 99.518%, 95.345%, 98.555%, and 13.162%; Terra produced 810/692, 100%, 99.259%, 99.383%, 91.728%, and 16.543%. Luna planning is the clean LSEG leader, but remains provisional.Cost / speedThe revised Luna planner artifact cost $0.277, about 4.8% below the $0.291 live control, versus $0.189 direct and $0.478 Terra. The cap-32 replay dropped no claims and peaked at 19 claims in one moment, but one planner packet omitted required moments and repeat inventories varied, so ceiling removal is insufficient without packet-local repair.

Evidence paths

.harness/gary-local/stage2-atom-core-luna-medium-uncapped-batch-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-atom-core-luna-planner-v2-luna-low-uncapped-batch-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-atom-core-terra-planner-luna-low-uncapped-batch-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-atom-core-uncapped-gold-comparison-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-atom-core-uncapped-quality-comparison-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-planner-final3-semantic-comparison-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-planner-final3-quality-comparison-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-atom-core-luna-planner-v2-luna-low-cap32-batch-20260716/transcript-classifier-ablation

Historical result · exact event time not capturedproven result

Question

What did Sonnet 5 and Grok 4.3 proposition planners after the fixed LSEG census, each followed by Luna-low writing tell us?

Sonnet 5 and Grok 4.3 proposition planners after the fixed LSEG census, each followed by Luna-low writing post-census-foundation-planner-challengers

What changed: Sonnet 5 and Grok 4.3 proposition planners after the fixed LSEG census, each followed by Luna-low writing
Result: Sonnet 5 kept 599 claims for $0.7055 in 480s; Grok 4.3 kept 529 for $0.2734 in 265s. Both missed the qualified DISH registered/registrar-party fact that the revised Luna planner preserved.
Why it worked or failed: Neither challenger displaced Luna on the decisive custody probe, and raw inventory does not justify additional judging. These are post-census planner results only; they do not change the Gemini census architecture.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Retain the artifacts as model-route evidence; spend the next judge budget on repaired Luna held-out downstream and reconstruction rather than broadening the planner shortlist.

QualityNeither challenger displaced Luna on the decisive custody probe, and raw inventory does not justify additional judging. These are post-census planner results only; they do not change the Gemini census architecture.Cost / speedSonnet cost about 2.55× the revised Luna lane and was slower. Grok was close to Luna's cost and faster than the measured Luna batch wall, but returned materially fewer claims and missed the same qualified fact.

Evidence paths

.harness/gary-local/stage2-atom-core-sonnet5-planner-luna-low-uncapped-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-atom-core-grok43-planner-luna-low-uncapped-20260715/transcript-classifier-ablation

Historical result · exact event time not capturedproven result

Question

What did Strict Gemini 3.1 Pro census plus recall boost on held-out JPM Testing Sync and DTCC Investor Flows tell us?

Strict Gemini 3.1 Pro census plus recall boost on held-out JPM Testing Sync and DTCC Investor Flows heldout-two-doc-census

What changed: Strict Gemini 3.1 Pro census plus recall boost on held-out JPM Testing Sync and DTCC Investor Flows
Result: The held-out census exposed 169/178 facts (94.94%) and 133/140 load-bearing facts (95.00%), with zero topics missing all census-span coverage.
Why it worked or failed: Coverage remains high but not perfect, preserving nine real census-bound misses. Both valid primary stages were reused; only the two invalid boost stages were retried, demonstrating bounded stage-local recovery without changing the one-census architecture.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Run the repaired Luna planner and direct control downstream of this exact census, then judge held-out semantic recall and claims-only reconstruction before any promotion decision.

QualityCoverage remains high but not perfect, preserving nine real census-bound misses. Both valid primary stages were reused; only the two invalid boost stages were retried, demonstrating bounded stage-local recovery without changing the one-census architecture.Cost / speed$0.587 total artifact cost including all six primary/boost attempts: four initial attempts plus two localized boost retries.

Evidence paths

.harness/gary-local/stage2-heldout2-fixed-census-realtime-preprocessfix-20260715/transcript-extraction-ablation

Historical result · exact event time not capturedproven result

Question

What did Sol-high, Cloudflare DeepSeek V4 Pro, and GLM 6.3 capability probes tell us?

Sol-high, Cloudflare DeepSeek V4 Pro, and GLM 6.3 capability probes foundation-model-route-probes

What changed: Sol-high, Cloudflare DeepSeek V4 Pro, and GLM 6.3 capability probes
Result: Sol-high kept 551 claims for $1.697 in 1,132s with one invalid row and two cap drops: useful for a hard residual, dominated as the bulk writer. Cloudflare DeepSeek V4 Pro completed all 26 realtime packets, but six hit the 8,192-token ceiling and the recovered artifact is not output-contract-valid. GLM 6.3 returned Cloudflare 404 / code 7003 and is not present in the configured account or catalog; GLM 5.2 was deliberately not substituted.
Why it worked or failed: Sol preserved a difficult qualified attribution, which supports a residual-specialist role after cheap lanes and verification. DeepSeek is transport-limited rather than cleanly model-ranked, so its output cannot be compared as an uncapped quality result.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Use Sol only for verified residual gaps, invalid rows, cap hits, or ambiguous propositions, followed by one re-verification. Do not spend on GLM 6.3 or rank DeepSeek until an uncapped native route exists.

QualitySol preserved a difficult qualified attribution, which supports a residual-specialist role after cheap lanes and verification. DeepSeek is transport-limited rather than cleanly model-ranked, so its output cannot be compared as an uncapped quality result.Cost / speedSol-high cost $1.697. The recovered DeepSeek artifact records about $0.72 and realtime-only latency; there is no native Fireworks/DeepSeek batch credential in this run.

Evidence paths

.harness/gary-local/stage2-atom-core-sol-high-unplanned-batch-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-atom-core-cf-deepseek-v4-pro-realtime-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-cf-glm-6-3-capability-probe-20260715/model-capability-probe.json

Historical result · exact event time not capturedproven result

Question

What did One full transcript; Luna-low versus Gemini 3.1 Pro-high planners with the same Luna-medium writer tell us?

One full transcript; Luna-low versus Gemini 3.1 Pro-high planners with the same Luna-medium writer planner-quality-provisional

What changed: One full transcript; Luna-low versus Gemini 3.1 Pro-high planners with the same Luna-medium writer
Result: Under the final sparse judge, Luna-low produced 607 claims at 97.20% precision and 81.55% canonical eligible quality; Gemini-high produced 635 claims at 96.69% precision and 84.09% eligible quality.
Why it worked or failed: Gemini leads exact recall by 3.28 pp and canonical eligible quality by 2.55 pp, but its duplicate membership is 34.65% versus Luna's 22.90%; redundant extras are 18.90% versus 11.86%. Both lanes tie at 90.13% load-bearing lower-bound recall.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Run entity consolidation and held-out retrieval/synthesis to test whether Gemini's added recall survives as useful answer quality or is outweighed by duplicate noise and cost.

QualityGemini leads exact recall by 3.28 pp and canonical eligible quality by 2.55 pp, but its duplicate membership is 34.65% versus Luna's 22.90%; redundant extras are 18.90% versus 11.86%. Both lanes tie at 90.13% load-bearing lower-bound recall.Cost / speedLuna planner lane: about $1.086 and 190s. Gemini planner lane: about $2.030 and 329s. Luna was about 46% cheaper and 42% faster.

Evidence paths

.harness/gary-local/stage2-planner-quality-judge-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-threeway-semantic-judge-batch-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-threeway-quality-sparse-duplicates-batch-20260715/transcript-classifier-ablation

Historical result · exact event time not capturedcontract only

Question

What did Google native batch adapter and one paid canary tell us?

Google native batch adapter and one paid canary gemini-batch-system-instruction

What changed: Google native batch adapter and one paid canary
Result: The first batch judges were transport-invalid because systemInstruction was sent at the wrong request level. Moving it into config.systemInstruction produced a strict claims-only canary and native STOP provenance.
Why it worked or failed: The broken artifacts are retained as transport evidence and excluded from model-quality scoring.
Decision: Use this as harness capability, not as evidence that a model or production path wins.
Next test: Finish the repaired paired judges and retain native finish reasons in every batch response.

QualityThe broken artifacts are retained as transport evidence and excluded from model-quality scoring.Cost / speedOne minimal paid canary plus batch-priced reruns; no production writes.

Evidence paths

lib/llm/batch-adapters/gemini.ts
lib/llm/batch-adapters/gemini.test.ts

Historical result · exact event time not capturedproven result

Question

What did Current live Gemini 3.5 Flash classifier shape on the fixed one-document census tell us?

Current live Gemini 3.5 Flash classifier shape on the fixed one-document census live-baseline-batch

What changed: Current live Gemini 3.5 Flash classifier shape on the fixed one-document census
Result: The repaired native batch run produced 132 claims from 129 moments with 13/13 strict envelopes and zero invalid rows, but paired judging found only 53 exact, 54 partial, and 15 missed facts out of 122.
Why it worked or failed: The live lane reaches 43.44% exact and 65.57% lower-bound semantic recall, versus 77.87–81.15% exact and 87.70–89.34% lower-bound for the two-pass lanes. The principal measured loss is therefore downstream of census, not at census.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Keep it as the cost/latency control, but do not promote it as the quality path without a treatment that closes the measured recall gap.

QualityThe live lane reaches 43.44% exact and 65.57% lower-bound semantic recall, versus 77.87–81.15% exact and 87.70–89.34% lower-bound for the two-pass lanes. The principal measured loss is therefore downstream of census, not at census.Cost / speed$0.291 measured artifact cost; 164s fresh invocation wall time.

Evidence paths

.harness/gary-local/stage2-live-baseline-batch-systemfix-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-threeway-semantic-judge-batch-20260715/transcript-classifier-ablation

Historical result · exact event time not capturedcontract only

Question

What did Classifier sweep ranking and promotion policy tell us?

Classifier sweep ranking and promotion policy gate-contract-separation

What changed: Classifier sweep ranking and promotion policy
Result: The sweep previously stored separate evaluation and promotion flags but still filtered rankings through production eligibility. Two independent gpt-5.6-sol xhigh audits found and verified boundary defects in reconstruction promotion, request re-certification, missing duplicate units, repair claim universes, returned-model aliases, ambiguous JSON keys, and mixed batch results; all eight concrete findings are now fixed and regression-tested.
Why it worked or failed: Hard no-go is reserved for evidence corruption or ambiguity. Unresolved semantic judge cells now publish as explicit conservative misses and remain rankable; missing, malformed, wrong-model, and ambiguous batch rows are retried locally while independent successes persist. Promotion still requires complete, re-certified evidence.
Decision: Use this as harness capability, not as evidence that a model or production path wins.
Next test: Report execution status, scoring coverage/bounds, and promotion blockers independently in every comparison.

QualityHard no-go is reserved for evidence corruption or ambiguity. Unresolved semantic judge cells now publish as explicit conservative misses and remain rankable; missing, malformed, wrong-model, and ambiguous batch rows are retried locally while independent successes persist. Promotion still requires complete, re-certified evidence.Cost / speed$0 provider spend; local tests only.

Evidence paths

scripts/gary-harness/run-transcript-classifier-sweep.ts
scripts/gary-harness/run-transcript-classifier-sweep.test.ts
scripts/gary-harness/run-transcript-classifier-ablation.ts
scripts/gary-harness/reprocess-transcript-duplicate-evidence.ts
lib/extraction/post-locate-taxonomy.ts
lib/llm/batch-adapters/openai.ts

Historical result · exact event time not capturedproven result

Question

What did Paired Luna-low/Luna-medium batch control versus durability-boundary classifier addenda v1 and v2 tell us?

Paired Luna-low/Luna-medium batch control versus durability-boundary classifier addenda v1 and v2 durability-boundary-prompt-v1

What changed: Paired Luna-low/Luna-medium batch control versus durability-boundary classifier addenda v1 and v2
Result: V1 improved load-bearing recall but lost overall recall. V2 restored overall recall to the control exactly (98 exact, 22 partial, 2 missed) while falling below the control on load-bearing exact recall, 80.26% versus 82.89%. The final quality judge found a real tradeoff rather than a clean rejection.
Why it worked or failed: Against the 601-claim control, v2 produced 645 claims and raised canonical retrieval-worthy quality from 83.53% to 88.84% and eligible quality from 79.53% to 84.19%, while lowering redundant extras from 16.64% to 15.35%. Canonical precision fell from 99.17% to 96.28%, so v2 advances to held-out retrieval/synthesis but does not replace the control yet.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Compare control and v2 through entity consolidation plus held-out retrieval/synthesis; retain v2 only if its larger useful inventory improves answer recall without a material fidelity regression.

QualityAgainst the 601-claim control, v2 produced 645 claims and raised canonical retrieval-worthy quality from 83.53% to 88.84% and eligible quality from 79.53% to 84.19%, while lowering redundant extras from 16.64% to 15.35%. Canonical precision fell from 99.17% to 96.28%, so v2 advances to held-out retrieval/synthesis but does not replace the control yet.Cost / speedControl: $0.5292 and 601 kept claims. V1: $0.5036 and 531 claims. V2: $0.5498 and 645 strict production-compatible claims. All used OpenAI batch pricing.

Evidence paths

scripts/gary-harness/prompt-treatments/classifier-durability-boundary-v1.txt
scripts/gary-harness/prompt-treatments/classifier-durability-boundary-v2.txt
.harness/gary-local/stage2-luna-durability-paired-semantic-batch-20260715/transcript-classifier-ablation
.harness/gary-local/stage2-luna-durability-v2-paired-semantic-batch-20260715/transcript-classifier-ablation

Historical result · exact event time not capturedproven result

Question

What did Sparse three-way quality judgment plus Luna durability-boundary v2 tell us?

Sparse three-way quality judgment plus Luna durability-boundary v2 batch-sweep

What changed: Sparse three-way quality judgment plus Luna durability-boundary v2
Result: The three-way semantic and canonical-quality judges are complete. Gemini-high reached 99 exact, 20 partial, and 3 missed facts; Luna-low reached 95/24/3; live Gemini 3.5 Flash reached 53/54/15. The control-v2 judge also completed all 1,246 canonical and packet claim cells plus two document-wide duplicate units.
Why it worked or failed: Batch remains the default experimental transport. The control-v2 quality judge used 130 requests and completed after one packet repair plus one isolated duplicate repair. Offline reprocessing now derives its expected universe from source documents × candidate lanes, verifies exact request identities and aliases, and cannot silently certify a deleted or narrowed unit.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Move control, v2, Luna-planner, and Gemini-planner candidates into context-rich entity consolidation and held-out retrieval/synthesis evaluation.

QualityBatch remains the default experimental transport. The control-v2 quality judge used 130 requests and completed after one packet repair plus one isolated duplicate repair. Offline reprocessing now derives its expected universe from source documents × candidate lanes, verifies exact request identities and aliases, and cannot silently certify a deleted or narrowed unit.Cost / speedControl-v2 quality judge: $9.3684 at explicitly pinned Google batch rates, 3.523M input tokens, 451K output tokens, and 523K thinking tokens. The expensive evidence is reusable without provider calls.

Evidence paths

.harness/gary-local/stage2-threeway-semantic-judge-batch-20260715
.harness/gary-local/stage2-threeway-quality-sparse-duplicates-batch-20260715
scripts/gary-harness/prompt-treatments/classifier-durability-boundary-v2.txt

Historical result · exact event time not capturedproven result

Question

What did Gemini 3.5 Flash low batch entity consolidation on Luna control and durability-v2 claims, with Gemini 3.1 Pro high blinded full-surface judging tell us?

Gemini 3.5 Flash low batch entity consolidation on Luna control and durability-v2 claims, with Gemini 3.1 Pro high blinded full-surface judging entity-context-ablation

What changed: Gemini 3.5 Flash low batch entity consolidation on Luna control and durability-v2 claims, with Gemini 3.1 Pro high blinded full-surface judging
Result: Plain claim context was not a universal win. On the control claims, production surface-only scored 95.14 versus 90.94 for context; on v2, context scored 93.36 versus 92.33 and reduced dangerous merges from 5 to 2. A stricter context prompt eliminated judged dangerous merges on v2 but dropped 13 valuable entities and its control judge exhausted the shared reasoning/output budget.
Why it worked or failed: Entity behavior depends on upstream claim shape. Context helps disambiguate v2's noisy surfaces but also promotes generic phrases; an over-strict null policy removes dangerous merges at the cost of useful retrieval entities. The current null conflates valuable-but-unresolved mentions with junk, so the target contract is tri-state: retained + bound, retained + unresolved, or discarded as junk. Separate judges disagreed on whether domain phrases such as gas station are generic or named, so these results rank paired maps within a run but cannot alone promote or reject a prompt.
Decision: Keep this result in the comparison, subject to its stated scope and promotion gates.
Next test: Carry the paired maps into held-out retrieval/synthesis and ablate the tri-state mention/binding contract. Keep unresolved valuable surfaces searchable; do not spend more on prompt micro-tuning until answer quality reveals whether false merges, junk, or valuable drops dominate.

QualityEntity behavior depends on upstream claim shape. Context helps disambiguate v2's noisy surfaces but also promotes generic phrases; an over-strict null policy removes dangerous merges at the cost of useful retrieval entities. The current null conflates valuable-but-unresolved mentions with junk, so the target contract is tri-state: retained + bound, retained + unresolved, or discarded as junk. Separate judges disagreed on whether domain phrases such as gas station are generic or named, so these results rank paired maps within a run but cannot alone promote or reject a prompt.Cost / speedFirst complete entity run: $0.8787 total ($0.1539 generation, $0.7248 judge). Conservative follow-up: $0.9245, with one truncated control judge left incomplete and explicitly excluded from promotion.

Evidence paths

.harness/gary-local/stage2-entity-control-v2-batch-20260715/transcript-entity-ablation/entity-e81c2307e7230abd/entity-scorecard.json
.harness/gary-local/stage2-entity-conservative-context-batch-20260715/transcript-entity-ablation/entity-789f0470efaafa91/entity-scorecard.json
scripts/gary-harness/run-transcript-entity-ablation.ts
lib/extraction/eval/transcript-entity-ablation.ts

Where the measured loss lives

The census-to-stored comparison is a diagnostic ceiling, not a causal same-run lift claim.

Census visibility

95.12% Stored atom recall

71.70% 23.42 percentage points of diagnostic downstream headroom

Primary: Maximize end-to-end atom quality, full coverage, retrieval recall, and synthesis fidelity.Near ties: cost and speed decide only around 0.2 pp.Spend: Spend more time or money for material quality lift; use cost and latency only to resolve practical near-ties.

Baseline	Scope	Metrics	Interpretation
Stored v9 baseline`docs/reviews/2026-07-14-atomization-quality-eval.md`	10 transcripts / 594 gold facts	71.7% semantic recall 76.2% load-bearing recall 97.0% precision 12.2% duplicate rate 0 hallucinations	High precision and grounding, but whole load-bearing topics disappear.
Old Gemini census visibility ceiling`docs/reviews/2026-07-14-atomization-quality-eval.md`	Same 10-document gold; diagnostic ceiling	95.12% all-gold visibility 95.10% load-bearing visibility 23.42 pp downstream gap	Census sees most facts; custody is lost during downstream atom construction.

What is live now

Observed from the exact production bundle deployed Jul 15, 2026, 3:01:07 PM ET. This is configuration evidence, not a quality score.

Vercel production deployment dpl_GLSEhcd5dyctJ45huMuLikeWvtm3, built from the local prebuilt bundle at 15:00 ET. No production CENSUS_MODEL or EXTRACTION_CLASSIFY_* override is registered, so code defaults apply.

Lane	Stage	Current implementation
Transcript / meeting	Census + recall boost	Gemini 3.1 Pro Preview over the full marked transcript
Transcript / meeting	Locate	Deterministic utterance-marker span recovery
Transcript / meeting	Classify	Gemini 3.5 Flash, 10 located moments per batch, four concurrent batches; optional planner disabled
Transcript / meeting	Recovery	Retry omitted moments, repair malformed batches with the writer, then escalate failures to Gemini 3.1 Pro
Transcript / meeting	Finalize	Repair, entity consolidation, enrichment, narrative threading, AtomDocument adaptation, then storeExtractionResults()
Document	Census + recall boost	Gemini 3.1 Pro Preview over normalized document blocks
Document	Downstream	Gemini 3 Flash Preview through the shared classify, repair, entity, enrichment, and threading core

Evidence boundaryThe 97.0% figure is historical stored-v9 precision, not a score for this deployed bundle. The deployed transcript locator keeps the full located span for classification while bounding its raw fallback provenance quote to 400 characters; emitted atom text is not capped at 400 characters.

Harness index tree

Every stage names its custody boundary, proof artifacts, and fail-closed gates.

Status	Stage	Purpose	Execution and promotion gates
verified	Gold and source custody`gold_and_source`	Bind raw transcripts, quote-verified gold, roster, and source hashes before comparing lanes.	every gold fact is source-verified no production writes
verified	Census visibility`census`	Run the one exhaustive semantic discovery stage and measure whether it exposes each gold fact before judging downstream loss. A second LLM planner is never placed before census.	strict schema returned model proven clean non-truncated native termination classifier source preprocessing exactly matches census WEBVTT normalization and utterance-marker assignment one census only; improve its prompt, model, or boundary mechanics when source discovery misses remain
in progress	Downstream proposition planner and bounded fanout`planner_and_fanout`	After census and deterministic location, optionally inventory source-grounded propositions, distribute bounded writer work, audit written atoms against the fixed census moments, and fan all normalized ledgers into one whole-document verifier.	exact serialized payload bounds packet-local planner omission and schema repair without rerunning certified packets post-write gap audit is bounded to fixed census moments and repairs only uncovered propositions explicit spend reserve receipt-bound dependency closure one whole-document verifier
in progress	Classify, repair, and atom fidelity`classify_repair_store`	Convert propositions into self-contained, source-supported atoms while preserving conditions, owners, numbers, stance, time, and entity detail.	support and fidelity floors no silent 400-character cutoff complete transformation accounting
verified	Semantic and reconstruction quality`quality_and_reconstruction`	Judge all-fact recall, full misses, atom fidelity, duplication, stage attribution, and whether downstream recall/synthesis can reconstruct the source obligations.	blinded semantic judge deterministic arithmetic all-fact coverage and full-miss minimization are primary; importance and load-bearing cuts remain diagnostic deterministic identity ledger is the lossless reconstruction control; LLM reconstruction remains evaluation-only artifact-bound metrics
verified	Independent model review`independent_review`	Give independent long-context reviewers the census prompt, classifier prompt, source, gold, evidence, and current diagnosis before altering prompts.	no-spend preflight authorization hash native terminal receipts atomic manifest-last persistence
in progress	Transport and durable resume`transport_and_resume`	Use provider batch as the default experimental transport while preserving request-level results, resumable job receipts, and explicit operational-risk diagnostics.	system instructions reach the native provider field one failed request does not erase successful rows idempotent resume and late job-id reconciliation submission crash-window risk blocks unattended promotion, not evaluation
planned	Promotion and tenant reingestion`promotion_and_reingestion`	Promote only on held-out downstream evidence, then reingest Precise and Blockdaemon and converge entities through governed tenant-scoped harnesses.	held-out quality floors tenant dry-run reconciliation retrieval substrate readiness

Promotion gates

Evidence corruption is a hard no-go. Execution and parser defects are repairable. Quality is compared with paired coverage bounds before a separate production decision.

No candidate lane has cleared production promotion.Provisional quality results remain rankable while semantic recall, duplication, retrieval, synthesis, and strict production-contract evidence finish.

evidence integrity · hard no-goContract exact source, gold, candidate, judge, and model-provenance bindings · Evidence: pending or candidate-specific
execution defects · repairableContract retain valid rows; retry only missing, invalid, refused, or truncated cells · Evidence: pending or candidate-specific
semantic recallContract maximize paired exact/lower-bound and load-bearing recall with uncertainty reported · Evidence: pending or candidate-specific
support and fidelityContract no material paired regression; report exact distorted and unsupported counts · Evidence: pending or candidate-specific
retrieval utilityContract Pareto dimension beside recall, not a strategic-only hard filter · Evidence: pending or candidate-specific
duplicate burdenContract minimize without suppressing distinct propositions or load-bearing recall · Evidence: pending or candidate-specific
downstream retrieval and synthesisContract material held-out answer-quality lift before production promotion · Evidence: pending or candidate-specific
production output contractContract strict enums/envelope after targeted repair; separate from evaluation ranking · Evidence: pending or candidate-specific

Models and current rate cards

USD per million tokens. Cloudflare rows include a separately tracked 5% funding surcharge where applicable.

Model	Context	Input	Output	Surcharge	Current experimental role
grok-4.5xai · direct	500,000	$2.00	$6.00	0%	independent reviewer / challenger
gemini-3.5-flashgoogle · direct	1,048,576	$1.50	$9.00	0%	fast writer challenger
claude-sonnet-5anthropic · cloudflare	1,000,000	$2.00	$10.00	5%	independent reviewer / challenger
gemini-3.1-pro-previewgoogle · direct	1,048,576	$2.00	$12.00	0%	census / independent review
claude-fable-5anthropic · cloudflare	1,000,000	$10.00	$50.00	5%	premium planner / reviewer
claude-fable-5anthropic · direct	1,000,000	$10.00	$50.00	0%	premium planner / reviewer

Ideas worth trying

Queued because they could change the quality frontier or prevent a misleading win.

nextDownstream proposition ledger after the single census
Let a reasoning planner decompose fixed, located census moments while preserving hedges, corrections, tentative identity, ownership, and dependencies; it never performs a second source census.
nextRepaired Luna-low planner versus the direct Luna control
Luna v2 leads the paired LSEG semantic and all-atom judges, while Terra's larger inventory is less retrieval-worthy and more redundant. Packet-local retry, held-out downstream scoring, and reconstruction must now test the lead without rerunning certified packets.
laterTune census directly only for confirmed census misses
A planner before census would recursively duplicate source discovery. Broader census challengers remain valid only as direct prompt/model/boundary ablations on held-out meetings.
nextAblate commitments, questions, conditions, and negation independently
Prompt micro-ablations can identify which semantic obligations drive downstream reconstruction rather than changing everything at once.
nextMeasure answer recall and synthesis after atom recall
More atoms only matter when retrieval and synthesis recover the right obligations without contradiction or overload.
guardrailMake batch the default and isolate failures per request/cell
Batch is materially cheaper; recoverable provider or parser defects should not erase valid evidence or disqualify a model.
guardrailBounded Sol residual specialist
Sol-high is too slow and expensive for bulk extraction but preserved a difficult qualified attribution; invoke it only on verifier-identified gaps, cap hits, invalid rows, or ambiguity, then reverify once.

Discovered local evidence

The local build indexes recognized scorecards without embedding raw customer source text. Artifact times are normalized to ET.

Schema	Run	Status	Generated ET	Cost	Path
`konstant-extraction-review-panel-scorecard-v2`	`review-hard2-20260715-v6`	executed	—	$2.3017	`.harness/gary-local/review-hard2-20260715-v6/extraction-review-panel/scorecard.json`
`konstant-extraction-review-panel-scorecard-v2`	`review-preflight-hard2-20260715-v5`	preflight-only	—	$0.0000	`.harness/gary-local/review-preflight-hard2-20260715-v5/extraction-review-panel/scorecard.json`
`konstant-extraction-review-panel-scorecard-v2`	`review-preflight-hard2-20260715-v7`	preflight-only	—	$0.0000	`.harness/gary-local/review-preflight-hard2-20260715-v7/extraction-review-panel/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:11:32 PM ET	$0.2734	`.harness/gary-local/stage2-atom-core-grok43-planner-luna-low-uncapped-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-xai__grok-4.3__low__qpb-5__out-32768__cfg-4ef0dc9ceb29__p-bd955c825b4f__i-8634fc45a25a__lane-b6cf61faca7e/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 9:59:52 PM ET	$0.2356	`.harness/gary-local/stage2-atom-core-luna-low-planned-batch-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-8192__cfg-9b424f08f16c__p-197796b07638__i-8634fc45a25a__lane-159efcdce8fa/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 10:28:54 PM ET	$0.2548	`.harness/gary-local/stage2-atom-core-luna-low-planned-uncapped-batch-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-16384__cfg-cafdae7e4b2c__p-bd955c825b4f__i-8634fc45a25a__lane-8c01ae238fcc/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 9:51:41 PM ET	$0.1350	`.harness/gary-local/stage2-atom-core-luna-low-unplanned-batch-20260715/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__low__contract-atom-core__direct__temp-0.2__batch__cap-8__qpb-5__conc-4__out-8192__cfg-c63add414fbc__p-197796b07638__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 10:26:42 PM ET	$0.1638	`.harness/gary-local/stage2-atom-core-luna-medium-cap16-batch-20260715/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-16__qpb-5__conc-4__out-16384__cfg-b42c4a9ce0e9__p-197796b07638__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 10:30:15 PM ET	$0.1890	`.harness/gary-local/stage2-atom-core-luna-medium-uncapped-batch-20260715/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-16__qpb-5__conc-4__out-16384__cfg-453f2799ce9a__p-bd955c825b4f__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 9:56:43 PM ET	$0.1747	`.harness/gary-local/stage2-atom-core-luna-medium-unplanned-batch-20260715/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-8__qpb-5__conc-4__out-8192__cfg-f9ed0591f957__p-197796b07638__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:24:39 PM ET	$0.2662	`.harness/gary-local/stage2-atom-core-luna-planner-v2-luna-low-cap32-batch-20260716/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-32768__cfg-3550fccbaa52__p-bd955c825b4f__i-8634fc45a25a__lane-68d447fd1e98/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 10:46:36 PM ET	$0.2769	`.harness/gary-local/stage2-atom-core-luna-planner-v2-luna-low-uncapped-batch-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-16384__cfg-c85f027c2bfa__p-bd955c825b4f__i-8634fc45a25a__lane-5148c523dae5/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:31:09 PM ET	$0.2678	`.harness/gary-local/stage2-atom-core-luna-planner-v3-fidelity-cap32-batch-20260716/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-32768__cfg-766b9deae6f3__p-853202b7cee2__i-8634fc45a25a__lane-2065ddef94d4/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:44:19 PM ET	$0.3845	`.harness/gary-local/stage2-atom-core-luna-planner-v4-custody-luna-low-cap32-batch-20260716/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-32768__cfg-fc4910885bb6__p-853202b7cee2__i-8634fc45a25a__lane-bb090e86bb71/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:42:59 PM ET	$0.4231	`.harness/gary-local/stage2-atom-core-luna-planner-v4-custody-luna-medium-cap32-batch-20260716/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-openai__gpt-5.6-luna__low__qpb-5__out-32768__cfg-f83f3ae64fb7__p-853202b7cee2__i-8634fc45a25a__lane-3c053fb76154/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 10:18:56 PM ET	$1.6967	`.harness/gary-local/stage2-atom-core-sol-high-unplanned-batch-20260715/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-sol__high__contract-atom-core__direct__temp-0.2__batch__cap-8__qpb-5__conc-4__out-8192__cfg-13fb64eeca0f__p-197796b07638__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 10:02:14 PM ET	$0.7434	`.harness/gary-local/stage2-atom-core-sol-low-unplanned-batch-20260715/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-sol__low__contract-atom-core__direct__temp-0.2__batch__cap-8__qpb-5__conc-4__out-8192__cfg-79ca4e804d0e__p-197796b07638__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:14:29 PM ET	$0.7055	`.harness/gary-local/stage2-atom-core-sonnet5-planner-luna-low-uncapped-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-anthropic__claude-sonnet-5__low__qpb-5__out-32768__cfg-d4af4dd29968__p-bd955c825b4f__i-8634fc45a25a__lane-88b98ce3eb8d/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 10:42:08 PM ET	$0.4775	`.harness/gary-local/stage2-atom-core-terra-planner-luna-low-uncapped-batch-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-terra__low__qpb-5__out-16384__cfg-a8f2f1c4f64c__p-bd955c825b4f__i-8634fc45a25a__lane-a1bf6439f070/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 4:35:52 PM ET	$2.0298	`.harness/gary-local/stage2-contract-planner-gemini31-high-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-google__gemini-3.1-pro-preview__high__qpb-5__out-16384__cfg-4f369a45d8b6__p-40fabcab2373__i-8634fc45a25a__lane-496e094ffb84/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 4:33:33 PM ET	$1.0862	`.harness/gary-local/stage2-contract-planner-luna-low-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-openai__gpt-5.6-luna__low__qpb-5__out-16384__cfg-f594cc70f907__p-40fabcab2373__i-8634fc45a25a__lane-26ee4d15b5e8/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 4:25:04 PM ET	$2.4236	`.harness/gary-local/stage2-current-qpb10-out16384-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-anthropic__claude-fable-5__low__qpb-10__out-16384__cfg-7b209abe0911__p-397ec2a75aa0__i-8634fc45a25a__lane-01025f4bf906/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 4:26:06 PM ET	$2.7747	`.harness/gary-local/stage2-current-qpb5-out16384-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-anthropic__claude-fable-5__low__qpb-5__out-16384__cfg-5c4544b0eb28__p-397ec2a75aa0__i-8634fc45a25a__lane-d5b0dcecac3f/scorecard.json`
`transcript-entity-ablation-scorecard-v1`	`stage2-entity-conservative-context-batch-20260715`	—	Jul 15, 2026, 7:37:30 PM ET	$0.9245	`.harness/gary-local/stage2-entity-conservative-context-batch-20260715/transcript-entity-ablation/entity-789f0470efaafa91/entity-scorecard.json`
`transcript-entity-ablation-scorecard-v1`	`stage2-entity-control-v2-batch-20260715`	—	Jul 15, 2026, 7:24:30 PM ET	$0.8787	`.harness/gary-local/stage2-entity-control-v2-batch-20260715/transcript-entity-ablation/entity-e81c2307e7230abd/entity-scorecard.json`
`transcript-extraction-ablation-scorecard-v2`	`—`	—	Jul 15, 2026, 4:21:26 PM ET	—	`.harness/gary-local/stage2-fixed-census-20260715/transcript-extraction-ablation/production__google__gemini-3.1-pro-preview__primary-high__boost-medium__realtime__production-census-json-schema-strict-v1__temp-0.3__max-65536__p-be41ab289fc9__b-3e8ba591d198__runner-79bb05f5b2f3/scorecard.json`
`transcript-extraction-ablation-scorecard-v2`	`—`	—	Jul 15, 2026, 11:59:08 PM ET	$0.4698	`.harness/gary-local/stage2-heldout2-census-durable-eligibility-v2-realtime-20260716/transcript-extraction-ablation/production__google__gemini-3.1-pro-preview__primary-high__boost-medium__realtime__production-census-json-schema-strict-v1__temp-0.3__max-65536__p-d36b7079da0d__b-da1c95f83d7d__runner-d66d39a09c0d/scorecard.json`
`transcript-extraction-ablation-scorecard-v2`	`—`	—	Jul 16, 2026, 12:23:49 AM ET	$0.8101	`.harness/gary-local/stage2-heldout2-census-durable-eligibility-v3-realtime-20260716/transcript-extraction-ablation/production__google__gemini-3.1-pro-preview__primary-high__boost-medium__realtime__production-census-json-schema-strict-v1__temp-0.3__max-65536__p-1e700f6987e2__b-89fdba945205__runner-a877e27dc6af/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 12:15:15 AM ET	$0.1979	`.harness/gary-local/stage2-heldout2-census-v2-luna-medium-control-cap32-batch-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-d872f02c4960__p-853202b7cee2__i-d63b46b37de2/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 12:16:14 AM ET	$0.1969	`.harness/gary-local/stage2-heldout2-census-v2-luna-medium-experimental-turn-envelope-cap32-batch-20260716/transcript-classifier-ablation/production__experimental-turn-envelope__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-e02fa2a0128e__p-853202b7cee2__i-d63b46b37de2/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 1:35:02 AM ET	$1.0615	`.harness/gary-local/stage2-heldout2-census-v3-direct-writer-openai-ceiling-gemini-screen-20260716/transcript-classifier-ablation/production__full-span__google__gemini-3.1-pro-preview__high__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-dc537d9ce790__p-853202b7cee2__i-02528b44a1a9/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 2:06:34 AM ET	$2.4826	`.harness/gary-local/stage2-heldout2-census-v3-direct-writer-openai-ceiling-gemini-screen-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-sol__high__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-db0b4d734b2f__p-853202b7cee2__i-02528b44a1a9/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 1:29:10 AM ET	$0.4906	`.harness/gary-local/stage2-heldout2-census-v3-direct-writer-openai-ceiling-gemini-screen-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-terra__low__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-13a66f0b8547__p-853202b7cee2__i-02528b44a1a9/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 1:31:41 AM ET	$0.5115	`.harness/gary-local/stage2-heldout2-census-v3-direct-writer-openai-ceiling-gemini-screen-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-terra__medium__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-1e3ce7644f7a__p-853202b7cee2__i-02528b44a1a9/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 1:31:42 AM ET	$2.0200	`.harness/gary-local/stage2-heldout2-census-v3-direct-writer-openai-ceiling-gemini-screen-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-terra__medium__reasoning-pro__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-946f3a91326a__p-853202b7cee2__i-02528b44a1a9/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 12:35:01 AM ET	$0.1655	`.harness/gary-local/stage2-heldout2-census-v3-luna-low-direct-cap32-batch-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__low__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-66503cdb0665__p-853202b7cee2__i-02528b44a1a9/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 12:29:28 AM ET	$0.2129	`.harness/gary-local/stage2-heldout2-census-v3-luna-medium-control-cap32-batch-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-d872f02c4960__p-853202b7cee2__i-02528b44a1a9/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 1:25:21 AM ET	$0.4243	`.harness/gary-local/stage2-heldout2-census-v3-luna-planner-low-writer-low-cap32-batch-20260716/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-32768__cfg-c9f0edaea1c3__p-853202b7cee2__i-02528b44a1a9__lane-40a38e4726f3/scorecard.json`
`transcript-extraction-ablation-scorecard-v2`	`—`	—	Jul 15, 2026, 11:31:55 PM ET	$0.5872	`.harness/gary-local/stage2-heldout2-fixed-census-realtime-preprocessfix-20260715/transcript-extraction-ablation/production__google__gemini-3.1-pro-preview__primary-high__boost-medium__realtime__production-census-json-schema-strict-v1__temp-0.3__max-65536__p-be41ab289fc9__b-3e8ba591d198__runner-0f1b07dd75d6/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:36:19 PM ET	$0.1837	`.harness/gary-local/stage2-heldout2-luna-medium-control-cap32-batch-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-d872f02c4960__p-853202b7cee2__i-4db1c58305ec/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:46:28 PM ET	$0.2053	`.harness/gary-local/stage2-heldout2-luna-medium-control-preprocessfix-cap32-batch-20260716/transcript-classifier-ablation/production__full-span__openai__gpt-5.6-luna__medium__contract-atom-core__direct__temp-0.2__batch__cap-32__qpb-5__conc-4__out-32768__cfg-d872f02c4960__p-853202b7cee2__i-4db1c58305ec/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 16, 2026, 12:10:26 AM ET	$0.3177	`.harness/gary-local/stage2-heldout2-luna-planner-v4-custody-luna-low-preprocessfix-cap32-batch-20260716/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__low__planner-openai__gpt-5.6-luna__low__qpb-5__out-32768__cfg-fc4910885bb6__p-853202b7cee2__i-4db1c58305ec__lane-772389ecbad4/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 11:50:11 PM ET	$0.3705	`.harness/gary-local/stage2-heldout2-luna-planner-v4-custody-luna-medium-preprocessfix-cap32-batch-20260716/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-openai__gpt-5.6-luna__low__qpb-5__out-32768__cfg-f83f3ae64fb7__p-853202b7cee2__i-4db1c58305ec__lane-11034b76fc5d/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 5:10:10 PM ET	$0.1232	`.harness/gary-local/stage2-live-baseline-batch-20260715/transcript-classifier-ablation/production__production__google__gemini-3.5-flash__low__direct__temp-0.2__batch__cap-8__qpb-10__conc-4__out-8192__cfg-cc7a49796852__p-3ca2c5808846__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 5:20:32 PM ET	$0.2910	`.harness/gary-local/stage2-live-baseline-batch-systemfix-20260715/transcript-classifier-ablation/production__production__google__gemini-3.5-flash__low__direct__temp-0.2__batch__cap-8__qpb-10__conc-4__out-8192__cfg-cc7a49796852__p-3ca2c5808846__i-8634fc45a25a/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 5:51:51 PM ET	$0.5036	`.harness/gary-local/stage2-luna-durability-boundary-batch-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-openai__gpt-5.6-luna__low__qpb-5__out-16384__cfg-c5cda46fa7dd__p-e25483e31944__i-8634fc45a25a__lane-49a6df9a1463/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 6:12:01 PM ET	$0.5498	`.harness/gary-local/stage2-luna-durability-boundary-v2-batch-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-openai__gpt-5.6-luna__low__qpb-5__out-16384__cfg-9714ddb29a61__p-fda01d631f84__i-8634fc45a25a__lane-7cb48ce44c6d/scorecard.json`
`transcript-classifier-ablation-scorecard-v3`	`—`	—	Jul 15, 2026, 5:51:00 PM ET	$0.5292	`.harness/gary-local/stage2-luna-two-pass-batch-control-20260715/transcript-classifier-ablation/two-pass-moment-plan__full-span__openai__gpt-5.6-luna__medium__planner-openai__gpt-5.6-luna__low__qpb-5__out-16384__cfg-2033cde34861__p-40fabcab2373__i-8634fc45a25a__lane-1b20bbaa191e/scorecard.json`

Operator commands

Provider spend and production writes are explicit properties of every entrypoint.

catalognpm run harness:gary:extraction-ablation:lab -- --run-id current: Refresh this index tree and local HTML experiment journal.Provider spend: none · Production writes: none
censusnpm run harness:gary:transcript-extraction -- <explicit lane args>: Run or score strict census lanes.Provider spend: possible · Production writes: none
classifiernpm run harness:gary:transcript-classifier -- <explicit lane args>: Run fixed-census downstream lanes and quality judges.Provider spend: possible · Production writes: none
sweepnpm run harness:gary:transcript-classifier:sweep -- <explicit matrix args>: Compare bounded planner/writer/prompt lanes.Provider spend: possible · Production writes: none
entitynpm run harness:gary:transcript-entity -- <candidate lanes and explicit model args>: Compare production and context-rich entity maps with full-surface blinded quality evidence.Provider spend: possible · Production writes: none
review-preflightnpx tsx scripts/gary-harness/run-extraction-review-panel.ts <typed inputs and matrix>: Emit the authorization hash and maximum planned spend without calls.Provider spend: none · Production writes: none
system-treenpm run harness:gary:system-tree -- --run-id current: Refresh the repo-wide Gary capability and evidence tree.Provider spend: none · Production writes: none

Help steer this

Which v3 downstream branch should get the next paid judge budget?

Where should the retrieval follow-up isolate loss first?

How strict should the first entity canary be?

The map we use to make the next decision

01 · Frozen map

02 · Candidate sticks

03 · Independent courts

04 · Route decision

What happened

Is Sol-high's 1.97-point semantic lift made of faithful, self-contained, atomic, retrieval-worthy information, or costly duplication and fragmentation?

Does Sol-high's 53.5% larger atom inventory preserve materially more facts than Luna-medium without relying on side-by-side judge context?

Can a stronger OpenAI writer extract materially more useful atoms than Luna-medium at an acceptable quality and cost?

Which v3 lane preserves the most meaningful facts, and does the planner's larger inventory actually close full misses?

Can the planner preserve the zero-full-miss advantage without repeating its self-containedness and retrieval-quality losses?

Can a changed lane directory safely reuse an already-paid Batch receipt when the exact 22-request manifest is identical?

Does preserving full atom text in an otherwise identical retrieval packet improve the final CEO answer?

Can a cheaper large-context Luna census match Gemini v3 visibility under the exact strict census contract?

Why does the cheap planner improve coverage while degrading self-contained, retrieval-worthy atom quality, and what is the highest-information next test?

Can native Gemini Batch reproduce the v3 realtime census more cheaply without weakening the output contract?

Does the cheaper low writer retain enough meaningful, self-contained atoms to displace medium before adding orchestration?

Did the planner's recall gain produce good retrievable atoms, or only more rows?

Did the revised census prompt raise the source ceiling before downstream work?

How do we avoid choosing between unsafe identity merges and deleting valuable-but-unresolved entity mentions?

What did 10 Blockdaemon transcripts, 594 adjudicated facts tell us?

What did Full old Gemini 3.1 Pro census visibility over the 10-document gold tell us?

What did Three hardest transcripts, fixed Gemini 3.1 Pro census tell us?

What did Census, direct providers, review panel, fanout, quality, and reconstruction contracts tell us?

What did Two hardest transcripts reviewed by Gemini 3.1 Pro, Sonnet 5, and Fable 5 tell us?

What did Fresh Gemini 3.1 Pro census on the 122-fact LSEG workshop gold tell us?

What did Extraction architecture ordering tell us?

What did Fresh LSEG fixed-census atom-core lanes with paired semantic, all-atom quality, and cap-32 replay evidence tell us?

What did Sonnet 5 and Grok 4.3 proposition planners after the fixed LSEG census, each followed by Luna-low writing tell us?

What did Strict Gemini 3.1 Pro census plus recall boost on held-out JPM Testing Sync and DTCC Investor Flows tell us?

What did Sol-high, Cloudflare DeepSeek V4 Pro, and GLM 6.3 capability probes tell us?

What did One full transcript; Luna-low versus Gemini 3.1 Pro-high planners with the same Luna-medium writer tell us?

What did Google native batch adapter and one paid canary tell us?

What did Current live Gemini 3.5 Flash classifier shape on the fixed one-document census tell us?

What did Classifier sweep ranking and promotion policy tell us?

What did Paired Luna-low/Luna-medium batch control versus durability-boundary classifier addenda v1 and v2 tell us?

What did Sparse three-way quality judgment plus Luna durability-boundary v2 tell us?

What did Gemini 3.5 Flash low batch entity consolidation on Luna control and durability-v2 claims, with Gemini 3.1 Pro high blinded full-surface judging tell us?

Where the measured loss lives

What is live now

Harness index tree

Promotion gates

Models and current rate cards

Ideas worth trying

Discovered local evidence

Operator commands