Lab Notebook

Metric evolution, seed variance, and experimental verdicts for the depth-limited search layer. Every number is traced to a commit or a committed log (hover the ⎇ ref chips).

Why gates, seeds, and CIs

This project has a documented history of confident corrections invalidated by data — a distilled network that regressed head-to-head, richer card abstractions that only looked worse because of sample starvation, three independent “zero” results for river re-solving. So every mechanism is gated: it must beat the baseline on a measured, seed-replicated metric before it is kept.

The load-bearing lesson is seed variance. The same strategy measures anywhere from 544 to 1308 mbb/decision across seeds. A single-seed number is one draw, not a level — so verdicts use pooled means with CIs, and paired (CRN) deltas for A/B, never a lone seed.

Metric evolution — LBR proxy by blueprint

Comparability caveat: LBR is a lower-bound proxy, not true exploitability, and it is notcomparable across action-set sizes (a wider tree gives the exploiter a higher max) or across metric generations. Bars are shaded by role; compare only within a tree/metric group. Whiskers are ±1 stderr.

25050075010001250v14d4-action (single raise)1384 ±120v15_c2_v5_aux4-action + advisor cache761 ±88v16a4-action (single raise, no cache)1194 ±177v16b6-action (2 raise sizes, no cache)944 ±138v16c6-action (2 raise sizes + cache)753 ±118v16c6-action (2 raise sizes + cache)998 ±26v16e6-slot capacity (3 raise sizes + cache)1116 ±50
production superseded rejected (a result, not a gap)

Metric generations

Clairvoyant regret ▤ project_exploitability_rewrite

best-response saw the villain’s exact cards → measured deal variance, not strategy quality. Disagreed with head-to-head results.

LBR v1 ⎇ 92f41ce

opponent hand marginalised uniformly. Correct far from equilibrium, but floors near it — cannot separate a 0.22 strategy from a 0.08 one on Leduc.

LBR v2 ⎇ 92f41ce

Bayes (reach-weighted) beliefs + stratified depth sampling + per-node samples for correct CRN-paired SE. Ordering restored (r=0.996 vs exact on Leduc).

CFV network generations — river-value holdout MAE

The ReBeL on-policy loop: the value net must be trained on the PBSs the depth-limited solver actually queries, not just blueprint self-play. The on-policy MAE (red) started 2.9× the base MAE and closed with each retraining generation.

0.000.050.100.15b2b4b5on-policybaseMAE (pot units) · lower is better
netdatabase MAEon-policy MAEnotesrc
b25k base0.0460.134first river net; on-policy 2.9× base — solver-iterate ranges are sharper⎇ 5920077
b415k base + 4.5k on-policy0.0400.068ReBeL on-policy retraining halves the on-policy gap⎇ acff67d
b525k base + 10.5k on-policy0.0380.062PRODUCTION CFV net; more data, best aggregate MAE▤ gate_b4_vs_b5.log

Seed variance — the gate, replicated

The search layer’s gate: whole-strategy LBR of the depth-limited agent minus the v16c blueprint, CRN-paired per seed. Below Δ = 0 means the search agent is less exploitable. The result is replicated across two independent depth-sampling schemes — the discipline the whole page is about.

uniform depth sampling · 20 seeds · b4 netΔ = -36.3 ± 16.4 · t = -2.22 · 13/20 negative ▤ gate_uniform_20seed.txt
Δ = 0 (no gain)mean -36.3
stratified · b5 netΔ = -30.2 ± 27.6 · t = -1.1
Δ = 0 (no gain)mean -30.2
stratified · b4 netΔ = -32.6 ± 26.7 · t = -1.22
Δ = 0 (no gain)mean -32.6

Stratified sampling (stratified depth sampling · seeds 42–45 · ~28 override nodes/seed) lifts turn/river nodes from ~8 to ~28 per seed, tightening the paired SE. Both nets land at Δ ≈ −31, consistent with the uniform 20-seed −36.3 ± 16.4. ▤ gate_b4_vs_b5.log

Decision log

verdictmeasured basiswhysrc
Unsafe re-solving kept (no gadget)Leduc Phase 1: agent expl 1.365 < blueprint 1.562 (Δ −0.197)unsafe re-solving already lowered exploitability; the safe gadget was only to be added if unsafe raised it — it did not.⎇ 6caadad
LBR metric: Bayes beliefs over uniformLeduc: uniform belief floors near equilibrium (exact 0.22 & 0.08 both read ~0.31, order inverts); Bayes r=0.996uniform opponent belief cannot separate good from great near equilibrium; reach-weighting restores ordering.⎇ 92f41ce
River-only search abandonedper-river-node Δ = +0.008 ± 0.121 (z=0.07); third independent zero4-action river decisions are already at the blueprint ceiling; the win had to come from turn + real ranges.⎇ 70e8436
2 raise sizes over 1 (v16b→v16c)LBR v1 944±138 (2 sizes) vs 1194±177 (single) @2500 itersricher action tree measurably lowered exploitability at equal budget.⎇ 91d2d02
3 raise sizes REJECTED (v16e)LBR v2 1116±50 (3 sizes) vs 998±26 (2 sizes), 5 seedsa still-richer tree lost — sample starvation at this budget outweighs the finer action grid. @5000-iter retest deferred.▤ gate_b4_vs_b5.log
b5 CFV net chosen over b4gate tie (b5 Δ −30.2±27.6, b4 −32.6±26.7); b5 better aggregate MAEgates were a statistical tie; b5 wins the tiebreak on more data + lower holdout MAE.▤ gate_b4_vs_b5.log
Depth-limited turn+river search beats blueprintuniform 20-seed Δ −36.3±16.4 (t=−2.22); stratified 4-seed ≈ −31the gate passes and replicates across two independent sampling methods — the search layer is a real, measured improvement.▤ gate_uniform_20seed.txt