Lab notebook · chapter 4 of 7 · page 4 of 4

LLM agents continued

Result: the veto lost money

463 rebalances, zero model failures, $9.48. Full output in results/veto_ab_2026-09-08.txt.

return maxDD Sharpe turnover
momentum +361.7% 27.5% 1.04 29.8x
momentum + veto +249.8% 24.4% 0.99 29.3x
buy-and-hold SPY +262.7%
equal-weight 111 +324.3%

−111.9 points. It bought a 3.1-point drawdown reduction for a third of the return, and Sharpe fell. The vetoed strategy underperforms both benchmarks; the unvetoed one beat them.

Two diagnostics say why, and neither is "the model was unlucky".

It vetoed the wrong names. Over the following 5 sessions the vetoed candidates returned +0.633% against +0.343% for the kept ones — the veto systematically removed the better half (t = +1.38, so not significant, but pointing the wrong way). Its stated reasons were sound-sounding — high realised volatility, negative longer-horizon returns — and in this universe those were the names that went up.

It never adapted. The veto rate sat between 16.7% and 21.2% every single year, with 2022 (21.2%) indistinguishable from 2021 (16.7%). It was sold as regime awareness and delivered a flat ~19% haircut — which a rule could apply for free, and which is exactly the failure mode the design was meant to avoid.

The one thing it did do is cut the drawdown, which is what indiscriminately holding back ~19% of buying does.