Thirty-One Signals, Six Bets · part 2 of 10
Each was run because the previous one left a specific objection standing. They are reported in the order they were run, including the ones that failed for reasons we had not anticipated.
The first agent, Scout, saw price-derived features for the whole universe and returned names to hold. Its prompt deliberately carried returns and volatility only — no sector, no market capitalisation, no company names — so that any drift toward mega-caps would be attributable to the model's priors rather than to the prompt. A test asserts none of those fields leak into the rendered table.
It did not need a full backtest to fail. Every visible pick sat inside the momentum strategy's own top 20, because four of its five inputs were momentum features. An agent can only express what its inputs allow. This is a design result, not a performance result, and it is the reason the second design exists.
The second design wraps any strategy: the strategy proposes, the agent may drop some buys, and it may do nothing else. It cannot add a name, resize one, or veto a sell. This is cleanly measurable (identical order flow with and without), has bounded downside (the worst it can do is leave you in cash), and asks something a cross-sectional rank structurally cannot answer — its prompt carries market context: index return, breadth, median volatility.
463 rebalances, zero model failures, $9.48 of API spend.
| Book | Return | Max DD | Sharpe | Turnover |
|---|---|---|---|---|
| Momentum | +361.7% | 27.5% | 1.04 | 29.8× |
| Momentum + LLM veto | +249.8% | 24.4% | 0.99 | 29.3× |
| Buy-and-hold SPY | +262.7% | — | — | — |
| Equal-weight, same 111 names | +324.3% | — | — | — |
Two diagnostics say why, and neither is "the model was unlucky." It vetoed the wrong names: over the following five sessions the vetoed candidates returned +0.633% against +0.343% for the kept ones — it systematically removed the better half (t = +1.38, not significant, but pointing the wrong way). It never adapted: the veto rate sat between 16.7% and 21.2% in every single year, with 2022 indistinguishable from 2021. Sold as regime awareness, delivered as a flat ~19% haircut that a rule could apply for free.
Choosing among strategies on a training window and measuring on the next one. Fourteen folds, six candidates, primary configuration pre-registered as rolling/return before running.
| Outcome | Rolling (primary) | Anchored (check) |
|---|---|---|
| Walk-forward selection | +100.2% | +283.3% |
| Average of all candidates | +114.5% | +117.6% |
| Commit to one momentum variant | +287.0% | +298.0% |
| Buy-and-hold SPY | +185.4% | +185.4% |
| Selection hit rate | 36% | 71% |
| In-sample / out-of-sample rank corr. | +0.11 | +0.39 |
Selection did worse than a coin flip and worse than averaging every candidate. In 0 of 14 folds did it pick the actual best one; in the last three it had rotated into the strategy that won folds 8–10, exactly as momentum won the next three. That is performance chasing with extra steps.
The anchored column disagrees and there is a mechanism for it — training on all history produces a more stable ranking (churn 15% vs 54%). It is reported as a hypothesis, not a result: p = 0.09 alone, ≈0.17 across two configurations, and adopting it because it scored better is precisely the error this design exists to prevent.