Walk-forward (the thing that makes comparisons mean anything)
python wf.py --bars data/bars_10y.csv
Testing ten strategies on one window and keeping the best is how a backtest
lies. The winner is whichever one best fit that window's noise. walkforward.py
selects on a training window and measures on the next window, so nothing is
ever scored on the data that chose it.
The output to read is not the winner's return. It is:
| diagnostic | means |
|---|---|
selection hit rate |
how often the in-sample winner beat the median next fold; 50% is a coin flip |
rank correlation |
Spearman, in-sample vs out-of-sample rank; 0 = last window says nothing |
average candidate |
equal weight across all candidates — the bar selection must beat |
always X |
committing to one strategy and never re-choosing |
perfect hindsight |
upper bound, not achievable |
Result on 2016–2026, 14 folds, six candidates: selection did not work.
Hit rate 36%, rank correlation +0.04, and walk-forward selection returned
+53.2% against +50.1% for simply averaging every candidate. Committing to one
strategy and never re-choosing (always trend-daily, +115.4%) beat adaptive
selection by 62 points, and buy-and-hold SPY beat everything at +184%.
Four configurations were tried (rolling/anchored × return/Sharpe). The best, anchored-on-return, showed a 64% hit rate — which is 9 of 14 folds, p=0.21 alone and p=0.61 after accounting for having tried four. That is not evidence, and searching configurations for the best hit rate is the same error one level up. The script now prints that p-value and says so.
Candidates carry their own RiskLimits, because scoring a strategy under
limits that forbid its trades measures the limits (see TRENDCASH_LIMITS).
Strategies are passed as factories, never instances — they carry state across a
run, and reusing one silently corrupts every fold after the first.