Thirty-One Signals, Six Bets · part 8 of 10

§ 5

Errors found, and why they are the evidence

A null result is only worth reading if the apparatus that produced it was capable of producing a positive one. The strongest available evidence for that is the list of errors the harness caught — several of which would have manufactured an edge had they survived.

  1. 01Split double-adjustment, wrong by 40× Yahoo's quote values are already split-adjusted. Applying a split factor divided NVDA's pre-split history by 40 twice. Caught by a sanity check requiring split-only history to sit above dividend-adjusted history; a regression test now fails if the committed file ever shows a split-shaped gap.
  2. 02The test suite silently ran 110 of 141 tests An unterminated f-string meant a module would not import, so unittest discover collected a subset — and still printed OK.
  3. 03A commit landed with failing tests A tail in a shell pipeline masked the exit code. Commits are now gated on the suite's actual status.
  4. 04Preflight reported "all checks passed" on an API that could not place orders Read-only mode was enabled. An explicit order-permission probe was added.
  5. 05A paper-only guard that guarded nothing The port check was described as one of three independent safeguards; the gateway in fact served live accounts on that port. The account-prefix assertion is the only load-bearing check, and the documentation was corrected to say so.
  6. 06Path divergence invalidated a cost sweep Raising the assumed half-spread increased measured excess, because one changed fill changes every later position. Cost sensitivity was rebuilt to hold order flow fixed.
  7. 07A recorder double-called a stateful strategy This desynchronised the rebalance cadence and silently produced an empty sample.
  8. 08The market-neutral book went $103k net short on $100k Dollar neutrality decays asymmetrically — a losing short grows as a share of the book while a losing long shrinks. NAV reached −$12,091 because the simulator had no margin logic at all. Both were fixed; forced liquidations are now recorded, and any non-zero count means the run hit a wall a real account would have hit.
  9. 09Limits measuring themselves, five or six times Scoring a strategy under limits that forbid its trades measures the limits. Each occurrence is documented in place.
  10. 10An overstated claim, corrected by replication The combination-lift agreement was reported as accurate "within 6%" and called the cleanest result in the project. The small-cap run showed the statistic is unstable. See §3.3.

Items 01, 06, 07 and 08 would each have produced a flattering number. Item 10 is included because the correction arrived from a replication we had already committed to running, not from a reviewer.