Do the numbers hold up on the real market?

A backtest tool is only worth trusting if its arithmetic is exactly right. The first test ignores the Desk's own code entirely: a second backtester was written from scratch in plain Python — separate entry logic, separate P&L, separate cost model — and pointed at the same 13,152 real BTC hourly candles. If the two disagree by a cent, the engine is suspect.

The worst disagreement across all five rules was 0.000000 percentage points, with identical trade counts. Reconciling the two required pinning down three exact behaviours the independent version had to replicate — entries priced at the candle close, exits exactly hold candles later, one position at a time with re-entry blocked until the candle after an exit, and any position still open at the window's end force-closed on the final candle. Once those matched the engine's, the numbers were bit-identical. The Desk reports precisely what it does.

A correct engine can still flatter a bad idea if it only ever reports one number. The Desk's real job for a training team is the opposite: to say “this looked good, but it won't repeat.” Three of its rigor tools were run on real data to check they do exactly that.

Walk-forward — was the edge consistent, or only on average?

Each rule is re-run on the earlier 60% and the later 40% of the window separately, and its beat-the-random-baselines score is compared across the two halves. A number that is strong early and weak late is flagged as overfit.

The ETH momentum rule beat 7 of 10 baselines early and only 2 of 10 late — correctly labelled overfit. No rule here beat a passive buy-and-hold over the full window; the Desk says so plainly rather than cherry-picking the flattering half.

The 36-variant sweep — does the best-of-many survive?

This is the trap every backtester falls into: try 36 variations, keep the best, and be fooled by the luckiest one. The Desk's haircut takes that same in-sample winner and re-scores it out of sample against the no-skill line (a coin flip beats ~5 of 10 baselines by chance).

Bootstrap & regime — is it luck, and where did it work?

Two more read-outs on a 247-trade dip-buyer. A Monte-Carlo bootstrap resamples the actual trades 1,000 times to build a confidence band; a regime split attributes the result to the market state at each entry — and, as a correctness check on itself, the three buckets must sum back to the headline exactly.

In-sample and walk-forward tests still touch data the rule was chosen on. The only fully honest check is out-of-time. So the best-of-36 dip-buyer was frozen on the seen data, and then — with its parameters locked — measured on 5,467 fresh BTC candles fetched live from Binance for the window right after the training data ends. No re-tuning, no second look.

The rule that looked like a +27% winner in the backtest lost −7.6% on the future it never trained on, and matched only 5 of 10 random baselines — the coin-flip line. It “beat” buy-and-hold's −26.5% only by sitting in cash while BTC fell (9 trades in seven months), not by any demonstrated skill. That is not a disappointment — it is the whole point.