PBO 0.112: The One Number That Keeps Me Honest

What PBO Means

Probability of Backtest Overfitting (PBO) answers a terrifying question: what is the probability that your strategy's good backtest performance is the result of overfitting rather than genuine edge? A PBO of 0.5 means it is a coin flip. A PBO of 0.0 means you definitely have real edge. A PBO of 1.0 means you definitely overfit.

V7's PBO is 0.112. That means there is an 11.2% chance that the backtest results are the product of overfitting. Put differently, there is an 88.8% chance that the observed edge is real. In quantitative finance, PBO below 0.20 is considered strong evidence against overfitting.

How CSCV Calculates It

PBO uses Combinatorially Symmetric Cross-Validation (CSCV), which is the most rigorous overfitting test available. The process splits the backtest data into S subsets, then tests every possible combination of IS (in-sample) and OOS (out-of-sample) partitions. For each combination, it checks whether the IS-optimal strategy also performs well OOS.

S21 uses S=16 subsets, generating 12,870 unique IS/OOS combinations. Each combination provides one data point: did IS optimization transfer to OOS? The fraction of combinations where it did not transfer is the PBO estimate. With 12,870 tests, the estimate is statistically robust.

Why 0.112 Matters

PBO 0.112 means that across 12,870 ways to slice the data, 88.8% of the time the strategy that looked best in-sample also performed well out-of-sample. This is strong evidence that V7's edge is genuine. The 11.2% that did not transfer mostly came from partitions containing the 2020 COVID crash, which was a genuinely anomalous regime that no model trained on pre-2020 data could handle well. If you exclude the COVID period, PBO drops to 0.07. I report 0.112 because honest evaluation includes the hard parts. PBO is the single most important validation metric in the system because it directly addresses the fundamental question every systematic trader must answer: is this real?