Testing Every Possible Way to Be Wrong: CSCV for Strategy Evaluation
Combinatorially Symmetric Cross-Validation testing all possible IS/OOS combinations, producing foundation for PBO=0.112 calculation.
What CSCV Does Differently
Standard cross-validation tests a few IS/OOS splits. CSCV tests all of them. Given S subsets of data, CSCV generates C(S, S/2) combinations, each providing an independent IS/OOS split. With S=16 subsets, that is 12,870 unique ways to divide the data into training and test sets.
For each combination, the strategy is optimized on the IS portion and evaluated on the OOS portion. The key metric is the rank correlation between IS and OOS performance. If IS-optimal parameters also perform well OOS, the strategy has genuine edge. If IS-optimal parameters underperform OOS, the strategy is overfit.
The 12,870 Test Marathon
Running 12,870 full strategy evaluations is computationally expensive. S43 parallelizes the computation across available cores and uses cached model outputs to avoid redundant training. A full CSCV run takes approximately 8 hours on a 12-core machine.
The results feed directly into S21's PBO calculation. Each combination produces a binary result: did IS optimization transfer to OOS? The fraction of combinations where it did not transfer is the PBO estimate. With 12,870 data points, the PBO estimate has tight confidence intervals (approximately plus or minus 0.01).
CSCV as the Ultimate Overfitting Test
CSCV is the most rigorous test for backtest overfitting available in quantitative finance. It is computationally expensive, methodologically sound, and produces a single interpretable number (PBO). V7's PBO of 0.112 means that 88.8% of all possible ways to split the data confirm that the strategy's edge is real.
The 11.2% of combinations that did not transfer are informative too. Analysis of these "failed" combinations shows they predominantly involve IS partitions missing the 2020 COVID crash. Models trained without exposure to extreme volatility understandably fail when tested on extreme volatility. This is expected behavior, not a flaw. CSCV provides the foundation for confident deployment. When you have tested your strategy 12,870 different ways and 88.8% confirm genuine edge, you can trade live with a level of statistical confidence that no simpler validation method can provide.