Backtesting Pitfalls: Why Your Strategy Isn't as Good as You Think

1. Overfitting: Fitting Noise, Not Signal

The most dangerous pitfall. If your strategy has too many parameters relative to the data, it will fit the random noise in historical data perfectly. It will look incredible in-sample and fail immediately out-of-sample. A moving average crossover with 2 parameters is less prone to overfitting than a neural network with 10,000 parameters.

Rule of thumb: if your strategy has N free parameters, you need at least 10N to 20N independent data points for any statistical significance. A strategy with 5 parameters needs at least 50-100 trades to be meaningful.

def demonstrate_overfitting(prices, n_strategies=1000):
    """
    Show how testing many strategies on the same data
    guarantees finding one that "works" by pure chance.
    """
    returns = prices.pct_change().dropna()
    n = len(returns)

    best_sharpe = -np.inf
    best_params = None

    for i in range(n_strategies):
        # Random strategy: buy when MA crosses
        fast = np.random.randint(5, 50)
        slow = np.random.randint(50, 200)

        fast_ma = prices.rolling(fast).mean()
        slow_ma = prices.rolling(slow).mean()

        signal = (fast_ma > slow_ma).astype(int).shift(1)
        strat_returns = signal * returns

        sharpe = strat_returns.mean() / strat_returns.std() * np.sqrt(252)

        if sharpe > best_sharpe:
            best_sharpe = sharpe
            best_params = (fast, slow)

    print(f"Tested {n_strategies} random strategies")
    print(f"Best in-sample Sharpe: {best_sharpe:.2f}")
    print(f"Best params: MA({best_params[0]}, {best_params[1]})")
    print(f"
This Sharpe is MEANINGLESS - it's the result of data mining.")
    print(f"Expected best Sharpe from {n_strategies} random trials:")
    print(f"  ~{np.sqrt(2 * np.log(n_strategies)):.2f} (even with random data)")

2. Look-Ahead Bias

Using information that would not have been available at the time of the trade. This is surprisingly easy to introduce accidentally. Common examples: using closing prices to make decisions at the open, using revised economic data (GDP is revised months later), or calculating a signal that includes today's data to make today's trade.

The fix: always .shift(1) your signals. If your signal uses data from day T, the earliest you can trade is day T+1. Be paranoid about this.

3. Survivorship Bias

If you test your strategy on today's FTSE 100 constituents, you are only testing on companies that survived. You exclude all the companies that went bankrupt, were delisted, or were acquired. These are precisely the stocks that would have generated the largest losses. Your backtest will be systematically too optimistic.

The fix: use point-in-time constituent lists. Know which stocks were in the index on each historical date, not just today. This data is expensive and hard to get, which is why many retail backtests are unreliable.

4. Transaction Costs

Spread, commission, slippage, and market impact are the silent killers of trading strategies. A strategy that trades 10 times a day incurs costs 10 times. A high-frequency strategy with a Sharpe of 3 before costs might have a Sharpe of -1 after costs. Always model costs explicitly.

def backtest_with_costs(prices, signal, spread_bps=10,
                       commission_bps=5, slippage_bps=5):
    """
    Realistic backtest including transaction costs.
    Many strategies that look profitable before costs
    are unprofitable after.
    """
    returns = prices.pct_change().dropna()
    signal = signal.reindex(returns.index).fillna(0)

    # Identify trades (when signal changes)
    trades = signal.diff().abs()
    total_cost_bps = spread_bps + commission_bps + slippage_bps

    # Strategy returns before costs
    gross_returns = signal.shift(1) * returns

    # Transaction costs
    costs = trades * total_cost_bps / 10000

    # Net returns
    net_returns = gross_returns - costs

    print(f"Gross Sharpe: {gross_returns.mean()/gross_returns.std()*np.sqrt(252):.2f}")
    print(f"Net Sharpe:   {net_returns.mean()/net_returns.std()*np.sqrt(252):.2f}")
    print(f"Total trades: {trades.sum():.0f}")
    print(f"Total costs:  {costs.sum():.2%}")
    print(f"Gross return: {gross_returns.sum():.2%}")
    print(f"Net return:   {net_returns.sum():.2%}")

5. Data Snooping

Every time you look at the results and tweak your strategy, you are implicitly fitting to the data. If you test 100 variations and pick the best one, that is data mining. The expected Sharpe of the best strategy out of N random strategies is approximately √(2 ln N), even if all strategies are worthless. With 1,000 random strategies, you expect the best to have a Sharpe around 3.7 — purely by chance.

The fix: set aside a test set that you never look at until the very end. Use it once. If the strategy works out-of-sample, it has a chance of being real.

6. Walk-Forward Analysis: The Right Way

Walk-forward testing is the gold standard for strategy validation. You train on a rolling window of historical data, test on the next period, then roll forward. This simulates what would actually happen in production: you fit your model on past data and trade on future data.

import numpy as np
import pandas as pd

def walk_forward_backtest(data, strategy_fn, train_window=252,
                         test_window=63, step=63):
    """
    Walk-forward analysis framework.
    
    1. Train on [0, train_window]
    2. Test on [train_window, train_window + test_window]
    3. Step forward and repeat
    
    This is the ONLY proper way to validate a trading strategy.
    """
    results = []
    n = len(data)
    start = 0

    while start + train_window + test_window <= n:
        # Split into train and test
        train_data = data.iloc[start:start + train_window]
        test_data = data.iloc[start + train_window:
                              start + train_window + test_window]

        # Fit strategy on training data
        params = strategy_fn.fit(train_data)

        # Test on out-of-sample data
        test_returns = strategy_fn.predict(test_data, params)

        results.append({
            'period_start': test_data.index[0],
            'period_end': test_data.index[-1],
            'return': test_returns.sum(),
            'sharpe': test_returns.mean() / test_returns.std() * np.sqrt(252)
                       if test_returns.std() > 0 else 0,
            'n_trades': len(test_returns[test_returns != 0]),
        })

        # Step forward
        start += step

    results_df = pd.DataFrame(results)
    print("Walk-Forward Results:")
    print(f"  Total periods: {len(results_df)}")
    print(f"  Mean return:   {results_df['return'].mean():.2%}")
    print(f"  Mean Sharpe:   {results_df['sharpe'].mean():.2f}")
    print(f"  Win rate:      {(results_df['return'] > 0).mean():.1%}")
    print(f"  Worst period:  {results_df['return'].min():.2%}")

    return results_df