Cross-validation for overlapping financial observations

If you label market events with a start time and an end time, you've probably watched a model score well in validation and then fall apart on fresh data. Rows can sit in clean chronological order and still share information, because a label's outcome interval can reach across the train/test boundary.

At 3Jane Technologies, our research pipeline handles this with a specific set of split and evaluation conventions. This article explains each one, with worked indices from our validation test suite.

Why observation intervals matter at the train/test boundary

In our research workflow, every observation has two timestamps. The start is when the event begins. The end is set by the label, for example the moment a profit, stop or time barrier is hit. The information an observation carries runs from its start to its end, not just at its row position.

That's why sorting rows by time isn't enough. A training row near the boundary can have a label that resolves after the first test row has already started. Then the model has learned from outcomes that overlap the period you're testing on. The reverse also happens: a test row can begin before the last training label has finished.

The conventions below each handle part of this problem:

• Walk-forward windows with a gap between training and testing.
• Combinatorial test-fold choices, with an option to keep previously used data in training.
• Test filtering by start and end timestamps.
• A separate embargo on training data after test blocks.
• Reconstruction of several evaluation paths from the split predictions.

Walk-forward windows

Walk-forward validation trains on earlier observations, skips a gap, then tests on a later interval. It repeats that pattern across the history. Our implementation lets you choose how the window sizes are set and how the training window behaves between splits.

Fixed or proportional, moving or expanding

Sizes can be fixed counts or proportions of the data. The training window can also move or expand:

• Moving: the training window slides forward and keeps a constant length, so old data drops out.
• Expanding: the training window keeps its start and grows with each split.
• Gap: a configurable number of observations between the end of training and the start of testing.

A small synthetic case with 10 observations shows the ordering. Train on [0,1,2], leave [3] as the gap, test on [4,5]. Later folds then either slide or extend the training block. Our tests cover sizing, expansion, invalid settings and fixed-size cases.

A fixed-size split from our test suite

One test case uses 20 observations with train_size=4, gap_size=1, test_size=5, n_splits=3 and fixed sizes. Each split trains on four rows, skips one and tests on the next five:

index     0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19
split 1   T  T  T  T  g  E  E  E  E  E  .  .  .  .  .  .  .  .  .  .
split 2   .  .  .  .  .  T  T  T  T  g  E  E  E  E  E  .  .  .  .  .
split 3   .  .  .  .  .  .  .  .  .  .  T  T  T  T  g  E  E  E  E  E

T = train, g = gap, E = test

Training indices are [0,1,2,3], [5,6,7,8] and [10,11,12,13]. The matching test intervals are [5..9], [10..14] and [15..19]. Indices 4, 9 and 14 fall in the gaps.

Combinatorial test folds

Walk-forward gives you one sequence of out-of-sample results. Combinatorial splitting gives you several. The data is cut into n_folds chronological groups, and every possible choice of k_tests groups becomes a test set. The remaining groups train the model.

How many fits and paths you get

With N groups and k test groups, you fit the model C(N,k) times. For k greater than one, those fits can be rearranged into C(N−1,k−1) evaluation paths, each covering every group once before purging.

Take N=5 and k=2. That's 10 fits and 4 paths. Each group is tested in four of the ten splits, so every fold starts with four separate out-of-sample predictions. Purging can then remove some of those rows, which shortens the affected test folds.

Reserving observations you've already used

The tainted_up_to setting marks an earlier segment of the index as off-limits for testing. Those observations are taken out of the candidate test folds and prepended to the training data in every split.

This represents data already seen during research. If you've used a period to explore features or tune ideas, testing on it again would score the model on information that shaped it. Keeping it in training still lets the model learn from it.

Test filtering: how our purge works

Combinatorial folds alternate between training and test blocks, so overlap can occur at many boundaries. The purged variant of our combinatorial splitter receives each bar's start and end timestamps and filters the test indices. It records which test indices it removed and the adjusted size of each test fold.

The rule

A test observation is kept only when its start timestamp is strictly later than the end timestamp of the last training observation before it:

keep test i  if  start[i] > end[last preceding train]

This method filters test observations and returns the training groups unchanged. Our tests assert both parts: test groups shrink, and training groups stay the same. Base combinatorial splits don't purge at all; they pass their indices through untouched.

A worked purge from our test suite

This test case alternates training groups [0,1,2], [6,7,8], [12,13,14] with test groups [3,4,5], [9,10,11], [15,16,17]. In its first case, the last training observation before each test group ends at timestamps 4, 16 and 28.

Index 9 starts at 15, before the preceding training label ends at 16. Indices 15 and 16 both start at 27, before 28. The test groups shrink to sizes 3, 2 and 1, while all nine training indices remain:

index     0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17
before    T  T  T  E  E  E  T  T  T  E  E  E  T  T  T  E  E  E
after     T  T  T  E  E  E  T  T  T  x  E  E  T  T  T  x  x  E

x = removed test observation

Relating this to published purged cross-validation

Purged and combinatorial purged cross-validation come from Marcos Lopez de Prado's Advances in Financial Machine Learning, and our pipeline draws on that work. If you know those methods, check the direction before comparing results. In our convention, purging removes test observations, and a separate embargo trims training observations. Describing this code as removing overlapping training labels would invert what it does.

Filtering the test side preserves the training groups, which keeps the training data identical across the combinatorial splits that share it. The cost is shorter test folds, which the path reconstruction has to account for.

Training embargo after test blocks

Embargo is a separate step. It removes the leading training observations that come right after a test block. The embargo length is the larger of two values: a fixed number of bars, or a percentage of the adjacent test block's length, truncated to a whole count. If it's long enough, the embargo can extend across successive training groups.

A test case flips the earlier layout. Test groups are [0,1,2], [6,7,8], [12,13,14]; training groups are [3,4,5], [9,10,11], [15,16,17]. With embargo_bars=2, the first two rows of each training group go:

index     0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17
before    E  E  E  T  T  T  E  E  E  T  T  T  E  E  E  T  T  T
after     E  E  E  x  x  T  E  E  E  x  x  T  E  E  E  x  x  T

x = embargoed training observation

Training shrinks to [5], [11] and [17], and all test groups stay as they were. Notice that the embargo works on counts, not timestamps. The purge checks label intervals before a test block; the embargo puts distance after it.

From split predictions to evaluation paths

Once the splits are filtered and embargoed, you still need to turn ten separate fits into something you can score like a trading history. That happens in two steps: predict every split, then reassemble the predictions into paths.

Fitting each split

For each split, the pipeline clones the estimator, following scikit-learn conventions, so no fitted state leaks from one split to the next. It fits on the training indices and predicts the test indices. Each split returns y_true, y_pred and the original observation indices.

In probability mode, it keeps the positive-class probabilities rather than hard labels. Splits can run in parallel because they're independent.

Assembling paths

The path builder partitions each split's predictions into their source test folds and assigns each split-and-fold pair to exactly one ordered path. Each path assembles chronological held-out fold segments from their retained predictions, and no prediction is reused across paths. Without purging, a path covers each fold once; where purging removed rows, its coverage is reduced.

Purging matters here. You pass the adjusted test-fold sizes from the purge step, so the builder knows a fold may now hold fewer predictions. In the purge example, that split's test folds contribute 3, 2 and 1 predictions instead of three each. An integrated test case pins both the split indices and the resulting path indices.

An illustrative path assignment

The table below is illustrative. It shows one valid assignment for N=5 and k=2, with splits numbered in enumeration order: S1=(0,1), S2=(0,2), S3=(0,3), S4=(0,4), S5=(1,2), S6=(1,3), S7=(1,4), S8=(2,3), S9=(2,4), S10=(3,4). Our tests fix the implementation's own ordering.

Read Path 2 top to bottom: fold 0 from S2, folds 1 and 2 from S5, fold 3 from S6, fold 4 from S7. All 20 split-and-fold pairs appear exactly once. You can then score each path, and the spread across four paths shows how much a result depends on one historical partition.

Why each convention exists

Each convention answers a different validation question:

• Moving versus expanding windows test different memory assumptions: whether old regimes still help, or recent data should dominate.
• Gaps separate the fit period from the test period in walk-forward runs.
• Test filtering removes test observations whose information overlaps the preceding training labels.
• Training embargo separates the held-out blocks from the training observations right after them.
• Multiple paths expose dependence on a single historical split.

None of these steps tells you whether a signal is tradable. That depends on costs, sizing and execution, which we cover in backtesting and position sizing. Financial dependence also changes the resampling unit; our visual comparison of bootstrap methods shows how serial dependence leads to block resampling.

Applying this to your own validation

If your labels carry end timestamps, start with one check: for each test fold, compare test start times with the end of the training label right before it. If any start earlier, your scores include overlapping information. From there:

1. Decide which side you filter, and state that convention wherever you report results.
2. Add an embargo after each test block, sized from bars or a share of the test length.
3. Pass the adjusted fold sizes into your path reconstruction.
4. Score every path, not just the best one.

You can see how this step fits between labeling and modeling in our quant research workflow, and find related work on the research page.