If you need the distribution of a statistic or a model score from one time series, the bootstrap is the obvious tool, but its literature is dense and hard to get through. I wrote this guide while working out which bootstrap method I needed for my own trading research at 3Jane Technologies. It's the guide I wanted: a plain explanation and a picture of one iteration for each established method.
Our autocorrelation paper found statistically significant autocorrelation on 5-minute and hourly bars in every market we tested, so resampling those observations as if they were independent would misrepresent our data. Below you'll find the mechanics of each method, the papers that proposed it, and the implementations you can use today.
Who this guide is for
You probably already know what resampling is for. When you have one dataset and no way to collect another, the bootstrap lets you estimate how much a statistic would vary. The hard part is choosing among the many variants, because each one is described in its own paper with its own notation.
The goal is a starting point. This guide is for anyone who needs the bootstrap but isn't sure which options exist or how to make a definitive choice. For each method I'll explain what happens, draw one iteration, and link the original paper, the analyses and comparisons, and a working implementation where one exists.
I have to give one honest caveat. I'm a programmer and machine learning specialist by trade, and applied math and statistics sit outside my specialty, so take everything here with a pinch of salt and re-verify it against the original papers.
The short answer
If you're in a hurry, here's the TL;DR:
- Your data is independently and identically distributed: use the original IID bootstrap. You can use numpy, sklearn or recombinator. For numpy and sklearn, set
replace = True. - Your data has serial dependence: use the Stationary Bootstrap with an optimal block size. recombinator implements both the method and the block size estimate.
The rest of the article explains why the second recommendation holds. Each method answers a specific flaw in the one before it. The diagrams show the flaw and its fix directly.
What makes a method a bootstrap
Bootstrap is a method of resampling data. Two characteristics classify a sampling method as a bootstrap:
- Sampling is done with replacement.
- The number of observations sampled equals the number of original observations.
A few variants below bend the second rule. The sufficient block methods discard repeated blocks, and the hybrid block bootstrap deliberately draws fewer blocks, so both can return shorter samples.
The decision that separates the methods is the resampling unit. The IID bootstrap draws individual observations. Block methods draw runs of consecutive observations, so the relationships inside each selected run survive into the resample.
The family of bootstrap methods
There are many proposed variations of the bootstrap. Here's a rough categorization of the ones covered here:
- Single-point IID bootstrap (IIDB)
- Block bootstrap methods
- Non-overlapping block bootstrap (NBB)
- Ordered NBB (ONBB)
- Sufficient NBB (SNBB)
- Sufficient Ordered NBB (SONBB)
- Moving block bootstrap (MBB)
- Circular block bootstrap (CBB)
- Stationary bootstrap (SBB)
- Other variations
The hierarchy tells you what each method changes. ONBB, SNBB and SONBB modify NBB, while CBB and SBB each fix a specific flaw in MBB. The hybrid block bootstrap also draws overlapping blocks, so I've put it after the moving-block family.
Every diagram uses the same toy dataset of 20 observations labeled by their positions 0 to 19. Each number tells you where an observation sat in the original series, and square brackets mark a block. The draws are illustrative examples chosen to show each mechanism.
Single-point IID bootstrap (IIDB)
The method was originally proposed in Efron (1979), and Chapter 2.2 of Lahiri (2003) gives the proper mathematical treatment. You draw one element at a time with replacement, usually as many times as there are elements in the dataset. Then you repeat the whole process many times. Typical parameters are:
sample_size = n = dataset_sizen_samples= hundreds to thousands
Fig. 1. One iteration of single-point IID bootstrap (IIDB)
Original (n = 20):
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
20 single draws with replacement, in draw order:
7 2 15 2 11 0 19 7 4 13 9 9 18 3 6 14 1 11 16 5
Drawn twice: 2, 7, 9, 11
Never drawn: 8, 10, 12, 17
A drawn element isn't removed, so it can occur more than once among the 20 results. Look at what happened to the neighbors. In the original series, 2 is followed by 3, but in the resample it's followed by 15 and later by 11. That doesn't matter for IID data, but it destroys the structure of a series with serial dependence.
Non-overlapping block bootstraps
The IID bootstrap assumes observations are independently and identically distributed. That mostly held for the problems that made the bootstrap widely adopted, but it doesn't hold for many others, including data with serial dependence. Block bootstrap authors all zeroed in on a similar solution: sample blocks of consecutive elements instead of single observations.
Non-overlapping block bootstrap (NBB)
Carlstein (1986) originally proposed NBB, and Lahiri (2003), Chapter 2.6, gives the mathematical treatment. In NBB, all blocks have the same length and can't overlap or share elements. Block positions are fixed before resampling starts and stay constant throughout the bootstrap process. Typical parameters are:
block_size= fromn^(1/5)ton^(1/2)n_blocks = n / block_size
Fig. 2. One iteration of non-overlapping block bootstrap (NBB), block size 2
Fixed blocks:
B0=[0 1] B1=[2 3] B2=[4 5] B3=[6 7] B4=[8 9]
B5=[10 11] B6=[12 13] B7=[14 15] B8=[16 17] B9=[18 19]
10 block draws with replacement, in draw order:
B3 B7 B3 B0 B9 B5 B7 B2 B8 B1
Resample (n = 20), merged in draw order without re-ordering:
[6 7][14 15][6 7][0 1][18 19][10 11][14 15][4 5][16 17][2 3]
Within each block, the original pair survives, so 6 is still followed by 7. The fixed boundaries have a cost, though. Because 1 and 2 sit in different blocks, the original 1→2 transition appears in a resample only when B1 happens to follow B0. For n = 20, the typical range gives block sizes of about 1.8 to 4.5, so a block size of 2 fits.
Ordered and sufficient variants of NBB
Beyaztas et al. (2016) proposed the next three modifications of NBB in the same article, and Beyaztas and Firuzan (2016) also discussed their performance. The authors describe them as "good competitors that often exhibit improved performance over the conventional block methods." Each variant changes only what happens between drawing the blocks and merging them, so the diagrams reuse the ten draws from Fig. 2.
Ordered NBB (ONBB)
ONBB sorts the drawn block identifiers into nondecreasing order before merging them. The selected blocks come back in their original chronological order, while a repeated block stays next to its copy. At those joins the series jumps back, so the two copies of B3 put 7 right before 6.
Fig. 3. One iteration of ordered non-overlapping block bootstrap (ONBB), block size 2
Drawn: B3 B7 B3 B0 B9 B5 B7 B2 B8 B1
Sorted: B0 B1 B2 B3 B3 B5 B7 B7 B8 B9
Resample (n = 20):
[0 1][2 3][4 5][6 7][6 7][10 11][14 15][14 15][16 17][18 19]
Sufficient NBB (SNBB)
SNBB discards repeated blocks after all blocks are drawn, then merges the remaining blocks in draw order. Here the second B3 and the second B7 disappear, so each distinct block appears exactly once, and this resample shrinks to 16 elements.
Fig. 4. One iteration of sufficient non-overlapping block bootstrap (SNBB), block size 2
Drawn: B3 B7 B3 B0 B9 B5 B7 B2 B8 B1
Repeats discarded: B3 B7 B0 B9 B5 B2 B8 B1
Resample (16 elements), merged in draw order:
[6 7][14 15][0 1][18 19][10 11][4 5][16 17][2 3]
Sufficient Ordered NBB (SONBB)
SONBB applies both changes. It discards repeated blocks, then sorts the remaining blocks in ascending order before merging. The result reads like the original series with the undrawn blocks removed, which are B4 and B6 in this draw.
Fig. 5. One iteration of sufficient ordered non-overlapping block bootstrap (SONBB), block size 2
Drawn: B3 B7 B3 B0 B9 B5 B7 B2 B8 B1
Repeats discarded, sorted: B0 B1 B2 B3 B5 B7 B8 B9
Resample (16 elements):
[0 1][2 3][4 5][6 7][10 11][14 15][16 17][18 19]
Moving block bootstraps
The second branch of block methods drops fixed block positions. Blocks from neighboring starts overlap, so a block boundary can't permanently separate two neighboring observations. MBB admits every start that leaves room for a full block, which is 0 through 16 for 20 observations and blocks of 4. CBB and SBB also admit starts 17 through 19 by wrapping around to the beginning.
Moving block bootstrap (MBB)
Künsch (1989) originally proposed MBB, and Lahiri (2003), Chapter 2.5, gives the mathematical treatment. MBB started another branch of bootstrap methods. Its main difference from NBB is that block positions aren't predefined, and blocks can overlap. The typical parameters match NBB's:
block_size= fromn^(1/5)ton^(1/2)n_blocks = n / block_size
Fig. 6. One iteration of moving block bootstrap (MBB), block size 4
Possible block starts: 0-16 (17 blocks)
5 block draws with replacement, in draw order:
start 3 -> [3 4 5 6]
start 11 -> [11 12 13 14]
start 5 -> [5 6 7 8]
start 0 -> [0 1 2 3]
start 14 -> [14 15 16 17]
Resample (n = 20), merged in draw order without re-ordering:
[3 4 5 6][11 12 13 14][5 6 7 8][0 1 2 3][14 15 16 17]
The first and third blocks share 5 and 6, which NBB could never produce. The 1→2 transition that NBB split at a fixed boundary now survives inside any block that starts at 0 or 1.
Circular block bootstrap (CBB)
Politis and Romano (1992a) and Politis and Romano (1992b) originally proposed CBB, and Lahiri (2003), Chapter 2.7.1, gives the mathematical treatment. They identified a problem with MBB: because block sizes are fixed, the edges of the dataset are systematically undersampled.
Unequal inclusion gives observations unequal expected weights in the resample. That can shift the expected mean of the bootstrapped datasets away from the mean of the original dataset. Whether it does, and by how much, depends on the values near the edges, so it needn't happen for every dataset.
Fig. 7. One iteration of circular block bootstrap (CBB), block size 4
The series wraps around: ... 17 18 19 | 0 1 2 ...
Possible block starts: 0-19 (20 blocks)
5 block draws with replacement, in draw order:
start 17 -> [17 18 19 0]
start 6 -> [6 7 8 9]
start 19 -> [19 0 1 2]
start 9 -> [9 10 11 12]
start 13 -> [13 14 15 16]
Resample (n = 20):
[17 18 19 0][6 7 8 9][19 0 1 2][9 10 11 12][13 14 15 16]
Number of possible blocks containing each position:
position: 0 1 2 3 ... 16 17 18 19
MBB (17): 1 2 3 4 ... 4 3 2 1
CBB (20): 4 4 4 4 ... 4 4 4 4
In MBB, position 0 appears in only one of 17 possible blocks, while every interior position appears in four. CBB fixes this by "wrapping the dataset around in a circle," so a block can run from the end back to the beginning. Every observation then appears in four of the 20 possible blocks and is sampled equally, and CBB keeps MBB's typical parameters.
Stationary bootstrap (SBB)
The next modification of MBB comes from the same authors, in Politis and Romano (1994), and Lahiri (2003), Chapter 2.7.2, gives the mathematical treatment. Even when the input dataset is stationary, the sample MBB produces lacks stationarity. With fixed block lengths, every join between blocks lands at the same output positions, which here are 4, 8, 12 and 16.
SBB uses randomized block sizes instead of fixed ones. Block sizes follow a geometric distribution around a selected mean, and you keep drawing blocks until you've sampled n elements. Typical parameters are:
mean_block_size= fromn^(1/5)ton^(1/2)output_size = n
Fig. 8. One iteration of stationary bootstrap (SBB), mean block size 4
The series wraps around: ... 17 18 19 | 0 1 2 ...
Block draws with replacement until 20 elements are sampled:
start 8, length 2 -> [8 9]
start 1, length 6 -> [1 2 3 4 5 6]
start 18, length 4 -> [18 19 0 1]
start 10, length 5 -> [10 11 12 13 14]
start 4, length 5 -> [4 5 6] (truncated to 3 at n = 20)
Resample (n = 20), merged in draw order without re-ordering:
[8 9][1 2 3 4 5 6][18 19 0 1][10 11 12 13 14][4 5 6]
A random start and length can carry a run past observation 19, so SBB continues it circularly from 19 back to 0, as the third block shows. The last run is truncated once the resample reaches 20 observations, so its drawn length of 5 becomes 3. Random lengths, rather than the wrapping it shares with CBB, set SBB apart.
Hybrid block bootstrap (HBB)
Kuffner et al. (2018) originally proposed HBB. It samples fewer than n / block_size blocks in each iteration and provides a procedure for selecting the optimal number of blocks and block size for a specific dataset. That deliberately breaks the same-size rule, because the resample is typically smaller than the original dataset. Typical parameters are:
block_size= fromn^(1/5)ton^(1/2)n_blocks = 0.1 … 0.2 * n / block_size
Fig. 9. One iteration of hybrid block bootstrap (HBB), block size 4
3 block draws with replacement, in draw order:
start 2 -> [2 3 4 5]
start 12 -> [12 13 14 15]
start 4 -> [4 5 6 7]
Resample (12 elements, smaller than n = 20):
[2 3 4 5][12 13 14 15][4 5 6 7]
With 20 elements and blocks of 4, the typical formula gives between half a block and one block. The picture draws three blocks so you can see the overlap and the shorter output.
Other variations
There are many more bootstrap variations. Since there are limits to what one article can describe and one picture can show intuitively, I'll only link these:
If none of the methods above fits your data, these are the next ones to read about.
Choosing a method and block size
The correct way to choose is to understand the problem you're trying to solve and the properties of your data. After that, read the original paper and the comparisons to judge how well a specific variation fits. Short of that, a few rules of thumb cover most cases.
Rules of thumb
Here are the rules I keep in mind:
- If your data is i.i.d., the original IID bootstrap is sufficient, and you may not need anything else.
- If your data has serial dependence, use one of the block bootstraps. Among them, the Stationary Bootstrap seems to solve most of the problems of the other variations, and it has procedures for selecting optimal block sizes.
- If you have more specific needs, you'll need to dive in and do more exhaustive research.
For market returns with measurable autocorrelation, the second rule applies, which is why the stationary bootstrap is my default.
Picking the block size
Every block method needs a block size, or a mean block size in SBB's case. The typical range of n^(1/5) to n^(1/2) is wide. For 10,000 observations, it runs from about 6 to 100.
That spread is why the optimal block size procedures matter. recombinator's optimal block length estimator gives you a data-driven value instead of a guess somewhere in that range.
Implementations
If you need the IID bootstrap, sklearn's resample with replace=True is enough. Otherwise, I strongly recommend taking a look at recombinator. It implements IIDB, MBB, CBB, SBB and others, and it provides convenience tools for estimating optimal block sizes and running statistical tests. It's a very underrated package.
Several packages implement the algorithms described above:
- Python: sklearn resample, recombinator and arch
- R: tsboot and meboot
Your next step
If your series has serial dependence, start with the stationary bootstrap. Estimate the mean block size with recombinator, generate hundreds to thousands of resamples, and see how much your statistic moves across them. Running the IID bootstrap on the same data alongside it shows you what ignoring the dependence changes.
Dependence affects more than resampling. It also changes how you should validate models on financial data, which I cover in financial cross-validation. Our quant research workflow shows where these steps fit, and the research page collects our papers.
This turned out to be a long read, and I hope it made things clearer. I tried my best to verify what I say and to represent each method correctly in the diagrams, but there's always room for mistakes. If you spot one, let me know, and I'll fix and update the article.