From model predictions to backtests and position size

You've trained a classifier, validated it on held-out data and the scores look promising. The next question is harder. What does a 0.62 probability actually mean for your account? It's not yet a trade. It becomes one only after a decision rule, a cost model, an execution assumption and a sizing rule have all had their say.

This article walks through how our research pipeline handles that path at 3Jane:

• how model probabilities become signals
• how we check whether those probabilities are trustworthy
• how simulated trades are costed and executed
• how Kelly and Monte Carlo methods constrain exposure

We finish with one synthetic scenario carried all the way from probability to net PnL.

Why a good classifier isn't yet a strategy

A predictive metric is an intermediate result. Accuracy scores thresholded classifications as right or wrong, and log loss scores the assigned probabilities against observed outcomes. Whether a signal is tradable depends on what happens after the prediction: which signals act, how much capital each gets, and what fees and fills cost along the way.

So we treat the stretch between model output and simulated PnL as its own engineered stage of the quant research workflow. It runs in a fixed order:

1. A decision rule turns probabilities into signals.
2. A calibration check tests whether probabilities match observed frequencies.
3. A sizing object translates each signal into desired exposure.
4. Pre-trade risk callbacks adjust that exposure.
5. A Backtrader simulation applies costs, the margin multiplier and entry behavior.

Each stage makes one assumption explicit. That's the main advantage over folding everything into a single backtest script: when a result changes, you can see which assumption moved it.

Turning probabilities into decisions

Our classifier wrappers sit around the research model families we support, including LightGBM, XGBoost, random forest, nearest-neighbor and voting ensembles. Each wrapper replaces the estimator's default decision with one of two explicit rules.

Fixed probability thresholds

The threshold classifier emits class 1 when the positive-class probability is at least its configured threshold. Everything below the threshold stays inactive. It's simple, and its meaning is easy to read: "act when the model says 60% or more."

A threshold only decides which signals act. It says nothing about whether a predicted 60% wins 60% of the time. That's a separate question, and sizing depends on it.

Train-derived quantile thresholds

The quantile classifier works differently. During fit, it estimates a chosen quantile of the model's probabilities on the training data. During predict, it compares every subsequent probability with that fixed value.

The cutoff never sees the test distribution. Setting a cutoff from test probabilities would quietly leak information about the evaluation period into the decision rule. A training quantile also gives you a rough handle on activity: a 0.8 quantile means about a fifth of training observations would have cleared the bar.

Checking whether probabilities mean what they say

Kelly-style sizing takes a win probability as input. If the model's probabilities are systematically optimistic, the sizing formula will be too. So before trusting a probability as a sizing input, we assess its calibration.

Our calibration function calls scikit-learn's calibration_curve on paired true labels and predictions. By default it uses 20 quantile bins and returns two arrays: the observed fraction of positives in each bin and the mean prediction in that bin. A well-calibrated model puts those points near the diagonal.

This function is an assessment, not a correction. It computes diagnostic points and leaves probabilities untouched. Any probability-adjustment model would be a separate fitted component with its own training data and its own validation.

The predictions fed into the curve also have to be genuinely out of sample. With overlapping financial labels, that takes more than a random split. Our purging and embargo approach draws on Marcos Lopez de Prado's Advances in Financial Machine Learning; our article on financial cross-validation explains how our implementation builds held-out predictions.

The simulation boundary

Once signals exist, we hand them to Backtrader through a thin glue layer. That boundary is where research assumptions become explicit simulation inputs, so it's worth looking at what crosses it.

What goes into Backtrader

The glue layer configures three things:

• The data feed. The input frame supplies OHLCV columns plus the signal column.
• The broker. It receives initial cash, cost fields from an exchange-cost model and a margin multiplier of 1/initial_margin, so each unit of equity can carry that multiple in position value.
• The strategy. Our strategy wrapper receives the sizing object, the pre-trade risk callbacks and the entry settings.

Keeping these separate means you can swap a sizing rule without touching costs, or stress costs without retraining anything.

How costs reach the broker

Costs come from three fields in the exchange-cost model: taker fee, impact and drift. The wrapper passes -taker_fee-impact-drift to Backtrader's commission setting. That sign convention depends on how the exchange-cost object represents each field, so check the representation before mapping a fee into a configuration.

These fields represent modeling assumptions about execution. The worked example later uses plain positive costs as arithmetic, separate from this configuration path.

Market and limit entries

Entry type is configurable. Market entries execute against the bar. Limit entries are offset from the close: a limit buy sits below close and a limit sell sits above it, by max(close*limit_offset_pct, limit_offset_min). The minimum keeps the offset meaningful at low prices.

Open orders are canceled at the next strategy step, so a stale limit order can't fill bars later under different conditions. Signal reversals include a separate closing operation before the new position is built.

Worst-price equity

An optional setting values unrealized PnL at the bar's worst price: the low for longs and the high for shorts. That's a more conservative mark than the close. It changes the valuation assumption only; it doesn't reproduce a full exchange order book or intrabar fill sequence.

Sizing inside the execution path

Sizing isn't a post-processing step in our setup. It runs inside the strategy on every bar. The wrapper's get_size asks the sizing object to translate the current signal into desired exposure, then passes that exposure through the pre-trade risk callbacks in order. Each callback can reduce or otherwise adjust what the previous one produced.

Exposure then becomes a quantity. Desired size is portfolio_value / close * exposure_desired, and the order quantity is the difference between desired size and the current position. If you already hold most of what the signal wants, the strategy only trades the gap.

The wrapper records both desired and current exposure, so you can see where callbacks or fills held the position away from what the sizer asked for. Our simulation tests exercise custom sizing, limit entries, feed requirements and the pre-trade callbacks.

Kelly and fractional Kelly

Our directional Kelly sizer works in return magnitudes. You supply a win probability p, an expected winning return win and an expected losing return loss. Positive exposure is:

max(0, p/abs(loss) - (1-p)/win)

The max(0, ...) means a signal without a positive edge under those assumptions gets zero long exposure rather than a negative one.

FractionalKelly multiplies that result by a configured fraction. Full Kelly maximizes long-run growth only if your inputs are right, and the inputs are estimates, so fractional Kelly is a hedge against your own estimation error. The worked example shows how quickly a small probability error eats that margin.

Drawdown preferences with Monte Carlo

Kelly answers "what maximizes growth?" Many traders care about a different question: "how likely am I to fall a given distance from my peak?" Our risk-adjusted Kelly sizer addresses that with simulated wealth paths, computed in a native C++ engine.

Path simulation and bet-size search

The engine simulates win/loss wealth paths for a candidate bet size, tracks each path's running high and counts paths that cross high * minimum_wealth. A bisection-like search adjusts bet size until the estimated crossing probability is close to the selected preference. OpenMP distributes the path calculations across cores.

Two path modes are available. Compounded paths multiply wealth by each outcome. The alternative adds or subtracts fixed increments measured against initial wealth.

The estimated Kelly fraction and its assumptions

estimate_kelly_fraction sets minimum_wealth = 1 - max_dd. It then averages the ratio of Monte Carlo bet size to Kelly bet size over a synthetic grid: probabilities from 0.6 to 0.8, and equal win/loss movement sizes starting at 0.015 in 0.05 increments.

Its source defaults are research inputs:

• 1,000 curves per estimate
• 50,000 steps per curve
• a search tolerance of 1e-4
• a 0.001 maximum threshold-crossing probability

The result is a model-dependent sizing heuristic, not a drawdown guarantee. It assumes independent binary outcomes on that grid. Real returns cluster and payoffs vary, so a live risk policy needs its own configured limits.

Worked example: one probability, one trade

Here's one synthetic scenario carried through the whole path. Every number is illustrative.

Threshold and size

Say the decision rule uses an illustrative fixed threshold of 0.55, inclusive, so any p of 0.55 or more acts. The model outputs p = 0.60, which clears it, so the long signal acts.

We assume a winning return of 0.20 and a losing return of 0.20. Full Kelly is 0.60/0.20 - 0.40/0.20 = 3 - 2 = 1, and half-Kelly gives 0.5 exposure.

With $10,000 of capital, a $100 close and no current position, desired quantity is 10000 / 100 * 0.5 = 50 units. Assume the pre-trade risk callbacks leave that target unchanged, so all 50 units are executed.

The costed simulated trade

The position is later sold at $110, so gross PnL is 50 × $10 = $500. Assume a positive cost of 0.1% on each executed notional. Entry notional is $5,000, costing $5; exit notional is $5,500, costing $5.50. Net PnL is $500 − $10.50 = $489.50.

Notice the mismatch. The sizer assumed a 20% winning return, but this trade realized 10%. That's one outcome from the distribution, not a test of the assumption.

Sensitivity to probability and costs

Now change only p. At 0.55, the signal still meets the inclusive threshold: full Kelly is 2.75 - 2.25 = 0.5 and half-Kelly is 0.25, so the strategy buys 25 units. The same price move yields $250 gross, $5.25 in costs and $244.75 net. At 0.50, the signal falls below the threshold and stays inactive. The table extends the same arithmetic:

The key point: if the model says 0.60 but the true probability is 0.55, half-Kelly sizing lands exactly at full Kelly for the true edge. A five-point calibration error consumes the entire safety margin. On costs, tripling them to 0.3% per side takes the 50-unit trade from $10.50 to $31.50, or $468.50 net.

Where to go next

If you're building this path yourself, start with the calibration curve on genuinely held-out predictions, because every sizing number downstream inherits its errors. Then make costs, entry type and sizing explicit, separate inputs so you can stress each on its own.

To see how the held-out predictions are built, read our guide to cross-validation with overlapping labels. For the wider research index, including feature generation and labeling, start there. And if you want to see how fees and inventory shape a strategy's economics, our article on grid trading picks up that thread.