Trading data infrastructure: normalization, instruments and observability

If you trade across venues, you already know every exchange speaks its own dialect. A Binance spot fill, a Liquidity Swap conversion and a futures contract definition all arrive in different shapes, with fees charged in different assets and limits that change without notice. Your research needs one meaning for a trade, and your execution needs one stable identity for each market.

Our trading business ran on exchange data like this, and we found that one shared representation couldn't serve both jobs well. So we split the work into two paths: a research library for normalized history and an instrument service for current market rules. Each path stayed small enough to make its own assumptions explicit.

This article walks through the data systems behind our trading technology. It follows one illustrative fill from raw response to research frame, and one constraint change from the exchange to the code placing orders. It also covers how we stored model output beside market and account data, and where our comparable units stop being exact.

Why one shared representation doesn't fit both jobs

The tempting move is one data model that research and execution both read. The trouble is that the two jobs want different things. Research needs history in comparable units. Execution needs the current rules for a market, delivered quickly and predictably. We gave each job its own system:

• A Python exchange library retrieved historical trades and pool records, validated them and returned normalized data frames for analysis.
• An instrument service stored market identities, precision, size limits, pool metadata and futures details, then served the active records through an API.

The research path ends in trade-flow and PnL analysis, the work described in our quant research workflow. The reference path ends in execution, where trading bots check a market's identity and limits before sending orders.

Neither path had to carry the other's baggage. An execution event didn't need a full exchange response attached, and a notebook didn't need to know how orders were sized.

One trade record for every response shape

The library's first job was to make trades from different venues and market types look the same, without losing the details that change their economics.

What the record carries

Our common trade schema had one row per execution, with these groups of fields:

• venue, market type and symbol, plus base and quote asset identities
• side and maker/taker role
• amount spent, amount received, fee and fee asset
• effective price, base and quote volume, and timestamp

Spot normalization converted the exchange's buyer and maker flags into explicit side and maker/taker fields, sorted fills by exchange time and assigned the spent and received assets by direction. A buy spends quote and receives base; a sell does the opposite. Writing that down once removed a whole class of sign errors from downstream analysis.

Fees took two paths. When the commission came out of the received asset, the normalizer subtracted it from the amount received, so the record matched what actually arrived. Otherwise, it applied a configurable maker or taker fee multiplier when calculating effective price. Every effective price therefore carried a fee assumption, and callers could replace the defaults for their own analysis.

Illustrative example: a fee taken in the received asset

This is a synthetic fill, not a real trade. A buy spends 1,000 quote units and receives 10 base units, and the exchange charges 0.01 base units as commission. We chose a commission in the received asset because it exercises the branch where the fee changes inventory, not just cost.

The quoted price was 100, but the strategy now holds 9.99 base units that cost 1,000. The effective price of 100.1001 quote per base sits about 10 basis points above the raw 100, which is the commission showing up as acquisition cost.

What ignoring the fee would distort

Suppose research used the raw 100 and 10 units instead. Two errors follow from that one shortcut:

• Inventory is overstated by 0.01 base, so the recorded position exceeds what the wallet holds.
• Acquisition cost is understated, at 100 rather than about 100.1 quote per base.

How much that inflates later profit depends on the sale. Selling all 9.99 units overstates profit by 1 quote; selling 5 units overstates it by about 0.5. The error isn't a fixed 10 bp on every sale. Expressed as a rate, it depends on the denominator: the 1,000 spent, the sale proceeds or a report's total spot volume.

In a report, this row contributes 1,000 to quote volume. If the quote asset is on the caller's list of stablecoins, the USD-volume estimate counts it as $1,000 of turnover, a convention we return to below.

Pool conversions keep their own semantics

In our data, "swap" meant Binance Liquidity Swap pools, not perpetual swaps. The pool client used the /sapi/v1/bswap endpoint family to fetch pool configuration, liquidity, conversion quotes and history, and claimed rewards. These pools were the hedge venue in our spot/swap arbitrage, so their records had to sit beside spot fills without pretending to be order-book trades.

The conversion normalizer kept only successful swaps, resolved each conversion's direction against the requested pair and marked it as a taker execution. A pool conversion always takes liquidity, so the role is fixed rather than read from a flag. Rewards and fees then got explicit fields:

• Estimated rewards multiplied base or quote volume by a configurable BNB reward factor.
• Claimed rewards came from a separate history endpoint recording what the exchange actually paid.
• Effective pool fee combined a reference conversion fee with the pool's swapRewardsRatio: swap fee × (1 − reward ratio).

The fee helper took its reference fee from a BTC/ETH conversion quote, so effective fees for other pools inherit that assumption. Separating estimates from payouts mattered because the strategy's economics included rewards, and a blended figure couldn't show which side was wrong.

Getting history out reliably

Historical retrieval is where exchange APIs become tedious: page limits, rate limits and occasional changes in response shape. The library handled each one in a fixed place.

Pagination and concurrency

Time pagination worked backward. Starting from the requested end, each new page ended one millisecond before the earliest timestamp in the previous batch, so no record was fetched twice. After the loop, the combined records were filtered down to the requested start time.

Parallel mode split the time range into intervals and gathered the pages concurrently, then flattened the results. Splitting by time works because pages in different intervals don't depend on each other's cursors. Pool history requested both conversion directions unless the caller asked for one side.

Caching and validation

A disk-backed cache memoized synchronous and asynchronous calls, with optional expiry and a selectable set of arguments forming the cache key. The intent was simple: reuse expensive history requests while a researcher iterated on normalization or analysis.

Validation was optional and sat at two boundaries, both active when the normalizer's validation flag was enabled:

• Raw responses were checked against JSON schemas derived from Pydantic response models.
• Normalized data frames were checked against Pandera schema models.

A changed API field failed where exchange data entered our representation, rather than surfacing three steps later as a strange number in a notebook.

Instruments: stable identity, current constraints

Execution needed a different kind of data: the current rules for each market, attached to an identity that never moved. That was the instrument service's job.

Identity and market-specific metadata

Each instrument had a stable numeric ID tied to exchange, account type, symbol and base/quote assets. Specialized records carried the fields each market type supports:

• Spot: precision, order-amount limits and minimum notional.
• Liquidity Swap pools: minimum and maximum conversion amounts and both sides of pool liquidity.
• Futures: contract status, type and dates, margin percentages, precision, size limits and the margin asset.

The API reported spot and pool instruments against the spot wallet, and mapped futures to a linear-futures account with margin-asset fields. Wallet and margin distinctions travelled with the instrument, so execution code didn't infer them from symbol names.

What the refresh loop covered

Supporting a field isn't the same as refreshing it. Our configured updater fetches instrument definitions concurrently from Binance, CommEX and venues reached through XTRD on a five-minute loop. Pool definitions aren't part of that active loop, so the service's pool liquidity fields aren't updated on that schedule.

The venue list describes integration capability, not where we traded.

Illustrative example: a constraint refresh

Suppose the exchange raises the minimum notional on the pair from our fill example. On the loop's next pass, synchronization found the existing identity, compared the incoming limit with the active spot metadata, updated it in place and refreshed its timestamp.

A consumer polling with a nanosecond since cursor then received only records changed after its last refresh. The instrument ID stayed the same; only the constraint changed. Strategy state keyed on that ID kept working, and the next order check used the new minimum.

The economic consequence is direct. If a 1,000-quote order now fell below the minimum, the exchange would reject it, so the strategy had to size up or skip the opportunity. Learning the rule from reference data is cheaper than learning it from a rejection.

Assets kept a history

Assets followed a different rule. When balance precision changed, the service deactivated the old asset record and inserted a new active one. Asset precision kept a history; instrument constraints held the current state. A consumer could see when an asset's precision changed, while an instrument showed only its current limits and an updated timestamp.

Observability: from records to diagnostics

Normalized records and stored model output fed the tools we used to watch strategies across many bots and markets, alongside order and trade records and live monitoring dashboards. In daily operations, these tools relied on the earlier normalization work.

A registry for stored market data

Alongside the research library, we kept a datastore for observations and model output. A registry mapped each market-data type to three things:

• a storage model defining how its rows were persisted
• a validation schema for incoming records
• the filters a query was allowed to use

Registered types included OHLCV candles, Kalman filter output, bot performance, price indices and account trades. Storing model output beside the market and account data around it made the model inspectable.

When an estimate looked wrong, we could pull the candles and trades from the same period and compare them directly. The trading algorithm built on that Kalman output belongs to its own article.

Trade-flow reports for spot/pool arbitrage

For spot/pool arbitrage, the trade-flow report merged spot fills and pool conversions in timestamp order, kept their USD volumes separate and assigned zero pool rewards to spot rows. From that flow, it derived diagnostics such as:

• Sequence errors: adjacent rows from the same venue, breaking the expected maker-fill-then-hedge alternation.
• Spot/pool trade-count and volume imbalance.
• PnL per spot volume in basis points, with and without rewards.
• Share of the venue's spot volume.

The sequence flag is a diagnostic, not a fill-to-hedge match. Clean alternation doesn't prove each fill was hedged correctly, and a flagged pair doesn't say which hedge was missing.

Duplicate timestamps were nudged forward in 1 ms steps to make a unique report index, so latency analysis used the original execution timestamps instead. What these numbers say about the strategy's economics is covered in our spot/swap arbitrage article; here, they're simply what the data path produced.

Comparable units and what they don't measure

A common unit makes cross-market analysis possible, but every conversion bakes in a choice. Two numbers deserve that caution.

USD-equivalent turnover

Pairs quote in different native units, so screening across them needs a common one. Our USD-volume estimate treated crypto/stable, stable/crypto and crypto/crypto pairs separately. Caller-listed stablecoins counted as exactly one dollar. Any other asset was valued at the last price of its stable-quoted pair with the highest quote volume.

This is an approximation built for research. It ignores stablecoin deviations from one dollar and price movement within the period. Company-level volume figures are reported separately by the company and shouldn't be read as outputs of this helper.

The 200-million-transaction dataset was a separate job

When we needed to document our source of funds, we engaged Clear Judgment, an outside advisory firm. Clear Judgment built custom forensic methods to process a dataset of 200 million transactions, and advised an audit firm on processing it.

What the split bought us, and where to start

The pattern is the same throughout: separate the jobs, then make each one explicit. The historical-trade library turned raw exchange responses into normalized research frames, with direction, fees and rewards spelled out. The instrument API distributed identity and current constraints to execution, refreshing limits beside an ID that never moved.

The two paths complement each other rather than share one representation. A notebook can compare a spot fill with a pool conversion in common units without knowing any order rules. A strategy can pick up a new minimum notional without parsing a trade history. The datastore stores market observations and model output; the trade-flow reports analyze normalized executions.

If you're weighing a shared model against two paths, start with a test. Run the synthetic fill above through your pipeline. It should record 9.99 base received and an effective price of about 100.1001 quote per base. If it records 10 base at 100, your research frames are carrying the distortion described here.

For how these systems fit the rest of our stack, see our trading technology overview.