How we validate components

Section 01

Guiding principles

Six rules apply to every piece of work covered by this document. They are not aspirational; they are constraints. Work that violates one of them is not eligible for promotion, regardless of the numbers it produces.

1. Data integrity before component quality. A component trained on data that would not have been available at prediction time is not a component. It is a curve fit. Point-in-time correctness is the first gate, not the last.
2. Baselines before models. No predictive component is evaluated against a null. Historical mean, persistence, and zero-return baselines are computed for every target before any non-trivial model is fit. A model that does not beat its baselines out-of-sample has no claim on being a model.
3. Chronology, not randomness. Time series are never split with train_test_split or equivalent random shuffling. Splits are chronological, with purge and embargo applied where horizons overlap.
4. Each component is validated against its own question. A state engine, a forecast model, a retrieval system and a decision layer are validated against different evidence. No component's validation is evidence for another's, and no track gates the others.
5. Failures are published. A validation report is only meaningful if it discloses what did not work. Failed windows, unstable coefficients, non-converged fits, and rejected approaches are recorded alongside the successful results.
6. Reproducibility is a property of the pipeline, not the notebook. Every experiment must be reproducible from an identifier plus the recorded dataset, feature, code, and configuration versions. Anything less is not research; it is exploration.

Section 02

Validation by component class

Tickflow Capital validates every component against the question it was built to answer. There are two independent tracks - strategy validation and analytical validation - and within analytical validation there are several component classes, each with its own evidence type. No track gates another. No class's validation is evidence for another's.

Component classes within Track B

Each class is validated against its own evidence type. Applying the wrong evidence to the wrong class is itself a validation failure.

Section 03

Point-in-time data integrity

The single most common source of inflated backtest performance in macro and financial ML is the use of revised data. This section describes how Tickflow Capital prevents that class of error.

The requirement

For every row of training data, the pipeline must be able to demonstrate that every feature value used was published on or before the prediction timestamp. This is not a property of the code; it is a property of the storage layer. Data is stored with an available_at field, and every query filters on available_at <= as_of.

Vintage handling

Macroeconomic series are revised. GDP, employment, and inflation series are subject to multiple revisions after first publication. Using today's value of a series to backtest a decision made two years ago is look-ahead, even if no future-dated observations are used. Every series in the pipeline is graded:

Data quality gates

Before a dataset is eligible for training, the following checks must pass. Any failure halts the pipeline and produces a machine-readable reason.

Section 04

Split design and leakage prevention

Time-series data cannot be randomly split. Adjacent observations are correlated, targets overlap, and a randomly chosen test set contains rows whose information was already visible to the model during training.

Chronological splits

Every dataset is split chronologically. The default structure is train–validation–test, with each split covering a contiguous block of time and no overlap between them. Expanding-window and rolling-window walk-forward are supported and preferred for final evaluation.

Purge and embargo

When the target is a forward return over horizon h, the last h rows of the training set have targets that extend into the validation set. Those rows must be purged. Immediately after each split boundary, an embargo period is applied so that the first rows of the validation set do not share information with the last rows of the training set.

The test set is touched once

The final test set is not consulted during model development, feature selection, hyperparameter tuning, or any other iterative process. Consulting it more than once per artifact version converts it into a validation set, and invalidates the out-of-sample claim.

Leakage tests

Every pipeline runs an automated leakage test suite. A test failure is a hard block, not a warning.

Section 05

Baselines before models

Every predictive target is evaluated against a set of trivial baselines before any non-trivial model is fit. A model that does not beat its baselines out-of-sample has no claim on being a model.

Standard baselines

Section 06

Validation gates

Every component must clear a set of gates before it can be promoted between lifecycle stages. Gates are hard thresholds. Where a threshold is not yet defined, the gate is listed as pending and no promotion is possible until it is defined. The full gate set is applicable to components that produce predictions; infrastructure and retrieval components have a reduced gate set defined by their class.

Data gates

Statistical gates

Stability gates

Coherence gates

Section 07

Reproducibility and experiment tracking

Every experiment must be reproducible from an identifier plus the recorded inputs. If the same experiment is run twice on the same dataset, it must produce the same outputs. If it does not, the discrepancy is a bug and must be resolved before the experiment is treated as evidence.

What every experiment records

The reproducibility contract

An experiment is considered reproducible if, given the experiment ID and the code commit, a full re-run produces:

1. The same dataset snapshot, with the same row count and date range.
2. The same feature matrix, element for element.
3. The same model configuration, including fitted parameters, to within the documented numerical tolerance.
4. The same out-of-sample metrics, to within the documented numerical tolerance.
5. The same artifact hash, or a documented reason why the hash differs (e.g. platform-level floating-point divergence).
6. The same validation report, modulo timestamps.

Section 08

Lifecycle and decision authority

Every component moves through a defined lifecycle. Promotion between stages is a deliberate act, requires the evidence appropriate to the target stage, and is recorded permanently in the registry. In addition to lifecycle stage, every component carries a decision authority - the maximum scope of action its output is permitted to inform. Lifecycle and authority are separate axes; a component can be in Production and still be scoped to context-only authority.

Lifecycle stages

Decision authority

Every component carries a decision authority, independent of its lifecycle stage. A component in Production with Context authority cannot inform a position; a component in Shadow with Decision-support authority is not yet permitted to act on it. Authority is assigned explicitly and recorded in the registry.

Section 09

Shadow deployment

A component that has cleared its validation gates has proven itself on historical data. It has not proven itself on live data. Shadow deployment is the step between the two.

Evidence types — what "simulated" means

Five kinds of evidence appear at different stages of a component's life, and they are not interchangeable. Where the registry or the applied record describes a figure as simulated, it was produced by one of the first three categories below — not by live trading. Only the final category constitutes a live track record.

Where the applied record labels a period simulated holdout, that label states which category of evidence the figures belong to. It is not a claim of live trading. The registry uses this classification when assigning lifecycle stages: a component that has passed walk-forward and holdout evaluation but has not been deployed is marked Validated, not Live.

What shadow means

A shadow component runs against live data on a fixed schedule. It produces outputs on the same cadence as the production system would. Those outputs are stored, timestamped, and preserved alongside the input features that produced them. They are not used to make decisions.

What is measured

Duration and exit

Shadow duration is set by the component's output cadence and, where applicable, the target horizon. A monthly regime classifier requires a longer shadow period than a daily directional model because fewer outcomes are observed per unit time. Exit criteria are defined before shadow deployment begins; they are not adjusted mid-period to accommodate the observed results.

Section 10

Monitoring and drift

A component in production is not assumed to remain valid indefinitely. The following are monitored continuously, and departures trigger defined responses.

Section 11

What is published, what is withheld

This page, and the system registry it governs, are intended to be substantive. They are not exhaustive. The following boundary applies to every public document.

Section 12

Rejected approaches

A validation methodology is only as strong as the practices it rejects. The following approaches have been considered and are not permitted in any pipeline governed by this document.

Section 13

Governance

This document is a working standard. It is reviewed quarterly, and any change to it is recorded in the system registry's changelog.

Ownership

This methodology is owned by the firm, not by any individual. The founder is bound by it in the same way as any future team member. No component is promoted without the evidence required by the sections above, regardless of who is doing the work.

Challenging a result

Any validation claim may be challenged. A challenge is a request for the raw data, the configuration, the code commit, the lockfile, and the metrics that support the claim. If those cannot be produced, the claim is withdrawn until they can. This applies to claims made in public documents, in private reports, and in internal discussions.

Response to failure

When a validation claim turns out to be wrong, the response is:

1. Record the correction in the changelog, dated and attributed.
2. Determine whether the error was in the evidence, the interpretation, or the process.
3. If the process is the source of the error, amend this document.
4. Re-validate any component that relied on the withdrawn claim.