Training action foundation models without peeking
Upstream sequence models trained on future logs leak outcomes into past embeddings and inflate offline backtests.
Part 2 of 4 in our series on action foundation models: Part 1: Foundations · Part 2: Training · Part 3: Serving · Part 4: In Zipline.AI
In Part 1, we introduced a chain of models built around an action FM, a sequence model pretrained on raw user events. A content encoder embeds item text or media, the action FM outputs an actor embedding summarizing user behavior, and downstream models score live tasks. Before serving that chain online, you have to pretrain the action FM and evaluate its lift on historical data.
1. Self-supervised training and downstream integration
Action FMs learn behavioral representations from unlabeled event logs through self-supervised sequence objectives. Most teams pretrain with one of these objectives:
- Predicting the upcoming action: Given a user's actions up to step
k, predict actionk+1. The final hidden state summarizes likely upcoming behavior. - Masked-action prediction: Mask out a fraction of actions in the sequence and train the transformer to reconstruct them from surrounding actions.
- Contrastive objectives: Pull representations of two slices of the same user's history together while pushing other users apart. Teads ("Building a User Foundation Model for the Open Web," RecSys '26, arXiv 2607.28019) combined masked language modeling with a contrastive objective before fine-tuning for click prediction.
Downstream task models then need a way to consume that pretrained sequence model. Write-ups on Netflix's personalization foundation model compare three integration modes:
- Embeddings: Pass the action FM's actor embedding as an input feature into downstream models. Embeddings are the easiest mode to adopt across teams, though vectors go stale if refreshed only in daily batch jobs.
- Subgraph: Run part of the action FM inside the downstream model. Running a subgraph cuts staleness at the cost of more complex live inference.
- Fine-tuning: Adapt the action FM weights to one specific downstream task.
Because user behavior shifts over time, teams in the embedding setup retrain the action FM on a regular schedule. Each retrain produces a new snapshot (a model fixed at a specific code version and cutoff date), usually warm-started from the prior snapshot's weights. Skipping that time-stepped schedule during offline evaluation often produces a large backtest gain and almost no live A/B lift.
2. Peeking across a model chain
That offline-to-online gap often comes from the peeking problem, where an upstream model learned from logs after a training example's timestamp. For a counter like orders_last_30d, clamping the input window at timestamp t preserves point-in-time correctness (using only data from before t) because COUNT(*) has no learned parameters. An actor embedding at t depends on both the user's actions up to t and the weights that encode them. If those weights trained on logs after t, the model peeks at the future.
A checkout fraud example
Suppose a fraud team collects 12 months of user logs from January through December. The team trains one action FM on those 12 months of unlabeled actions to predict upcoming actions. Using that model, the team runs a backfill (recomputing historical features across past dates) for checkout fraud from March through November.
On a fraudulent checkout recorded May 10 at 14:00, the pipeline slices the user's actions strictly up to May 10 at 14:00. It runs the action FM on that slice and attaches the actor embedding to the training row. Offline precision and recall jump in the backtest, yet live fraud detection barely improves at launch.
What happened on May 11 shows why. After seeing the unauthorized charge, the real cardholder logged in, reset their password, ran account recovery, and opened a dispute. Because the action FM trained on the full year, gradient updates from that user's May 11 dispute shaped the weights used on May 10. On the full dataset, predicting upcoming actions also clustered pre-checkout browsing prefixes that preceded password resets and disputes near each other.
On the May 10 row, the input sequence ends at t, while the weights know what happened through t + 90d. The actor embedding at t carries traces of future disputes and password resets, a symptom called intent leakage. The downstream classifier reads that leaked signal offline and fails on live checkouts whose future has not happened yet.
Upstream encoders and peeking
Any upstream link that learns from internal logs can leak future signal the same way. Running a content encoder ahead of the action FM creates two distinct cases:
- A frozen off-the-shelf encoder does not peek. Calling a frozen public model like Vertex AI's
gemini-embedding-001on listing text is safe. Its vendor trained it without internal logs, and it never saw your users' future behavior. - A fine-tuned encoder can peek. Fine-tuning that encoder on 12 months of internal co-purchase logs teaches its weights year-end buying patterns. Running that full-year encoder on a May 10 row leaks future co-purchases into the training set.
Internal interaction logs carry future outcomes at any stage of the pipeline. Any model in the chain trained on your own data must learn only from events before each downstream example's timestamp.
_Any model trained on your logs must stop learning before t._3. Stopping leaks in backtests
Stopping weight and label leaks takes explicit time cutoffs on retraining schedules and on the input event stream. Both model snapshots and label windows must stop before future outcomes appear.
Walk-forward retraining
Any model that embeds an example at time t must train only on actions recorded before t. Teams approximate continuous retraining with walk-forward retraining across discrete periods:
- Train snapshot
fm@t2on actions fromt1tot2. Embed downstream rows in[t2, t3)withfm@t2. - Train snapshot
fm@t3on actions fromt1tot3. Embed downstream rows in[t3, t4)withfm@t3, and continue stepping ahead.
Rows in [t2, t3) read vectors from fm@t2, whose weights never saw any action after t2. When the chain includes a fine-tuned content encoder, its snapshots step ahead on that same schedule.
Expanding and sliding training windows
As walk-forward snapshots step forward, each training run needs a clear start boundary as well as a cutoff date. Teams choose between expanding windows and sliding windows:
- Expanding windows keep the start fixed at
t1([t1, t2],[t1, t3],[t1, t4]). Retaining full history covers rare items and sparse users better, though it raises compute cost and mixes in older behavior. - Sliding windows keep a fixed width (
[t1, t2],[t2, t3],[t3, t4]). Dropping older history caps training cost and adapts faster to recent behavior shifts.
Alongside window shape, teams choose the period length between t2 and t3. Shorter periods keep embeddings fresher, while longer periods need fewer training runs.
Outcome events and delayed fraud labels
Even with clean walk-forward snapshots, leaks still appear where the raw action stream meets downstream labels. Two rules protect that boundary:
- Drop or delay labels in disguise. Outcome events like chargebacks, refunds, cancellations, and account locks record the label inside the event stream. Drop them from the action FM's input vocabulary, or delay them until the label settles.
- Lag fraud windows for 60-to-90-day delays. Click and conversion labels settle within hours or days, whereas chargebacks and disputes take 60 to 90 days to settle. Before 60 to 90 days pass, many real fraud cases still look clean. The downstream fraud training window must lag behind the action FM cutoff by 60 to 90 days.
Keeping walk-forward backtests affordable
A walk-forward backtest runs one action FM training job per period. Two habits keep offline compute budgets practical:
- Warm-start from the previous snapshot. Initialize
fm@t3fromfm@t2and train on the new slice of actions, just like scheduled production retrains. Warm-starting cuts compute cost and keeps consecutive embedding spaces close. - Use coarser periods in backtests than online. Training monthly snapshots in a backtest while retraining weekly in production cuts backtest runs without reintroducing peeking.
4. Spotting hidden leaks
Standard SQL timestamp audits check input feature windows and miss leaks carried inside model weights. Four practical checks surface warning signs of peeking, though none of them proves leakage on its own:
- Compare full-history lift against walk-forward lift. Train a second action FM on January through June logs only, and compare July through September lift against the full-year model. A sharp drop points to peeking.
- Check if lift fades past the training cutoff. Score downstream examples week by week around one snapshot's cutoff date. If lift drops right after the cutoff compared to dates just before it, the weights memorized future trajectories.
- Audit the action vocabulary for outcome events. Inspect the event types fed into the action FM. Events like
dispute_opened,chargeback_filed,refund_issued, oraccount_lockedfire after the outcome and leak the label. - Bucket lift by user activity after
t. Group backtest rows by how many actions each user took after timestampt. Gains concentrated among users with heavy post-tactivity suggest the weights leaked that future behavior.
Walk-forward snapshots keep offline evaluations accurate and produce time-pinned models ready for live traffic. Part 3: Serving moves this chain online with write-time streaming updates, snapshot versioning (fm@v1.t2), and atomic model promotion.