Foundation models for real-time actor embeddings
Action foundation models summarize ordered user actions into one shared vector that captures live intent across downstream tasks.
Part 1 of 4 in our series on action foundation models: Part 1: Foundations · Part 2: Training · Part 3: Serving · Part 4: In Zipline.AI
A user's actions across a site or app show what they intend to do. What someone searches for, edits in settings, or buys in sequence carries more signal than static profiles or 30-day counters.
Take two users who each trigger five page views and one payment attempt in ten minutes. User A searches for a flight, compares three return dates, reads the baggage policy, and enters a card. User B lands on a login page, changes the account email, views two gift cards, and tries to check out.
A table of 10-minute counts (views_last_10m = 5, checkouts_last_10m = 1) makes those two sessions look identical. Each counter sums events inside a time bucket and throws away event order. The raw sequence separates a normal traveler from an account takeover right away.
1. Learning intent from action sequences
Teams now train action foundation models (action FMs) on raw user event streams to capture that sequence signal without hand-built rules. An action FM is a sequence model, usually a transformer, pretrained without task labels on ordered user events. Its output is an actor embedding: a numeric vector that summarizes a user's recent behavior and intent.
One actor embedding serves many downstream models, the task-specific models that score live decisions. On the two sessions above, a checkout fraud model reads the actor embedding to block User B's gift-card purchase before money leaves. A search ranker reads User A's embedding to surface return flights within seconds as their dates narrow. If User A hits three payment errors at checkout, a support router can send them straight to a billing specialist.
_One actor embedding per user, shared across downstream models._One pretrained sequence model also replaces duplicate feature work across teams. In a classic setup, separate teams each build hundreds of SQL counters over the same clickstream, whereas an action FM learns from that stream once. Supervised models learn only from labeled purchases or chargebacks, while an action FM pretrains without labels on active users. Downstream services then read that single actor embedding alongside their own tabular features.
2. Published results
Several consumer and fintech companies have published production results on sequence foundation models. A recent benchmark also studies how these models scale with data and compute:
- Pinterest (PinFM, RecSys 2025): Pretrains a transformer on user activity sequences. Downstream ranking models reuse the pretrained transformer and embedding table alongside their existing features.
- Netflix (personalization foundation model write-up): Trains one transformer on interaction sequences to represent user behavior across many homepage models.
- Airbnb (JourneyFormer, arXiv 2606.19108 and Airbnb Tech Blog): Trains a transformer over the full guest journey for search ranking. The ranker reads a stored guest embedding at query time.
- Teads ("Building a User Foundation Model for the Open Web," RecSys '26, arXiv 2607.28019): Pretrains on browsing histories with masked language modeling and a contrastive objective. Teads fine-tunes that pretrained model for click prediction.
- Revolut (PRAGMA transaction foundation model): Reported at 10M, 100M, and 1B parameters. In that report, PRAGMA improved fraud-detection precision by 21%.
- Compute and data scaling (Scaling Laws for Behavioral Foundation Models over User Event Sequences, arXiv 2606.05257): Runs roughly 600 training jobs on real interaction data to study compute and data budget splits.
These reports span homepage ranking, search, ad click prediction, and fraud detection. In each case, one pretrained sequence representation feeds downstream task models.
3. Adding content encoders for cold-start items and text
Those systems learn strong behavioral patterns from known item IDs, yet bare IDs break down on new inventory and unstructured media. In many logs, an event records only an action type and an identifier, like viewed item #123. When item #123 is a brand-new marketplace listing, the action FM never saw that ID during training. A bare identifier tells the model nothing about what the user viewed.
Many actions also carry unstructured content such as listing photos, product text, reviews, voice notes, or messages. Off-the-shelf content encoders, such as pretrained text or image embedding models, convert that raw content into a dense vector. When you attach that vector to the event, the action FM reads item meaning alongside its ID. Even on a brand-new listing with zero prior clicks, the sequence model places the view in context.
Attaching content vectors to user sequences places the action FM in the middle of a three-model chain. Each stage feeds its output into the stage below it:
- Content encoder: An off-the-shelf model embeds the text, image, or audio attached to each item or message.
- Action FM: A sequence model reads the user's recent actions, each carrying its content vector, and outputs an actor embedding.
- Downstream model: A task model combines the actor embedding with tabular features to score a live event.
Running that upstream encoder inside the feature pipeline keeps content vectors aligned with user events. In Zipline.AI, ModelTransforms calls off-the-shelf encoders like Vertex AI's gemini-embedding-001 on listing text and joins the vector onto each action.
4. Four production guarantees for a model chain
Chaining models together multiplies the ways offline training and live serving can drift apart. Zipline.AI provides four system guarantees to run this chain safely in production.
Point-in-time correctness for features and weights
Model weights are features too, and each historical training example must be point-in-time correct; it can use only data from before its timestamp. During a backfill (recomputing historical features across past dates for a training set), clamping event timestamps is only half the job. An action FM trained on full history still leaks future actions into past rows through its weights. Zipline.AI pairs point-in-time feature windows with time-pinned model snapshots (models fixed at a specific code version and cutoff date).
Training-serving consistency from one definition
Time-bounded training sets still fail online if live serving computes slightly different inputs. A model chain breaks if offline training runs one set of scripts while live serving runs another. Small differences in slicing the last 128 actions or joining item vectors will skew live scores. In Zipline.AI, you declare each aggregation, join, and model call once in Python, and Zipline.AI compiles it into Spark backfills, Flink jobs, and key-value lookups.
Versioned, surgical recomputes
Downstream training tables often pair an actor embedding with hundreds of tabular columns. Recomputing unchanged columns whenever you update a content encoder or bump your action FM version wastes time and money. Zipline.AI puts the version in output column names and tracks a semantic hash (a fingerprint of the full column-level definition) per feature. Backfills reuse unchanged tabular columns and recompute only columns downstream of the changed model.
Isolated branches with atomic release
Swapping one model in a live chain breaks downstream models that still expect the old embedding space. Zipline.AI deploys each new version from a git branch with isolated batch jobs, streaming pipelines, serving indices, and schedules. Teams shadow the branch (scoring live traffic and logging predictions without acting on them) before promoting the new snapshot, stored user embeddings, and downstream model at the same instant.
_What Zipline.AI provides for a model chain._Point-in-time correctness comes first during offline development, before any model reaches live traffic. Part 2: Training looks at self-supervised pretraining objectives, weight leakage in model chains, and walk-forward retraining.