Giga Tile Design
Problem
The mega tile architecture (see megatile.md) reduces KV reads from O(N tiles) to 3 point gets, but the fetcher still does work on every read: decode batch IR + decode today/yesterday streaming entries + merge collapsed + scan tail hops + merge streaming + finalize. This compute happens on every feature request.
Goal
Push the merge to the write path. Flink produces a fully finalized feature vector per entity. The fetcher does a single point get with zero compute.
Current (mega tile, pull-based):
Fetcher: 3 KV gets → decode → merge batch + streaming → finalize → return
Proposed (giga tile, push-based):
Flink: on event → merge batch + streaming → finalize → KV write
Fetcher: 1 KV get → returnKey Insight: Hop-Based Pruning
The FinalBatchIr structure stores tail hops as individual hop-aligned partial aggregates:
collapsed: aggregate of [alignedCollapsedBoundary, batchEnd)
tailHops[hopIndex]: time-sorted array of [baseIr_0, ..., hopStartTs]mergeTailHops selects relevant hops using queryTail = round(queryTs - windowMillis, hopSize).
As queryTs advances, queryTail advances, and older hops are excluded (not subtracted).
This gives us "invertibility for free" — even for non-invertible aggregations like MIN, MAX, FIRST,
LAST. We simply re-run mergeTailHops with the current queryTs and get the correct answer.
Architecture
Iceberg (batch IR, written daily by GroupByUpload)
│
│ Flink reads periodically (connected keyed stream)
▼
Flink state per entity:
batchIr: FinalBatchIr // latest batch IR (collapsed + tail hops)
batchEndTs: Long // when batch was last computed
runningLargeIr: Array[Any] // fully merged: batch + streaming for large window cols
largeTodayIr: Array[Any] // today's streaming delta (for rotation tracking)
largeYesterdayIr: Array[Any] // yesterday's streaming delta
tiles + cachedSmallWindowIr // (existing mega tile state for small windows)
│
│ On event: update streaming state + running sum → emit finalized vector
▼
KV Store: (entity) → finalized feature vector
│
│ Single point get
▼
Fetcher: read → returnBatch IR Ingestion
Flink reads the batch IR from the existing GroupByUpload Iceberg table — the same table that Spark already writes to. No new tables, no new Spark changes.
GroupByUpload (Spark)
│ Writes partition ds=YYYY-MM-DD to Iceberg upload table
│ Columns: key_bytes, value_bytes, key_json, value_json, ds
│ Iceberg commits a new snapshot on write
│
▼
UploadTableSource (FLIP-27)
│ Enumerator on the JobManager: polls the table every monitorInterval, plans splits for
│ partitions above the committed floor (latest ds on cold start), assigns one split per
│ reader request. Readers: Iceberg's IcebergSourceReader, one row per pollNext.
│
│ Decode key_bytes → entityKey
│ keyBy(entityKey) → network shuffle to correct task slot
│
▼
CoProcessFunction (same task slot as Kafka events for this entity)
│ processElement1: Kafka event → onEvent
│ processElement2: batch IR row → onBatchUpdateStream 1: real events (Kafka) → keyBy(entityKey) ──┐
├→ CoProcessFunction
Stream 2: batch IR (Iceberg upload table, monitor mode) → keyBy(entityKey) ──┘Why this works without any Spark changes:
- GroupByUpload already writes
(key_bytes, value_bytes)to an Iceberg table partitioned byds - Each write commits an Iceberg snapshot (standard Iceberg behavior)
- Flink's Iceberg source detects new snapshots and reads the new data
value_bytescontains Avro-encodedFinalBatchIr— same format the fetcher reads todaykey_bytescontains Avro-encoded entity keys — same encoding as the Kafka event keys
Key superset: The Iceberg source emits ALL entities from the batch table. Entities that only exist in batch (no streaming events) get a Flink key slot via the batch stream. This solves the key superset problem without requiring Kafka events for every entity.
Checkpointing during the batch read. The upload table is read by UploadTableSource, a
FLIP-27 source. UploadTablePartitionEnumerator runs on the JobManager, polls for new ds partitions and hands out IcebergSourceSplits as readers ask for them; nothing is pushed into
network buffers, so the enumerator has no checkpoint lock to block on. Readers are Iceberg's IcebergSourceReader, which fetches a split on a background thread and emits one row per pollNext, so a checkpoint barrier waits for at most one row rather than a whole file, and a
restore resumes mid-split from the checkpointed (fileOffset, recordOffset). The partition
floor is committed in notifyCheckpointComplete: a checkpoint taken after a partition is planned
covers all of it (unassigned splits in enumerator state, assigned ones in reader state), and a
crash before completion re-plans the partition, which onBatchUpdate dedupes.
The earlier planner/flatMap reader topology emitted scan tasks under the source's checkpoint
lock and blocked there under backpressure; on a large upload table no checkpoint could be
injected until the readers had drained most of the partition, and the reader passed a barrier
only between files.
Knobs: iceberg.reader.parallelism (source parallelism, sets GigaTile width), iceberg.monitor.interval.minutes, iceberg.split.size.bytes (assignment granularity only), iceberg.catalog.credential-renew-minutes (default 40, positive whole minutes): the source re-opens
the catalog session and its FileIO on that interval, so a REST catalog's OAuth2 token and the vended
S3 credentials baked into the FileIO are refreshed well inside their lifetime. Needed on Iceberg
1.6.1, whose bundled runtime only refreshes that token through the token-exchange grant; raise it on
a catalog that issues longer-lived tokens, lower it on shorter ones.
Upgrading a job from the planner/reader topology: the source uid changed to iceberg-upload-source-<gb>, so set execution.state-recovery.ignore-unclaimed-state: true for
the first deploy; GigaTile state restores and the batch source cold-starts the latest partition.
The enumerator loads the Iceberg catalog on the JobManager; budget JobManager metaspace
accordingly (the 256m Flink default was not enough with a REST catalog and S3 client).
State Layout
Extends the existing mega tile TileStore with batch-side state:
// Existing (mega tile, small windows):
tiles: MapState // per-tier base IRs for small window columns
cachedSmallWindowIr: ValueState // sawtooth running sum, corrected on eviction
// Existing (mega tile, large window streaming delta):
largeTodayIr: ValueState // daily accumulator [todayStart, now)
largeYesterdayIr: ValueState // daily accumulator [yesterdayStart, todayStart)
// New (giga tile, batch + merged):
batchIr: ValueState // FinalBatchIr (collapsed + tail hops)
batchEndTs: ValueState // Long
runningLargeIr: ValueState // fully merged large window IR (batch + streaming)
// Bookkeeping:
currentDayStart, earliestTileStart // (existing)Processing Logic
On Event (hot path)
Same cost as mega tile — O(1) per event for large windows, O(tiers) for small windows.
def onEvent(row, eventTs):
// Small windows: unchanged from mega tile
updateTiles(row, eventTs)
updateCachedSmallWindowIr(row)
// Large windows: update daily accumulator AND running sum
if eventTs >= todayStart:
updateLargeWindowColumns(largeTodayIr, row)
updateLargeWindowColumns(runningLargeIr, row) // running sum stays current
else if eventTs >= yesterdayStart:
updateLargeWindowColumns(largeYesterdayIr, row)
updateLargeWindowColumns(runningLargeIr, row) // late event also merged into running sum
// Emit finalized feature vector
emit(finalize(pack(cachedSmallWindowIr, runningLargeIr)))The runningLargeIr is the fully merged value: batch collapsed + relevant tail hops + all
streaming events. Updated incrementally on each event. Sawtooth at the tail — corrected on eviction.
On Eviction (periodic, every minTileSize)
Corrects both small window sawtooth and large window tail hop selection.
def onEviction(timerTs):
// Small windows: rebuild from tiles (existing mega tile logic)
evictStaleTiles(timerTs)
cachedSmallWindowIr = buildMegaTileIr(tiles, timerTs, todayStart)
// Large windows: recompute running sum from batch + streaming.
// queryTs advanced → tail hops may have shifted (oldest hop excluded).
if batchIr != null:
runningLargeIr = clone(batchIr.collapsed)
mergeTailHops(runningLargeIr, queryTs=timerTs, batchEndTs=batchEndTs, batchIr)
else:
// No batch yet (new entity, batch hasn't run). Streaming-only.
runningLargeIr = windowedAgg.init
for col where !isNoBatch(col):
runningLargeIr(col) = merge(runningLargeIr(col), largeTodayIr(col))
if batchEndTs < todayStart:
runningLargeIr(col) = merge(runningLargeIr(col), largeYesterdayIr(col))
emit(finalize(pack(cachedSmallWindowIr, runningLargeIr)))On Batch IR Update (daily, from Iceberg stream)
Stores the new batch IR immediately. Recomputation of runningLargeIr is deferred if the
watermark hasn't caught up to the new batchEnd (prevents double-counting from overlap
between batch and largeTodayIr). The next eviction picks it up.
def onBatchUpdate(newBatchIr, newBatchEnd):
oldBatchEnd = batchEndTs
if newBatchEnd <= oldBatchEnd: return // same or older batch, skip
// Always store the new batch IR — eviction and future events use it.
batchIr = newBatchIr
batchEndTs = newBatchEnd
// Register eviction timer unconditionally — needed for:
// - batch-only entities (no streaming events to trigger it)
// - large-windows-only GroupBys (no small window tiles to drive eviction)
// minEvictionInterval = min(activeTiers) — the smallest hop size across ALL windows.
// Matches hop granularity so tail hop corrections fire at the right cadence.
registerEvictionTimer(now + minEvictionInterval)
if newBatchEnd > currentDayStart:
if currentDayStart < 0:
// Uninitialized (no events yet). Safe to set from batch — largeTodayIr is init,
// no overlap concern. Without this, all startup batch loads would defer.
currentDayStart = newBatchEnd
// fall through to recomputation
else:
// Real defer: watermark hasn't caught up. largeTodayIr has events that overlap
// with batch. Wait for advanceWatermark to rotate, then eviction recomputes.
return
// Safe: newBatchEnd <= currentDayStart — no overlap between batch and largeTodayIr.
// Save old running sum for comparison
oldRunningLargeIr = clone(runningLargeIr)
// Clear yesterday if batch now covers it
if newBatchEnd >= currentDayStart:
largeYesterdayIr = init
// Recompute running sum: new batch + tail hops + streaming
runningLargeIr = clone(newBatchIr.collapsed)
mergeTailHops(runningLargeIr, queryTs=watermark, batchEndTs=newBatchEnd, newBatchIr)
for col where !isNoBatch(col):
runningLargeIr(col) = merge(runningLargeIr(col), largeTodayIr(col))
// Include yesterday if batch doesn't cover it
if newBatchEnd < currentDayStart and largeYesterdayIr(col) != null:
runningLargeIr(col) = merge(runningLargeIr(col), largeYesterdayIr(col))
// Emit only on mismatch — most entities won't change materially
if !equal(oldRunningLargeIr, runningLargeIr):
emit(finalize(pack(cachedSmallWindowIr, runningLargeIr)))Why defer when newBatchEnd > currentDayStart?
largeTodayIr covers [currentDayStart, now). If newBatchEnd > currentDayStart, batch covers [..., newBatchEnd) which overlaps with [currentDayStart, newBatchEnd) in largeTodayIr.
We can't subtract the overlap (non-invertible aggregations). We can't clear largeTodayIr (loses post-batchEnd events). So we wait for the watermark to advance past newBatchEnd,
which rotates largeTodayIr via advanceWatermark. The next eviction then recomputes cleanly.
In normal operation, this defer never triggers: batch lands at ~6 AM, watermark passed midnight
hours ago, newBatchEnd = currentDayStart. The defer is a safety net for the narrow race window
when batch arrives just before the watermark crosses midnight.
Day Transition (advanceWatermark)
Same as mega tile, plus runningLargeIr gets yesterday cleared.
def advanceWatermark(watermarkTs):
wmDay = round(watermarkTs, DayMillis)
if wmDay > currentDayStart:
// Rotate yesterday → discard (or keep if single-day hop, clear if multi-day)
largeYesterdayIr = if (wmDay == currentDayStart + DayMillis) largeTodayIr else init
largeTodayIr = init
currentDayStart = wmDay
// runningLargeIr is NOT reset here — it's a cumulative sum.
// Yesterday's data is still valid in the running sum.
// The eviction timer will re-run mergeTailHops with the new queryTs
// to prune any tail hops that fell off. Between the day transition
// and the next eviction, the running sum is slightly over-inclusive
// at the tail (sawtooth).Emit and KV Key
The emitted value is a finalized feature vector — the same format that the fetcher would return to the ML model. No further processing needed.
KV dataset: {GROUP_BY_NAME}_PUSH (not _STREAMING)
KV key: entityKeyBytes (plain entity key, no TileKey wrapper, no day suffix)
KV value: finalized feature vector (Avro encoded output schema)Single entry per entity. Overwritten on every emit. The fetcher reads one key, decodes, returns.
Why a dedicated _PUSH dataset
Push writes use simple key→value semantics (one value per entity, overwritten on every emit).
The existing _STREAMING dataset has time-series semantics baked into all KV implementations:
| Implementation | _STREAMING behavior | Problem for push |
|---|---|---|
| DynamoDB | Composite key (partition + sort key = timestamp) | GetItem on plain entity key fails — sort key not specified |
| BigTable | Day-based row key generation for time-series reads | Won't find rows written with plain entity keys |
| Redis | Sorted sets (zadd/zrangebyscore) | Each timestamp creates a new member — unbounded growth |
The _PUSH dataset avoids all of this:
| Implementation | _PUSH behavior | Why it works |
|---|---|---|
| DynamoDB | Partition key only (no sort key) | PutItem overwrites; GetItem returns single value |
| BigTable | Non-time-series; cellsPerRow(1) GC | setCell overwrites; read returns latest cell |
| Redis | Simple setex/get | Overwrites on every write; single value per key |
| Cosmos | Batch-style upsertItem by key hash | Document ID has no timestamp; upsert overwrites |
The _PUSH suffix is chosen intentionally: DynamoDB's isStreamingTable check (dataset.endsWith("_STREAMING"))
determines whether to add a sort key. _PUSH doesn't match, so the table is created as a simple key-value store.
Planner and upload changes
For OnlineStrategy.PUSH GroupBys:
- Planner: The
uploadToKVNodeis excluded from the plan —KVUploadNodeRunner(bulkPut of entity rows) is not scheduled. Flink reads entity batch IRs directly from the Iceberg upload table. - GroupByUpload: Writes
GroupByServingInfodirectly to KV viakvStore.put()at the end of the upload job. This replaces theKVUploadNodeRunnerpath for the metadata row. The serving info is already in memory frombuildServingInfo()— no extra table read needed. - Upload table: Still receives both entity rows and the serving info row (unchanged). Flink's Iceberg source reads entity rows from here.
Scenario Tables
All windows, tailBuffer = 2d, now = Mar 26 14:00.
Scenario 1: Batch fresh (batchEnd = Mar 26 00:00, 14h stale)
Streaming covers [batchEnd, now) = [Mar 26 00:00, Mar 26 14:00) = 14h. largeTodayIr covers [Mar 26 00:00, Mar 26 14:00). largeYesterdayIr is empty (batch is fresh).
| window | category | runningLargeIr composition | tail hops selected | streaming added |
|---|---|---|---|---|
| 6h | SMALL | — (uses cachedSmallWindowIr) | — | — |
| 1d | SMALL | — | — | — |
| 2d | SMALL | — | — | — |
| 49h | LARGE | collapsed [Mar 24 23:00, Mar 26 00:00) + 1 hop [Mar 24 01:00, Mar 24 23:00) | hops where hopStart >= round(Mar 26 14:00 - 49h) = Mar 24 13:00 | + largeTodayIr [Mar 26 00:00, 14:00) |
| 3d | LARGE | collapsed [Mar 24 00:00, Mar 26 00:00) + 24 hops [Mar 23 00:00, Mar 24 00:00) | hops where hopStart >= round(Mar 26 14:00 - 3d) = Mar 23 14:00 | + largeTodayIr |
| 7d | LARGE | collapsed [Mar 20 00:00, Mar 26 00:00) + 48 hops [Mar 19 00:00, Mar 20 00:00) | hops where hopStart >= round(Mar 26 14:00 - 7d) = Mar 19 14:00 | + largeTodayIr |
On event at Mar 26 14:00:
runningLargeIr(49h)= batch portion (collapsed + selected hops) +largeTodayIrcontribution + new event- Emit:
finalize(pack(cachedSmallWindowIr, runningLargeIr))→ single KV write
On eviction at Mar 26 14:05:
queryTsadvanced by 5min →queryTailfor 49h advances → no hop falls off (1hr hops)queryTailfor 3d advances → no hop falls offrunningLargeIrrecomputed from batch hops + streaming. Same value (no tail shift). No-op.
Scenario 2: Batch stale (batchEnd = Mar 25 00:00, 38h stale)
Streaming covers [batchEnd, now) = [Mar 25 00:00, Mar 26 14:00) = 38h. largeYesterdayIr covers [Mar 25 00:00, Mar 26 00:00). largeTodayIr covers [Mar 26 00:00, Mar 26 14:00).
Both are included because batchEnd < todayStart.
| window | category | runningLargeIr composition | tail hops selected | streaming added |
|---|---|---|---|---|
| 6h | SMALL | — | — | — |
| 1d | SMALL | — | — | — |
| 2d | SMALL | — | — | — |
| 49h | LARGE | collapsed [Mar 22 23:00, Mar 25 00:00) + hops | hops where hopStart >= Mar 24 13:00 | + largeYesterdayIr + largeTodayIr |
| 3d | LARGE | collapsed [Mar 23 00:00, Mar 25 00:00) + hops | hops where hopStart >= Mar 23 14:00 | + largeYesterdayIr + largeTodayIr |
| 7d | LARGE | collapsed [Mar 19 00:00, Mar 25 00:00) + hops | hops where hopStart >= Mar 19 14:00 | + largeYesterdayIr + largeTodayIr |
On batch refresh (batchEnd advances Mar 25 → Mar 26 00:00):
- Store new batchIr (collapsed now covers [Mar 20 00:00, Mar 26 00:00) for 7d window)
- Clear
largeYesterdayIr— batch now covers [Mar 25 00:00, Mar 26 00:00) - Recompute
runningLargeIr:- 49h: new collapsed + selected hops + largeTodayIr only (yesterday cleared)
- 3d: new collapsed + selected hops + largeTodayIr only
- Compare with old
runningLargeIr - Emit only if mismatch (likely: batch incorporated Mar 25's full data, replacing streaming's accumulation)
State transitions through a day
Time Event runningLargeIr state
─────────── ─────────────────────── ──────────────────────────────────────────────
Mar 26 00:00 advanceWatermark yesterday→today rotation. Running sum unchanged
(day transition) (sawtooth: slightly over-inclusive at tail)
Mar 26 00:05 eviction timer Recompute from batch hops + streaming.
Tail hops re-selected with queryTs=00:05.
Running sum corrected.
Mar 26 00:05 event arrives Merge into largeTodayIr + runningLargeIr.
to 06:00 (continuous) Emit finalized vector on each event.
Mar 26 06:00 batch refresh New batchIr loaded from Iceberg.
(Iceberg snapshot) Clear largeYesterdayIr.
Recompute running sum.
Emit only if value changed.
Mar 26 06:00 events continue runningLargeIr updated incrementally.
to 24:00 Eviction corrects tail every minTileSize.Comparison with Mega Tile
| Mega tile (current) | Giga tile (proposed) | |
|---|---|---|
| KV reads per query | 3 (today + yesterday + batch) | 1 |
| Fetcher compute | decode + merge + finalize | decode only |
| Flink state | ~17 KB/entity | ~19 KB/entity (+batchIr ~1-2 KB) |
| KV writes per event | 1 (streaming entry) | 1 (finalized vector) |
| Batch data in Flink | No | Yes (via Iceberg connected stream) |
| Cold entity serving | Fetcher reads batch KV | Flink emits on batch load |
| Batch correction | Implicit (fetcher merges latest) | Flink re-merges, emits on mismatch |
Cost Model
Per event (hot path):
- Small windows: ~6 codec ops (unchanged from mega tile)
- Large windows: 1 additional
updateLargeWindowColumns(runningLargeIr, row)— same column aggregator update as largeTodayIr. Negligible. - Emit: encode finalized vector instead of windowed IR. Similar cost.
Per eviction (every minTileSize):
- Small windows: rebuild from tiles (unchanged)
- Large windows:
clone(collapsed) + mergeTailHops + merge(streaming)— O(tailHops × columns). For 48 hops × 7 columns = 336 merge ops per eviction. At 5min cadence, that's ~1.1 merges/sec. Negligible.
Per batch refresh (daily):
- Recompute
runningLargeIrfor all entities. O(entities × tailHops × columns). - Mismatch check + conditional emit. Most entities won't emit.
- Spread over the Iceberg scan duration (~minutes). Not bursty.
Bootstrapping and Startup
Emit strategy
Flink continues rebuilding state during replay, but PUSH writes are fenced until the connected
watermark is near wall clock. A batch source that is still active can hold that watermark at MIN; source idleness lets it advance but does not prove the initial scan completed. An optional
write cadence can further coalesce hot-key updates after the fence opens.
Emit triggers:
- Event arrival: update state and publish only when the replay fence is open.
- Eviction timer: rebuild expired state and publish a changed or deferred snapshot.
- Batch IR update: emit if
runningLargeIrchanged. Single rule covers null→value (first batch load), batch correction, tail shift, no-change skip.
def onBatchUpdate(newBatchIr, newBatchEnd):
// ... store batchIr, register timer, defer if overlap (see detailed pseudocode above) ...
oldRunningLargeIr = clone(runningLargeIr)
// ... recompute runningLargeIr from batch + hops + streaming ...
if !equal(oldRunningLargeIr, runningLargeIr):
emit(finalize(pack(cachedSmallWindowIr, runningLargeIr)))Readiness
The implementation has no positive initial-Iceberg-scan-complete signal. A near-live connected watermark proves Kafka catch-up, not batch coverage. Rollout tooling must validate batch input and serving output separately before routing traffic.
Kafka replay on cold start
On cold start (no checkpoint), Flink rebuilds batch and streaming state from the configured
sources. Serving writes remain fenced until the connected watermark reaches Live.
Cold start / first startup:
1. Flink starts with no state
2. Iceberg source scans upload table (latest ds partition)
→ rows populate batch state as they arrive
3. Kafka replay rebuilds retained streaming state
4. Connected watermark reaches `Live`
→ the latest deferred snapshot can publish (subject to optional cadence)
5. Rollout tooling validates batch coverage and serving output before routing trafficKafka retention requirement
Kafka retention must cover the actual gap from batchEnd through replay plus the required
correctness horizon. For GroupBys with only small windows, it must also cover the maximum
window size.
Key completeness
Will all keys be in the KV store after startup?
Yes, for all temporal entities:
- Inactive entities (in batch table, no recent streaming events): Iceberg source loads their batch IR; the batch-only vector can publish after the live fence opens.
- Active entities: Iceberg and Kafka replay rebuild one deferred snapshot, which can publish after the live fence opens.
- New entities (in Kafka, not in batch): first event creates Flink state. Columns that require batch history remain unset until a batch row arrives, unless the batch-absent fallback below is explicitly enabled.
gigatile_first_seen_key_grace_millis enables the batch-absent fallback when set to a
positive value; it is disabled by default. The operator-local grace starts on the first keyed
callback after open or restore. Its deadline is a lower bound: the fallback installs on the next
eligible keyed callback, which may be the next 5m, 1h, or 1d eviction for a sparse key. A missing
row may then be treated as empty history, while a later real batch row remains authoritative.
The grace is an operational assertion, not proof that the initial Iceberg scan completed.
Enabling it therefore requires a configured Iceberg source and fails the job on source
construction or entity-row decode/update errors; the serving-info metadata row is ignored.
To roll back this option, fence serving first, disable it on the current binary, and keep serving fenced until null corrections are verified in KV and a later checkpoint completes. Only then roll back the binary or unfence. Correction waits for a live watermark and the affected key's next eviction callback (5m, 1h, or 1d), so binary rollback alone is not an immediate retraction.
Note: Giga tile only applies to Accuracy.TEMPORAL GroupBys (with a streaming topic). SNAPSHOT GroupBys bypass Flink — they use the traditional bulkPut from Spark to KV.
Entity behavior
- Active: combine batch state with replayed and live events; publish after the live fence opens.
- Batch-only: load and publish batch state after the fence opens; no per-key event is required.
- New: retain streaming state, but leave batch-backed values unset until a real batch row arrives or the explicit grace installs an empty baseline.
Daily state remains keyed by day and is pruned once covered by batch or outside every finite window.
Batch refresh (onBatchUpdate) edge cases
| Scenario | Guard | Behavior |
|---|---|---|
newBatchEnd <= oldBatchEnd | Early return | Skip (same or older batch) |
currentDayStart < 0 (uninitialized, no events) | Init currentDayStart = newBatchEnd | Safe: no events → no overlap. Recompute and emit. |
newBatchEnd > currentDayStart (batch ahead of the materialized horizon) | Defer (return) | Store batchIr but don't recompute. The next callback retries after the selected horizon advances. |
newBatchEnd <= currentDayStart (normal) | Prune slots with dayStart < batchEndDay | Recompute from the new batch plus uncovered daily slots. |
| Stale batch catch-up | Retain slots at or after batchEndDay | Uncovered streaming days remain in the recomputed running view. |
Eviction edge cases
| Scenario | Behavior |
|---|---|
batchIr = null (new entity, type C) | Retain streaming state, but keep batch-backed output unset unless the explicit grace is active. |
batchIr != null (types A, B) | Rebuild runningLargeIr from batch collapsed/tail hops plus retained daily slots. |
earliestTileStart = MaxValue (no tiles, large-windows-only) | Skip tile eviction. Still recompute runningLargeIr from batch hops. |
Batch-only entity (type B), no hasSmallWindows | Timer registered unconditionally by onBatchUpdate. Eviction fires at minEvictionInterval. |
Implementation Path
Building on the existing mega tile infrastructure:
Extend TileStore with batch state
- Add
getBatchIr/putBatchIr,getBatchEndTs/putBatchEndTs,getRunningLargeIr/putRunningLargeIr - InMemoryTileStore + FlinkTileStore implementations
- Add
Extend MegaTileStreamProcessor → GigaTileStreamProcessor
- Add
onBatchUpdate(newBatchIr, newBatchEnd)method - Modify eviction to recompute
runningLargeIrfrom batch hops - Emit finalized vectors instead of windowed IRs
- Add
Add Iceberg connected stream to Flink job
CoProcessFunctionhandling both event stream and batch IR stream- Periodic Iceberg snapshot monitoring (every 30 min)
Simplify fetcher path
- Single point get on entity key
- Decode finalized vector → return
- No merge logic needed
Integration test
- Traffic replay: events + batch uploads + queries, time-ordered
- Compare push-based results with backfilled results via
.diff
Steps 1-2 are pure aggregator/online changes (testable without Flink/Spark). Step 3 is Flink wiring. Steps 4-5 are cleanup and validation.