Changelog

What's Changed

Minor features

• Wire ModelTransforms-source GroupBys correctly into the DAG by @piyush-zlai in #2187
• Emit join-level fetch null/total counters without the feature tag by @sean-zlai in #2257
• Synchronize Teams for UI-managed webhooks by @david-zlai in #2049

Bug fixes

• Authenticate Redis bulk loader from executor environment by @chewy-zlai in #2225
• Say "not logged in" when a hub answers with its login page by @diya-zipline in #2240
• Stop quoting {VAR} placeholders in EMR Serverless overflow --conf flags by @kumar-zlai in #2256

Full Changelog: v1.27.1...v1.27.2

What's Changed

Minor features

• Give the Ray trainer structured logs and a pass/fail bar by @diya-zipline in #2226
• [Online] Serve joins whose left is a JoinSource by @BenMagyar in #2242
• Run monolith join parts N at a time (spark.chronon.join.part_parallelism) by @BenMagyar in #2245

Bug fixes

• Point crucible at its own Databricks secrets, not canary's by @diya-zipline in #2235
• Snap a bare --end-ds to the end of the day in zipline hub by @sean-zlai in #2224
• Avoid dropping tables for nested nullability-only schema changes by @varant-zlai in #2244
• Fix Kinesis connector lz4-java vulnerabilities by @cristian-zlai in #2243
• Make the Instacart model train end to end on crucible Ray, with in-job secrets by @diya-zipline in #2238
• Update Flink pkg deps to fix crucible gigatile Iceberg CNF by @piyush-zlai in #2247
• Fix computation range in the engine when recompute_days is set by @kumar-zlai in #2248
• Check pushed state against the branch's upstream, not its name by @diya-zipline in #2239

Full Changelog: v1.27.0...v1.27.1

What's Changed

Minor features

• Give agents verbs for trials, and teach them not to self-report metrics by @diya-zipline in #2204
• Add an Instacart pipeline, so a measured score can move for a reason by @diya-zipline in #2207
• [Redis writes 2/4] Add durable state and mutation primitives by @BenMagyar in #2136

Bug fixes

• feat: skills-eval for agent, use it to reduce size of skills by @cristian-zlai in #2195
• Give the GigaTile Iceberg source its RowData type information by @BenMagyar in #2216
• grype: ignore GHSA-9437-39hj-3c93 (Spark History Server XSS) until Spark upgraded by @piyush-zlai in #2223
• Giga tile Flink: multi-catalog Iceberg source + overwrite-aware polling by @piyush-zlai in #2222
• Extend Py GroupBy configs to support setting gigatile mode + example confs by @piyush-zlai in #2221
• fix: bump spark4 jackson to 2.21.7 (GHSA-p6pp-m3f8-5c89, GHSA-cxp5-3px4-pw24) by @piyush-zlai in #2228
• feat(cli): skip sync by @cristian-zlai in #2227

Full Changelog: v1.26.0...v1.27.0

What's Changed

Minor features

• Label the CTR example with what happened, not with rand() by @diya-zipline in #2209
• Resolve secrets in the job through vault refs by @kumar-zlai in #2211

Bug fixes

• Put the CTR model in the team its training data belongs to by @diya-zipline in #2203
• Branching and deployment flow doc by @piyush-zlai in #2180
• fix: bump jackson-databind to 2.18.10 (GHSA-q4xh-88c3-wmh7 et al) by @piyush-zlai in #2212
• fix: grype spark4 — jackson 2.21.6, log4j-layout-template-json pin, strip org.jline pom metadata by @piyush-zlai in #2213

Full Changelog: v1.25.0...v1.26.0

What's Changed

Bug fixes

• fix(integration): use raw_decode to tolerate trailing output after workflowId JSON by @piyush-zlai in #2205
• Share sensor compute across ranges on the same grid by @tchow-zlai in #2162

Full Changelog: v1.24.1...v1.25.0

What's Changed

Bug fixes

• Avoid per-request GroupBy deep copy in GroupByFetcher by @BenMagyar in #2194
• Name the trainer example for the business, not the runtime by @diya-zipline in #2193
• Revert trigger fire-on-window-close from #1575 by @piyush-zlai in #2201
• Set up guardrail around flagging unpushed changes by @piyush-zlai in #2199

Full Changelog: v1.24.0...v1.24.1

What's Changed

Minor features

• Add streaming Flink runtime for ModelTransforms-source GroupBys by @piyush-zlai in #2186

Bug fixes

• Make agent feature edits reach CTR training; fix click-history clock by @diya-zipline in #2181
• Correct the activity clock at the source instead of forking the GroupBy by @diya-zipline in #2183
• Score the CTR model on a held-out day instead of nothing by @diya-zipline in #2182
• fix(spark): skip log rows with unknown schema_hash instead of failing the flattener by @nikhil-zlai in #2185
• [Spark 4] Ship AWS engine artifacts in the release image by @tchow-zlai in #2189

Full Changelog: v1.23.4...v1.24.0

What's Changed

Bug fixes

• Rename KV publication override flag by @tchow-zlai in #2174
• Address remaining CTR and training review findings by @diya-zipline in #2179

Full Changelog: v1.23.3...v1.23.4

What's Changed

Minor features

• Add MegaTile streaming pipeline: 2 point gets instead of prefix scan by @nikhil-zlai in #1627
• [Redis writes 1/4] Define batch modes and publication contract by @BenMagyar in #2135
• Expose staging-query force overwrite in the CLI by @tchow-zlai in #2154
• Publish Bigtable KV uploads with version timestamps by @tchow-zlai in #2155
• GigaTile: push-based serving — design doc, core processor, Flink wiring by @nikhil-zlai in #1645

Bug fixes

• Right-size GCP Dataproc and AWS EMR canary clusters by @tchow-zlai in #2153
• Align EMR Serverless initial executors with canary cap by @tchow-zlai in #2156
• Fix Iceberg partition column case resolution by @diya-zipline in #2079
• Add opt-in DELTA temporality knob for OTLP metrics by @bobbetter in #2157
• [Spark 4] Run Crucible compute on Spark 4 by @tchow-zlai in #2152
• Add aws_databricks derivations join + ctr_labels SQ; publish Spark 4 cloud_aws jar per dev VERSION by @piyush-zlai in #2163
• Add sub-daily dayoffset canary confs by @kumar-zlai in #2167
• [Spark 4] Align Flink native dependencies by @tchow-zlai in #2168
• Add AWS TRAIN_MODEL hello-world canary and Hub integration test by @diya-zipline in #2158
• Pass submission context to direct model training by @diya-zipline in #2176
• [Spark 4] Select Spark 4 Flink dependencies for Crucible by @tchow-zlai in #2170
• Add AWS CTR training through Hub and Ray by @diya-zipline in #2161
• improv(stats): Compute nulls on complex types by @cristian-zlai in #2178
• Grid-floor non-string ds in scanDistinctPartitions fallback by @piyush-zlai in #2177

Full Changelog: v1.23.2...v1.23.3

What's Changed

Bug fixes

• [Spark 4] Add source compatibility CI by @tchow-zlai in #2142
• Add DirectTrainingSubmitter next to TrainingRequest by @diya-zipline in #2141
• [Spark 4] Apply Delta partial-stats data skipping by @tchow-zlai in #2144
• Revert Flink TaskManager memory back to 4G by @piyush-zlai in #2149
• [Spark 4] Use official Delta runtime for Crucible AWS by @tchow-zlai in #2145
• [Spark 4] Use official UC and Iceberg runtimes for Crucible AWS by @tchow-zlai in #2146
• [Spark 4] Publish a consistent Crucible engine set by @tchow-zlai in #2147
• Resolve GroupBy status via serving info instead of CHRONON_METADATA conf by @piyush-zlai in #2150
• [Spark 4] Build Crucible artifacts in the canonical build job by @tchow-zlai in #2151

Other changes

• Add Kafka tasks parallelism override by @JoeTime17 in #2148

New Contributors

• @diya-zipline made their first contribution in #2141

Full Changelog: v1.23.1...v1.23.2

What's Changed

Bug fixes

• Yank sensitive config values from Vertx /config endpoint response by @piyush-zlai in #2134
• [Spark 4] Deprecate configurable write formats by @tchow-zlai in #1661
• [Spark 4] Preserve query semantics across Spark 3 and Spark 4 by @tchow-zlai in #2140
• improv(flink): memory settings for dataproc submitter by @cristian-zlai in #2133

Full Changelog: v1.23.0...v1.23.1

What's Changed

Minor features

• Add opt-in BatchGetItem path for DynamoDBKVStoreImpl.doGetLookups by @piyush-zlai in #2114
• KVStore.cleanupTables + DynamoDB batch-table sweep by @piyush-zlai in #2128

Bug fixes

• Pipeline Redis batch and streaming reads by @BenMagyar in #2095
• fix(flink): prevent empty Flink manifest on restart by @cceasy in #2116
• Wait for DynamoDB replicas ACTIVE in bulkPut by @piyush-zlai in #2125
• Isolate TTL cache refresh execution by @BenMagyar in #2092
• Strip executionInfo from Join metadata on KV upload by @piyush-zlai in #2126
• Use 4G TaskManagers for canary configs by @tchow-zlai in #2131
• Scheduled refresher for DynamoDB batch table registry pointer by @piyush-zlai in #2129
• Make UNIQUE_TOP_K duplicate resolution deterministic by @tchow-zlai in #2130

Other changes

• feat(cloud_gcp): forward spark.bigtable.* confs to the BigTable bulk-load writer by @cceasy in #2127

Full Changelog: v1.22.0...v1.23.0

What's Changed

Bug fixes

• last values with sub-daily offset should write with batch end by @BenMagyar in #2109
• Use latest key filter snapshot for uploads by @BenMagyar in #2087
• Patch Iceberg HttpComponents security vulnerabilities by @nikhil-zlai in #2112
• Exclude metaData recursively from semantic hashes by @kumar-zlai in #2111
• Increase GCP integration executor memory by @tchow-zlai in #2113
• Align chained join schemas with response precedence by @BenMagyar in #2118
• Tweak IonWriter to add ttl field by @piyush-zlai in #2119
• Fix nightly Grype failures: bump netty to 4.1.137.Final by @piyush-zlai in #2121
• Add env var toggle to disable fetcher batch IR cache by @piyush-zlai in #2120
• Add pre-traffic warmup hook to FetcherVerticle by @piyush-zlai in #2104
• Bump test_run_adhoc timeout to 1800s by @david-zlai in #2122

Full Changelog: v1.21.1...v1.22.0

What's Changed

Bug fixes

• Add Flink latency stage metrics by @german-krikorian-ck in #2085
• Register Iceberg table broadcast payloads with JavaSerializer in ChrononKryoRegistrator by @nikhil-zlai in #2090
• Cast request keys before online derivations by @BenMagyar in #2089
• Normalize numeric Catalyst inputs to declared types by @BenMagyar in #2088
• Reuse Avro codecs in fetch path by @BenMagyar in #2094
• Reuse JavaFetcher metric contexts by @BenMagyar in #2098
• Cache localized Catalyst UDF jars by @BenMagyar in #2093
• Configure Redis pools without borrow-time pings by @BenMagyar in #2097
• Map Dataproc SETUP_DONE to JobStatusType.PENDING by @david-zlai in #2103
• Fix online-suite session-contamination flake: advertise loopback in CatalystUtil by @nikhil-zlai in #2106

Full Changelog: v1.20.1...v1.21.2

What's Changed

Bug fixes

• Add Flink latency stage metrics by @german-krikorian-ck in #2085
• Register Iceberg table broadcast payloads with JavaSerializer in ChrononKryoRegistrator by @nikhil-zlai in #2090
• Cast request keys before online derivations by @BenMagyar in #2089
• Normalize numeric Catalyst inputs to declared types by @BenMagyar in #2088
• Reuse Avro codecs in fetch path by @BenMagyar in #2094
• Reuse JavaFetcher metric contexts by @BenMagyar in #2098
• Cache localized Catalyst UDF jars by @BenMagyar in #2093
• Configure Redis pools without borrow-time pings by @BenMagyar in #2097
• Map Dataproc SETUP_DONE to JobStatusType.PENDING by @david-zlai in #2103
• Fix online-suite session-contamination flake: advertise loopback in CatalystUtil by @nikhil-zlai in #2106

Full Changelog: v1.20.1...v1.21.1

What's Changed

Bug fixes

• Add Flink latency stage metrics by @german-krikorian-ck in #2085
• Register Iceberg table broadcast payloads with JavaSerializer in ChrononKryoRegistrator by @nikhil-zlai in #2090
• Cast request keys before online derivations by @BenMagyar in #2089
• Normalize numeric Catalyst inputs to declared types by @BenMagyar in #2088
• Reuse Avro codecs in fetch path by @BenMagyar in #2094
• Reuse JavaFetcher metric contexts by @BenMagyar in #2098
• Cache localized Catalyst UDF jars by @BenMagyar in #2093
• Configure Redis pools without borrow-time pings by @BenMagyar in #2097
• Map Dataproc SETUP_DONE to JobStatusType.PENDING by @david-zlai in #2103
• Fix online-suite session-contamination flake: advertise loopback in CatalystUtil by @nikhil-zlai in #2106

Full Changelog: v1.20.1...v.1.21.0

What's Changed

Minor features

• feat: Per-aggregation filter conditions in GroupBy by @cceasy in #1977

Bug fixes

• CTRL-281: Add hostname to OTLP attributes for deduping on multipod by @cristian-zlai in #2082
• Surface async Dataproc cluster creation failures by @david-zlai in #2080
• CLI: Fix auth behind JWT-only API gateways by @sean-zlai in #1985

Full Changelog: v1.20.0...v1.20.1

What's Changed

Minor features

• feat(aws): add Redis/ElastiCache KV store support by @piyush-zlai in #2074
• Re-enable Hive UDF support in fetcher + flink by @piyush-zlai in #2031

Bug fixes

• Bypass FutureConverters on hot fetch path to avoid ForkJoinPool hop by @piyush-zlai in #2069
• handle v2 delta stats for readiness checks by @BenMagyar in #2046
• setting up benchmark path for redis kv impl by @BenMagyar in #2071
• Validate join request key schemas before Avro encoding by @tchow-zlai in #2056
• Fix nightly Grype failure: force lz4-java 1.11.1 in kinesis connector by @piyush-zlai in #2078
• Cache catalyst planning per query instead of per CatalystUtil instance by @nikhil-zlai in #2066

Full Changelog: v1.19.1...v1.20.0

What's Changed

Bug fixes

• Log output schema instead of df.show() after Join backfill by @david-zlai in #2062
• Surface branch and effective engine version as k8s labels on Flink jobs + wait for job manager deployment status error transitions by @piyush-zlai in #2057
• Fix nightly Grype failures: bump jackson-databind and jetty by @piyush-zlai in #2063
• Add zipline admin doctor infra-health grouped deployment checks by @kumar-zlai in #2061
• Deduplicate contextual GroupBy requests in bulk fetchJoin by @piyush-zlai in #2064
• Add DynamoDB SDK metrics bridge into Chronon OTel pipeline by @piyush-zlai in #2065
• fix(online): keep serving existing metadata when a TTLCache refresh fails by @cceasy in #2058

Full Changelog: v1.19.0...v1.19.1

What's Changed

Bug fixes

• Skip System.exit in cluster deploy mode to fix YARN status reporting by @cceasy in #1844
• Throttle warn/error logging on the fetcher request hotpath by @piyush-zlai in #2051
• Backfill day-for-day dep for ModelTransform EVENTS sources by @david-zlai in #2059

Full Changelog: v1.18.1...v1.19.0