Changelog

What's Changed

Minor features

• Add SCD2 GroupBy helper CTRL-206 by @nikhil-zlai in #1869

Bug fixes

• [codex] Add GroupBy upload status endpoint by @nikhil-zlai in #1860
• refactor(cli): centralize shared root options by @tchow-zlai in #1875

Full Changelog: v1.12.2...v1.13.0

Full Changelog: v1.12.2...v1.13.0

What's Changed

Bug fixes

• Docs cleanup by @sean-zlai in #1845
• Per-environment compile via teams..py by @david-zlai in #1858
• Wire Crucible Spark and Flink UI links by @tchow-zlai in #1859
• Fix nested JoinSource mutation dependencies by @nikhil-zlai in #1867
• fix(spark): stabilize GroupByTest "group by from chaining gb" by @bagipriyank in #1863
• Bump commons-configuration2 to 2.15.0 to clear grype GHSA-337m-mw94-2v6g by @kumar-zlai in #1868
• cloud_aws: bundle patched delta-spark + delta-storage into the assembly by @kumar-zlai in #1865
• docs: Add integration testing section to devnotes by @tchow-zlai in #1836

Full Changelog: v1.12.1...v1.12.2

What's Changed

Minor features

• VSCode: Add environment state / selection by @sean-zlai in #1045

Bug fixes

• fix(orchestration): prevent overlapping workflow execution cycles by @tchow-zlai in #1051
• UI: Update deps by @sean-zlai in #1048
• UI: Multiline entity labels (folder/namespace as secondary line) by @sean-zlai in #1047
• fix: extend streaming uniqueness constraint + add retry tracing CTRL-197 by @kumar-zlai in #1053
• UI: Strip SvelteKit Link headers (fix nginx 502 error) by @sean-zlai in #1057
• Resolve conf placeholders from the Hub process env, not just team env by @kumar-zlai in #1056
• Bump commons-configuration2 to 2.15.0 to clear grype GHSA-337m-mw94-2v6g by @kumar-zlai in #1058
• UI: Support master as default branch on features view by @sean-zlai in #1059
• Revert migration 0015: drop streaming uniqueness over retry states by @kumar-zlai in #1061

Full Changelog: v1.12.1...v1.12.2

What's Changed

Bug fixes

• Make schedule-all schedule all confs and not just the diffs by @david-zlai in #1831
• [crucible] Render Spark UI URL via gateway /spark/{ns}/{id} rewrite by @tchow-zlai in #1852
• fix(flink-eks): pin Flink and aws-cli images by digest/version by @kumar-zlai in #1856
• Enable JoinSource chaining across joins by @varant-zlai in #1853

Full Changelog: v1.11.0...v1.12.1

What's Changed

Bug fixes

• UI: Fix workflow logs when ran more than 24 hours ago (for log providers requiring start/end such as CloudWatch) by @sean-zlai in #1040
• Site: New blog posts and site improvements by @sean-zlai in #1029
• fix(eval): preserve backtick info end-to-end in table rewrite by @tchow-zlai in #988
• UI: Move schedule columns to be more visible by @sean-zlai in #1044

Full Changelog: v1.12.0...v1.12.1

What's Changed

Bug fixes

• Make schedule-all schedule all confs and not just the diffs by @david-zlai in #1831
• [crucible] Render Spark UI URL via gateway /spark/{ns}/{id} rewrite by @tchow-zlai in #1852
• fix(flink-eks): pin Flink and aws-cli images by digest/version by @kumar-zlai in #1856
• Enable JoinSource chaining across joins by @varant-zlai in #1853

Full Changelog: v1.11.0...v1.12.0

What's Changed

Bug fixes

• Fix workflow cancellation skipping Submitted steps by @kumar-zlai in #1035
• fix(eval): raise on data types on join keys by @cristian-zlai in #1037
• feat(eval): config helpers to align generated test data by @cristian-zlai in #1033
• Audit Logging in UI by @chewy-zlai in #1034

Full Changelog: v1.11.0...v1.12.0

What's Changed

Minor features

• Add groupBy schema fetcher API by @nikhil-zlai in #1835
• [crucible][azure] Enable Chronon runs on Azure Crucible by @nikhil-zlai in #1810
• feat(stats): drift fetching by @cristian-zlai in #1843
• Add generic compare-tables CLI subcommand by @nikhil-zlai in #1538
• feat(cli): zipline compare-tables for adhoc table comparison via EMR Serverless by @kumar-zlai in #1846

Bug fixes

• feat(flink): Support HTTPS + HTTP Basic Auth for Schema Registry in SchemaRegistrySerDe by @DucNgn in #1829
• Add Hub backfill concurrency option by @nikhil-zlai in #1839
• Use delta log/snapshot explicitly to pull partition off stats if available by @BenMagyar in #1830
• Merge into fails on on delta tables with deletion vectors by @BenMagyar in #1832
• patch(emr-serverless): add custom emr image with swappable delta-spark jar by @kumar-zlai in #1838
• patch(emr-serverless): bundle patched delta-spark jar + bootstrap script by @kumar-zlai in #1847
• patch(cloud_aws): refresh UC jars off upstream + add client/hadoop bundles by @kumar-zlai in #1848
• Use select in fetcher path to support transformed keys by @BenMagyar in #1809

New Contributors

• @DucNgn made their first contribution in #1829

Full Changelog: v1.10.7...v1.11.0

What's Changed

Minor features

• [crucible][azure] Enable isolated Hub for Azure Crucible by @nikhil-zlai in #1012
• Add audit logging by @chewy-zlai in #939

Bug fixes

• Expose Hub workflow concurrency override by @nikhil-zlai in #1026
• UI: Update deps (fix fast-xml-builder / fast-uri vulns) by @sean-zlai in #1027
• Address workflow concurrency review feedback by @nikhil-zlai in #1028

Full Changelog: v1.10.7...v1.11.0

What's Changed

Bug fixes

• Show batch/online mode labels in clear-downstream CLI by @kumar-zlai in #1816
• Remove KvPartitions and KvPartitionsStore CTRL-165 by @kumar-zlai in #1799
• Improve CI caching for Mill builds by @nikhil-zlai in #1840
• Only compute null rate for ts/ds cols by @BenMagyar in #1841
• Alt: dedup imported configs by id() instead of AST by @tchow-zlai in #1821

Full Changelog: v1.10.6...v1.10.7

What's Changed

Bug fixes

• [FIX] Upgrade Quickstart Workflow Runner by @chewy-zlai in #1025
• Sidestep netty-epoll grype check by @piyush-zlai in #1021
• Walk through continuous nodes in clear-downstream by @kumar-zlai in #1013

Full Changelog: v1.10.6...v1.10.7

What's Changed

Bug fixes

• Make row_ids optional by @david-zlai in #1820
• docs: Document EVAL_URL teams.py prerequisite for zipline hub eval by @tchow-zlai in #1798
• Add msk iam jar + streaming GB example by @piyush-zlai in #1813
• fix(metrics): flush OTel exporter after GroupByUpload null-count gauges by @tchow-zlai in #1774

Full Changelog: v1.10.5...v1.10.6

What's Changed

Bug fixes

• test(eval): full compute test by @cristian-zlai in #1019
• improv(eval): Check mutationTs exists + MutationTable as dependency by @cristian-zlai in #1018
• improv(eval): use queryops to get all mutation columns at once by @cristian-zlai in #1020
• Fix bug in empty kvReplicaRegions env var by @piyush-zlai in #1022
• UI: Grafana/Loki env vars by @sean-zlai in #1023
• CI: Improve slack notifications by @sean-zlai in #1024

Full Changelog: v1.10.5...v1.10.6

What's Changed

Minor features

• Add support for Dynamo globally replicated tables by @piyush-zlai in #1807

Bug fixes

• feat: specify nodeSelector when triggering Flink apps on EKS by @felipesai in #1790
• fix: vertx metrics to otlp and resource attributes by @sai-bojja-ck in #1815
• Make azure auth token acquire a warning message if unable by @david-zlai in #1822
• Register Spark StorageStatus with Kryo by @nikhil-zlai in #1825
• feat: Add Flink Application Mode deployment for Dataproc by @JoeTime17 in #1782

Other changes

• Revert "chore(spark): silence Inbox NotSerializableException noise from StorageStatus" by @nikhil-zlai in #1828

New Contributors

• @sai-bojja-ck made their first contribution in #1815

Full Changelog: v1.10.4...v1.10.5

What's Changed

Minor features

• Add support to pass through DynDB global region configs by @piyush-zlai in #1009
• UI: Log view improvements by @sean-zlai in #1007

Bug fixes

• UI: Defend against race condition with null nodeStepRun accessing runId by @sean-zlai in #1016
• improv(eval): fetch schemas for iceberg on glue if available by @cristian-zlai in #1017

Full Changelog: v1.10.4...v1.10.5

What's Changed

Minor features

• CLI: Add zipline config by @sean-zlai in #1814

Bug fixes

• feat: allow jvm to persist after Driver finishes by @cristian-zlai in #1803
• chore(spark): silence Inbox NotSerializableException noise from StorageStatus by @tchow-zlai in #1811
• test: integration tests for eval by @cristian-zlai in #1789
• refactor: support python 3.10 by @cristian-zlai in #1812
• Revert "Allow mixed types to be compiled in different folder types" by @david-zlai in #1818

Full Changelog: v1.10.3...v1.10.4

What's Changed

Minor features

• Site: Mermaid diagram support by @sean-zlai in #1010
• Site: Server-side render docs and add llms.txt support (with "Copy Page" / "Open in {AI} button) by @sean-zlai in #1011

Bug fixes

• fix(eval): set a specific tableUtils for dealing with local computation of joins/ group bys/ staging queries by @cristian-zlai in #996
• [FIX] Add Eval Url to Integration Tests by @chewy-zlai in #1015
• vscode: Improve config resolution, eval as terminal comm (zipline config, prompts w/ persistence). Run eval as termimal command by @sean-zlai in #1014

Full Changelog: v1.10.2...v1.10.4

What's Changed

Bug fixes

• Allow mixed types to be compiled in different folder types by @david-zlai in #1801

Full Changelog: v1.10.2...v1.10.3

Full Changelog: v1.10.1...v1.10.3

Full Changelog: v1.10.1...v1.10.2

What's Changed

Minor features

• feat: Add clear-downstream API with offset-aware range expansion CTRL-163 by @kumar-zlai in #960
• Extend plat to plumb through KV enable ttl flag by @piyush-zlai in #999

Bug fixes

• UI: Harden lineage by @sean-zlai in #1002
• Fix Node 20 deprecation warning by @sean-zlai in #1003
• Improve Slack notifications by @sean-zlai in #1004
• UI: Retain selected date range when switching conf versions by @sean-zlai in #1005
• fix: Show affected confs in clear-downstream preview CTRL-163 by @kumar-zlai in #1000
• fix: Prevent stack overflow in StepDao by using inSet + post-filtering by @david-zlai in #1001
• Remove tablePartitionsDataset references CTRL-165 by @kumar-zlai in #1006

Full Changelog: v1.10.0...v1.10.1

What's Changed

Minor features

• Remove kvPartitions from orchestration by @nikhil-zlai in #989
• UI: Log service profiles and filters by @sean-zlai in #993

Bug fixes

• UI: Update deps (fix postcss grype vuln, etc) by @sean-zlai in #990
• ci(canary): bump AWS UI rollout timeout to 600s by @tchow-zlai in #994
• UI: Consolidate using container name for service log querying and use zipline-system namespace for local docker / pocket platform by @sean-zlai in #997
• UI: Align kyuubi/spark log filters with Titan storage (labels / structured metadata) by @sean-zlai in #998

Full Changelog: v1.9.1...v1.10.0