Multi-Environment Compile & Deploy

Chronon supports compiling and deploying the same set of configs against more than one deployment environment from a single repo. Today this is limited to two envs: a default prod env and an optional canary env. The canary env is typically a parallel deployment with smaller cluster sizes, a non-production GCP project / S3 bucket, or different upstream data sources — useful for shadow-running a config before promoting it to prod.

This page is the single source of truth for the multi-env feature. The Project Setup and Deploying Chronon Entities pages reference back here for details.

Only canary is supported as a non-prod env today. The hub's wire contract — the Thrift Environment enum on metaData.environments — currently models PROD and CANARY only. Dropping in e.g. teams.staging.py will fail discovery with a clear error. Supporting a new env end-to-end requires extending thrift/api.thrift and the hub server in lockstep.

How it works

zipline compile discovers env-specific teams files in the repo root:

<repo-root>/
├── teams.py           # → compiles to compiled/         (prod)
├── teams.canary.py    # → compiles to compiled_canary/  (canary, optional)
├── group_bys/
├── joins/
└── staging_queries/

Each pass writes to its own output folder and uses only its own teams file — there is no implicit fallback from the canary pass to teams.py. Every team referenced by a config in the canary pass must be declared (or imported, see below) in teams.canary.py, or compile fails loudly.

After every run, zipline compile prints a per-env summary so you can see what landed on disk:

─────────── 📋 COMPILATION SUMMARY ───────────
  canary  ✓ OK      → compiled_canary  (165 written)
  prod    ✓ OK      → compiled         (212 written)

If a pass fails (e.g. a Python syntax error in a config), its staging dir is discarded and compiled_<env>/ is not updated — the summary will report (N error(s); output not written (M parsed and discarded)). Any env's failure causes zipline compile to exit with code 1, so a broken canary pass surfaces in CI even when the prod pass succeeded.

Authoring teams.canary.py

teams.canary.py is a regular Python module. The recommended pattern is to import teams from teams.py for the ones that are identical to prod, layer canary-only mutations on top, and only redeclare teams that diverge structurally:

# teams.canary.py
from ai.chronon.repo.spark_catalog_confs import *
from ai.chronon.types import ConfigProperties, EnvironmentVariables, Team
 
# Reuse identical-to-prod teams via plain Python import.
from teams import aws_databricks, azure, quickstart
 
# Layer a canary-only override on top of an imported team. This mutation only
# affects the canary compile pass — the prod pass re-loads teams.py from
# scratch and never sees this change.
aws_databricks.env.common['ARTIFACT_PREFIX'] = "s3://my-org-artifacts-canary"
 
# Teams that differ structurally from prod — redeclare in full.
default = Team(
    outputNamespace="default",
    env=EnvironmentVariables(common={"VERSION": "latest", "CUSTOMER_ID": "canary"}),
)
 
gcp = Team(
    outputNamespace="data",
    env=EnvironmentVariables(
        common={
            "CLOUD_PROVIDER": "gcp",
            "CUSTOMER_ID": "canary",
            "GCP_PROJECT_ID": "my-org-canary",
            "ARTIFACT_PREFIX": "gs://my-org-artifacts-canary",
            "WAREHOUSE_PREFIX": "gs://my-org-warehouse-canary",
        },
    ),
    conf=ConfigProperties(
        common={
            **BigQueryConfiguration({
                "spark.sql.catalog.spark_catalog.warehouse": "gs://my-org-warehouse-canary/data/tables/",
                "spark.sql.catalog.spark_catalog.gcp.bigquery.location": "us-central1",
                "spark.sql.catalog.spark_catalog.gcp.bigquery.project-id": "my-org-canary",
            }),
            "spark.chronon.partition.format": "yyyy-MM-dd",
            "spark.chronon.partition.column": "ds",
        },
    ),
)

The import-and-mutate pattern works because the prod pass re-loads teams.py from a fresh Python module each compile invocation — mutations applied during the canary pass don't survive into the prod pass. Same field structure as the prod-side Team API; see Project Setup for the full field reference.

Per-entity opt-in via environments=[...]

Authoring a teams.canary.py makes the canary compile possible, but each GroupBy/Join/StagingQuery still has to opt in to canary deploys explicitly:

GroupBy(
    ...
    environments=['prod', 'canary'],  # scheduled by both prod and canary deploys
)

Entities without an explicit environments= default to ['prod'] only — so a config running on production data never accidentally rolls out to canary until you opt in.

Deploying with --env

The hub schedule and deploy commands take an --env flag that selects which compile output to read and which configs to deploy:

zipline hub schedule-all --cloud aws --env prod     # default — reads compiled/
zipline hub schedule-all --cloud aws --env canary   # reads compiled_canary/

schedule-all reads each conf's metaData.environments field and skips entities whose list doesn't contain the target env. Behavior matrix:

The default (prod-only) is intentional: a config running on production data never rolls out to canary clusters until you explicitly add 'canary' to its environments list.

CI workflow

A typical CI pipeline runs both compile passes and both deploys:

# In CI, on merge to main:
zipline compile                                     # produces compiled/ + compiled_canary/
zipline hub schedule-all --cloud aws --env prod     # deploys prod-tagged confs
zipline hub schedule-all --cloud aws --env canary   # deploys canary-tagged confs

zipline compile exits non-zero if either env's pass had errors, so a broken canary won't slip through CI even when the prod pass is clean.