Streaming Sources

A temporal online GroupBy is fed by a stream: set topic on the source alongside the offline table. Spark backfills from the table, the Flink engine reads the topic for real-time features, and both are expected to describe the same events. See Sources for the offline side and EntitySource for snapshot plus mutation (CDC) sources.

A topic URI is <transport>://<topic-name>/<key>=<value>/... and carries three independent things: the transport, the serde (serde=), and whatever that serde needs in order to find a schema. Params are a flat map, so registry and transport options sit side by side.

Transports

Transport Example Options
Kafka kafka://user-activities/serde=schema_registry/registry_host=... bootstrap (falls back to the online properties), start_offset (epoch millis; default is committed offsets, resetting to latest), tasks, plus any Kafka client property as a URI param: security.protocol, sasl.mechanism, and so on. sasl.jaas.config is picked up from the SASL_JAAS_CONFIG env var (or fetched via SASL_JAAS_CONFIG_VAULT_URI) when it is not in the URI.
Pub/Sub pubsub://user-activities-v2/project=my-project/subscription=my-sub/serde=pubsub_schema/schemaId=user-activities Project comes from GCP_PROJECT_ID; project= scopes the schema and the derived subscription. Without subscription=, one subscription per GroupBy is created from the topic. Also: tasks.
Kinesis kinesis://user-activities/serde=glue_registry/registry_name=.../schema_name=... Region and credentials from the environment (AWS_REGION, AWS_ACCESS_KEY_ID, ...), plus initial_position (default latest), tasks, enable_efo / efo_consumer_name.

tasks sets source parallelism on every transport.

Passing transport options through CHRONON_ONLINE_ARGS

Transport options can also be passed to the streaming job as -Z<key>=<value> flags in the CHRONON_ONLINE_ARGS environment variable. Set it on the GroupBy, or in teams.py to cover a whole team. This keeps deployment-specific values such as parallelism or broker addresses out of the topic URI (python/test/canary/group_bys/aws/user_activities.py):

v1 = GroupBy(
    ...,
    env_vars=EnvironmentVariables(
        common={
            "CHRONON_ONLINE_ARGS": "-Ztasks=1 -Zbootstrap=b-1.my-cluster.kafka.us-west-2.amazonaws.com:9092",
        }
    ),
)

When a key is set both ways, the -Z value wins over the URI param. This applies to the transport options above: tasks, bootstrap and start_offset (Kafka), subscription (Pub/Sub), and initial_position, enable_efo and efo_consumer_name (Kinesis). Serde and registry options (everything in the next section) and Kafka client properties such as security.protocol are read from the topic URI only.

Serdes and schema registries

serde= selects the decoder. A topic that sets registry_host= but no serde= is treated as schema_registry.

serde Registry Schema formats
schema_registry Confluent (self-hosted or managed) Avro, Protobuf
apicurio_registry Apicurio Avro, Protobuf, JSON
glue_registry AWS Glue Avro, Protobuf, JSON
pubsub_schema Pub/Sub schema Avro, Protobuf
custom none - provider_class= names a SerDe implementation up to the provider

Registry options, all optional unless noted:

Registry Keys
Confluent registry_host (required), registry_port, registry_scheme (default http), subject (default <topic>-value), reader_schema_version, schema_registry_wire_format (default true), basic.auth.credentials.source + basic.auth.user.info (or the SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO env var), proto3_default_as_null
Apicurio registry_host (required), registry_port, registry_scheme (default http), group_id (default default), artifact_id (default the topic name), wire_format (apicurio, confluent or none; unset passes payloads through untouched), proto3_default_as_null
Glue registry_name (required), schema_name (required), region, version_number, wire_format, aws_access_key_id / aws_secret_access_key, proto3_default_as_null
Pub/Sub project (required), schemaId (required), schema_revision, resolve_writer_schema (default true), proto3_default_as_null

debezium=true switches the Avro and JSON decoders to the Debezium envelope shape; the Flink job injects it automatically for entity (mutation) sources, so it only needs setting by hand on an event topic that carries envelopes - see EntitySource. For local development, LOCAL_SCHEMA_DIR makes the Glue and Pub/Sub serdes read <schema_name>.avsc / .json from disk instead of the registry.

Examples

Kinesis with Glue registry (python/test/canary/group_bys/aws/user_activities.py):

topic="kinesis://user-activities/serde=glue_registry/registry_name=zipline-canary/schema_name=user-activities"

Pub/Sub with a Pub/Sub schema (python/test/canary/group_bys/gcp/user_activities.py):

topic="pubsub://user-activities-v2/project=canary-443022/subscription=user-activities-v2-sub/serde=pubsub_schema/schemaId=user-activities"

Kafka with Apicurio and SASL (python/test/canary/group_bys/azure/user_activities.py):

serde = "serde=apicurio_registry"
registry_host = "registry_host=apicurio-registry-apicurio-registry.zipline-system.svc.cluster.local"
registry_port = "registry_port=8080"
artifact_id = "artifact_id=user-activities-js"
topic = f"kafka://user-activities/{serde}/{registry_host}/{registry_port}/{artifact_id}/security.protocol=SASL_SSL/sasl.mechanism=PLAIN"

Schema evolution

For every format, the reader schema is what the registry returns when the job starts (or the pinned version, if one is set), and the tile schema is fixed from it. reader_schema_version (Confluent), version_number (Glue) and schema_revision (Pub/Sub) pin the reader instead of taking the latest at startup.

Two things to keep in mind:

  • A new field is not aggregated until the job restarts with a reader schema that has it. Pinning the reader makes that explicit; leaving it unpinned means the next restart picks up whatever the registry has then.
  • Removing or renaming a field is a schema change for the tiles too: plan a new GroupBy version and a backfill, not just a topic upgrade.

How records written with other schema versions are decoded depends on the format. Protobuf matches fields by tag number and JSON by field name, so neither needs the writer's schema. Avro records carry no field identifiers, so decoding them correctly needs the schema they were written with - see below.

Avro: writer schema resolution

Where the registry identifies the schema per record, Chronon resolves the writer schema for each Avro message and decodes through Avro's resolution logic:

Registry Writer schema per record Identified by
Confluent yes, by default schema id in the 5-byte wire header
Apicurio yes, when wire_format=apicurio or wire_format=confluent 9-byte global id, or the Confluent-compatible 5-byte header
Glue yes, with wire_format=glue 18-byte header: 0x03, compression byte (0x00 or 0x05 for zlib), 16-byte version id
Pub/Sub yes, by default, on topics with a schema attached googclient_schemarevisionid message attribute

With resolution, adding a field fills the reader's default, dropping a field just stops reading it, and inserting or renaming one resolves by name. Without it the decode is positional, so an appended field fails on old records and an inserted field silently mis-maps onto the following column - which is why Glue topics written by the AWS Glue serializer need wire_format=glue, and why Apicurio topics need a wire_format to get this at all.

Pub/Sub only stamps the revision attribute when the topic itself has a schema attached (its schema settings), which is also what makes Pub/Sub validate messages on publish. schemaId on its own only tells Chronon where to read the reader schema; on a topic without a schema attached, every record is decoded with the reader schema alone, as before.

On Glue and Pub/Sub, a writer schema that can't be used never costs records: they decode with the reader schema alone, as they did before writer resolution existed, and a throttled warning names the version. That covers versions deleted from the registry, writers the reader can't resolve (a renamed record, or a type change Avro can't promote such as long to int), and Pub/Sub messages validated against a different schema than the configured schemaId. Registry errors (outages, throttling, missing permissions) are handled the same way and the lookup is retried a minute later, rather than on every record.

Knobs:

Key Effect
schema_registry_wire_format=false (Confluent), resolve_writer_schema=false (Pub/Sub) Decode every record against the reader schema alone
wire_format unset or none (Apicurio), and the default on Glue Treat payloads as bare records; Glue logs a warning when it sees framing anyway

The wire format settings matter for Protobuf and JSON too: the framing has to be stripped before any record can be decoded, even though only Avro uses the schema id it carries.

Worked examples, each reading a topic where a v1 producer keeps writing after the schema gained a field: python/test/canary/group_bys/aws/schema_evolution.py (Kinesis, Glue, wire_format=glue) and python/test/canary/group_bys/gcp/schema_evolution.py (Pub/Sub topic with a schema attached).

Limitations

  • JSON is not supported by the Confluent and Pub/Sub serdes.
  • Protobuf payloads are self-describing, so no writer schema lookup happens for them on any registry; proto3_default_as_null controls whether proto3 defaults surface as nulls. With wire_format=glue, records must encode the reader schema's message type (the first message in the .proto); records of any other message type in the same file fail to decode.
  • Writer-schema lookups run on the task managers, so they need egress to the registry - the driver alone is not enough. For Glue that means glue:GetSchemaVersion on the execution role once wire_format=glue is set, scoped to the registry and schema ARNs (registry/<name> and schema/<name>/*) or to "Resource": "*" as AWS's own deserializer policy does. If the lookup is denied, records decode with the reader schema alone and a throttled warning is logged. For Pub/Sub it is getSchema, on by default; for Confluent and Apicurio it applies whenever the wire format is parsed.
Edit this page