Dynamic Aggregations API
Dynamic aggregations are rolling aggregations that are defined at runtime instead of in a GroupBy you commit. They are built for rule engines that need a new counter or a new window during an incident: an analyst adds an aggregation through the Hub, and the running streaming job starts filling it without a git checkout, a recompile, or a Flink restart.
Under the hood an aggregate is an ordinary PUSH (GigaTile) GroupBy that the Hub compiles, validates, publishes and deploys for you. One aggregate exists per (team, source, key columns) triple, and aggregations are only ever appended to it.
Two services are involved:
- The Hub owns teams, sources and aggregates, and does the writing and publishing.
- The fetcher serves current values to your rule engine.
This page is the reference for both. The same calls back the Profiles area of the Hub UI (Profiles -> Sources, Aggregates, Fetch, and the API tab, which prints the matching curl for every action you take).
Wiring it up
- Register a source: a stream topic plus the table its events land in. Sources are shared by every team.
- Pick the owning team. Teams synced from
teams.pyappear in the Hub automatically; a team that is not in the repo can be created through the API. - Create an aggregate for that team, source and key column set. The Hub derives its name, compiles the conf, uploads it so eval can check it, validates it, then syncs the branch, writes the conf to the KV metadata dataset and starts a deploy. The streaming job reads the conf from KV.
- Read values from the fetcher.
- Later, append aggregations to the same aggregate. The running job picks up the new conf and starts filling the new columns; nothing restarts.
- When an aggregate is no longer needed, pause it (stop its daily deploys and streaming job until you resume it) or delete it.
Before you start
The API is registered only when the Hub has both:
EVAL_URL(eval.url), used to read a source table's schema and to validate each conf, and- an online KV store, used to publish the conf for the streaming job and the fetcher.
Without them the routes are not mounted and every Hub call below returns 404. EVAL_AUTH_HEADER (eval.authHeader) is optional and is sent as the Authorization header on Hub-to-eval calls. When it is unset and the Hub runs on GCP, the Hub uses a Google identity token from the metadata server instead.
On GCP, the streaming job's service account also needs permission to create the aggregate's _PUSH Bigtable table. The default Dataproc role does not have it, and the failure is not surfaced as an API error.
Hub GET calls need the viewer role, and POST and DELETE calls need operator (see Authentication for roles and token acquisition). The fetcher does not authenticate requests itself, so the fetcher examples send no token; add whatever your gateway requires if it sits behind one.
export HUB_URL="https://zipline.example.com"
export TOKEN="$(zipline auth get-access-token)" # or: gcloud auth print-identity-token
export FETCHER_URL="https://zipline-fetcher.example.com"Every example below assumes those variables.
Aggregates live on a dedicated Hub branch named dynamic. That branch is written only by this API: upload, sync, schedule, unschedule, workflow-start and team-sync requests that target the dynamic branch are rejected with an error. So are upload, sync, schedule, unschedule and workflow-start requests for any conf named <team>.dynamic.*.
Hub API
| Method | Path | Role | Purpose |
|---|---|---|---|
POST | /dynamic/v1/sources | operator | Register a source |
GET | /dynamic/v1/sources | viewer | List sources with their columns |
GET | /dynamic/v1/teams | viewer | List teams that can own aggregates |
POST | /dynamic/v1/teams | operator | Create a team outside the repo |
GET | /dynamic/v1/teams/{name} | viewer | Read one team, with its own and resolved settings |
PUT | /dynamic/v1/teams/{name} | operator | Update a created team |
POST | /dynamic/v1/teams/{name}/archive | operator | Archive a created team |
GET | /dynamic/v1/team-defaults | viewer | Read the team defaults |
PUT | /dynamic/v1/team-defaults | operator | Save the team defaults |
POST | /dynamic/v1/aggregates | operator | Create an aggregate |
GET | /dynamic/v1/aggregates | viewer | List aggregates |
GET | /dynamic/v1/aggregates/{name} | viewer | Read one aggregate |
POST | /dynamic/v1/aggregates/{name}/aggregations | operator | Append aggregations |
POST | /dynamic/v1/aggregates/{name}/publish | operator | Re-publish the saved conf |
POST | /dynamic/v1/aggregates/{name}/pause | operator | Stop an aggregate's daily deploys and streaming job |
POST | /dynamic/v1/aggregates/{name}/resume | operator | Restart a paused aggregate |
DELETE | /dynamic/v1/aggregates/{name} | operator | Stop and remove an aggregate |
Aggregate names are identifiers joined by dots and underscores, so they need no URL encoding.
Register a source
curl -X POST "$HUB_URL/dynamic/v1/sources"
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
-d '{
"alias": "card_txns",
"topic": "pubsub://card-transactions/serde=pubsub_schema/project=my-project/schemaId=card-txn-schema/subscription=card-txns-sub",
"table": "dynamic_demo.card_transactions",
"timeColumn": "ts"
}'Response:
{
"alias": "card_txns",
"topic": "pubsub://card-transactions/serde=pubsub_schema/project=my-project/schemaId=card-txn-schema/subscription=card-txns-sub",
"table": "dynamic_demo.card_transactions",
"timeColumn": "ts",
"rawSchema": "event_id:string,card:string,amount:long,country:string,ts:long"
}| Field | Type | Required | Description |
|---|---|---|---|
alias | string | yes | Short name, used in aggregate names. Lowercase letters, digits and underscores, starting with a letter; registered once — re-registering it returns 400. |
topic | string | yes | Stream URI, including its serde, e.g. pubsub://<topic>/serde=pubsub_schema/project=<project>/schemaId=<schema>/subscription=<subscription> or kafka://<topic>/serde=schema_registry/registry_host=<host>. |
table | string | yes | Table the events land in. Supplies the columns and the history for long windows. |
timeColumn | string | yes | Event time column, in epoch milliseconds. |
settingsTeam | string | no | Team whose catalog settings read the table's schema instead of the team defaults, for a table only that team's catalog can read. Not stored. |
The Hub reads the table's columns and types at registration, with the catalog settings of the team defaults (or of settingsTeam), and stores them as rawSchema; aggregations and key columns must be drawn from those columns. A partition column named ds is dropped, because stream events never carry it. Supported column types are long, int, double, float, string, boolean and bytes; columns of any other type are left out of rawSchema and cannot be used.
Without saved team defaults or a settingsTeam, registration is rejected with 400. Sources belong to no team: every team can build aggregates on one, and each aggregate is validated with its own team's settings when it is saved.
Check the source before you register it, because it cannot be changed afterwards:
- Every column in
rawSchemamust be present in the stream's events, or the streaming job fails to start. That includes table-only columns, and a partition column with a name other thands. - The Hub only checks that
topicandtimeColumnare non-empty. A wrong stream URI or time column surfaces later, when an aggregate is validated or its streaming job starts.
List sources
curl "$HUB_URL/dynamic/v1/sources" -H "Authorization: Bearer $TOKEN"{
"sources": [
{
"alias": "card_txns",
"topic": "pubsub://card-transactions/serde=pubsub_schema/project=my-project/schemaId=card-txn-schema/subscription=card-txns-sub",
"table": "dynamic_demo.card_transactions",
"timeColumn": "ts",
"rawSchema": "event_id:string,card:string,amount:long,country:string,ts:long"
}
]
}List teams
curl "$HUB_URL/dynamic/v1/teams" -H "Authorization: Bearer $TOKEN"{
"teams": ["fraud"],
"details": [
{
"name": "fraud",
"origin": "repo",
"outputNamespace": "dynamic_demo",
"email": "fraud-oncall@example.com"
}
]
}origin is repo for a team synced from teams.py in your Chronon repo, and profiles for a team created with the call below. teams is kept for existing callers; details carries the same teams with their metadata.
Create a team
Only needed for teams that are not in the Chronon repo. A created team's settings are layered like a team in teams.py: its own settings over the team defaults, with the team's values winning.
curl -X POST "$HUB_URL/dynamic/v1/teams"
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
-d '{
"name": "payments",
"email": "payments@example.com",
"description": "Payments risk rules",
"outputNamespace": "dynamic_demo",
"settings": {
"env": {
"common": { "CUSTOMER_ID": "payments" },
"modeEnvironments": { "deploy": { "SPARK_CLUSTER_NAME": "payments-online" } }
},
"conf": { "common": { "spark.executor.memory": "4g" } }
}
}'{
"name": "payments",
"origin": "profiles",
"outputNamespace": "dynamic_demo",
"email": "payments@example.com",
"description": "Payments risk rules"
}| Field | Type | Required | Description |
|---|---|---|---|
name | string | yes | Lowercase letters, digits and underscores. Becomes the first segment of the team's aggregate names. default is reserved for the team defaults. |
email | string | no | Owner contact. |
description | string | no | Shown in team lists. |
outputNamespace | string | yes, unless baseTeam is given | Namespace for the team's tables; letters, digits and underscores, optionally catalog.namespace. Defaults to baseTeam's when that is given. |
settings | object | no | The team's own settings: env, conf and clusterConf, each with common and per-mode maps (modeEnvironments, modeConfigs, modeClusterConfigs), as EnvironmentVariables, ConfigProperties and ClusterConfigProperties are written in teams.py. |
baseTeam | string | no | Team whose resolved settings become the new team's own, so its aggregates run where that team's do. Ignored when settings is given. |
Without settings or baseTeam, the team has no settings of its own: it runs with the team defaults, or with no settings at all until defaults are saved. Nothing is copied from another team unless you name it in baseTeam.
A name already used by a repo team or a created team is rejected with 409; default, a missing outputNamespace or an unknown baseTeam with 400. If teams.py later adds a team with the same name, the team created here takes precedence for aggregates.
Read a team
curl "$HUB_URL/dynamic/v1/teams/payments" -H "Authorization: Bearer $TOKEN"{
"name": "payments",
"origin": "profiles",
"outputNamespace": "dynamic_demo",
"email": "payments@example.com",
"settings": {
"env": { "common": { "CUSTOMER_ID": "payments" }, "modeEnvironments": { "deploy": { "SPARK_CLUSTER_NAME": "payments-online" } } }
},
"resolved": {
"env": {
"common": { "CUSTOMER_ID": "payments", "HUB_URL": "https://zipline.example.com" },
"modeEnvironments": {
"deploy": { "CUSTOMER_ID": "payments", "HUB_URL": "https://zipline.example.com", "SPARK_CLUSTER_NAME": "payments-online" }
}
}
}
}settings is what the team sets itself, and is only returned for created teams. resolved is what its aggregates run with: the team defaults merged with settings by the same rules teams.py compiles with — the team's values win, and each layer's common is folded into every mode. For a repo team, resolved is its compiled settings. An unknown team returns 404.
Update a team
curl -X PUT "$HUB_URL/dynamic/v1/teams/payments"
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
-d '{
"email": "payments@example.com",
"settings": { "env": { "common": { "CUSTOMER_ID": "payments" } } }
}'Returns the team as Read a team does.
| Field | Type | Required | Description |
|---|---|---|---|
email | string | no | Owner contact. Cleared when left out. |
description | string | no | Cleared when left out. |
outputNamespace | string | no | Kept when left out. |
settings | object | no | Replaces all of the team's own settings — anything not sent is removed. Kept as they are when left out. |
Only created teams can be updated; a repo team is rejected with 400 and is changed by editing teams.py and syncing. The team's aggregates take the new settings the next time they are saved or published.
Archive a team
curl -X POST "$HUB_URL/dynamic/v1/teams/payments/archive" -H "Authorization: Bearer $TOKEN"Archives a created team, so it is no longer listed and can no longer own aggregates. A team that still owns aggregates — paused ones and ones still being deleted included — is refused with 409, naming them; delete them first. A repo team is rejected with 400; remove it from teams.py instead.
Team defaults
The team defaults are the settings every created team inherits, like the default team in teams.py. They hold the shared environment, engine conf and cluster settings (catalogs, project, cluster), so a created team only needs to set what differs. Registering a source also reads its table with them.
curl "$HUB_URL/dynamic/v1/team-defaults" -H "Authorization: Bearer $TOKEN"{
"settings": {
"env": { "common": { "HUB_URL": "https://zipline.example.com" } },
"conf": { "common": { "spark.sql.catalog.spark_catalog.warehouse": "gs://my-warehouse/data/tables/" } }
}
}settings is missing until the defaults are first saved. To save them, send the whole set — it replaces the previous defaults:
curl -X PUT "$HUB_URL/dynamic/v1/team-defaults"
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
-d '{ "settings": { "env": { "common": { "HUB_URL": "https://zipline.example.com" } } } }'{
"settings": { "env": { "common": { "HUB_URL": "https://zipline.example.com" } } },
"updatedTeams": 3,
"affectedAggregates": 5
}Saving recomputes the resolved settings of every created team in the same transaction. updatedTeams counts them; their affectedAggregates keep running with the settings they were built with until they are next saved or published. Repo teams are unaffected — their defaults come from teams.py.
Where a team lives
Both kinds of team are rows in the same zipline.teams table, keyed by (branch, name). A team synced from teams.py lives on the main branch; a team created here lives on the dynamic branch, as do the team defaults, in a row named default. That is why syncing the repo never removes a created team — a sync only reconciles its own branch — and why a dynamic row shadows a main row of the same name wherever the two could disagree.
There is one dynamic branch, and it is the only branch this API writes. You cannot branch from it, and the CLI is refused write access to it. Correspondingly, the UI can edit a team on the dynamic branch and shows a team on any other branch read-only, which is also how it treats confs: a teams.py team is changed by editing teams.py and syncing.
Compute namespaces
Creating a team also provisions the Kubernetes compute namespace its jobs run in, on deployments where the Hub is configured to do so (CRUCIBLE_TEAM_NAMESPACE_RECONCILE_ENABLED). The namespace is named from the team, labelled with it so cloud spend can be split per team, and given the engine service accounts. The call returns once that work is done, so a team you create is ready to run in.
Namespaces are only ever created, never deleted, so archiving a team leaves its namespace and anything still running in it alone.
Create an aggregate
curl -X POST "$HUB_URL/dynamic/v1/aggregates"
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
-d '{
"team": "fraud",
"sourceAlias": "card_txns",
"keyColumns": ["card"],
"aggregations": [
{
"name": "txn",
"operation": "count",
"expression": "amount",
"windows": ["1m", "10m", "1h", "30d"]
},
{
"name": "foreign",
"operation": "count",
"expression": "amount",
"filter": "country <> '"'"'GB'"'"'",
"windows": ["10m", "1h"]
}
]
}'Response:
{
"name": "fraud.dynamic.card_txns__card",
"team": "fraud",
"sourceAlias": "card_txns",
"keyColumns": ["card"],
"confHash": "3f9a1c",
"aggregations": [
{
"name": "txn",
"operation": "count",
"expression": "amount",
"windows": ["1m", "10m", "1h", "30d"],
"position": 0,
"status": "active",
"outputColumns": ["txn_count_1m", "txn_count_10m", "txn_count_1h", "txn_count_30d"]
},
{
"name": "foreign",
"operation": "count",
"expression": "amount",
"filter": "country <> 'GB'",
"windows": ["10m", "1h"],
"position": 1,
"status": "active",
"outputColumns": ["foreign_count_10m", "foreign_count_1h"]
}
]
}The '"'"' sequence closes and reopens the shell's single-quoted string, so a SQL string literal such as 'GB' can appear inside the JSON body. The Hub UI prints bodies this way already, so a curl copied from the API tab can be pasted as is.
| Field | Type | Required | Description |
|---|---|---|---|
team | string | yes | Owning team, from /dynamic/v1/teams. |
sourceAlias | string | yes | A registered source alias. |
keyColumns | string[] | yes | Source columns to aggregate by, e.g. ["card"]. Each must exist in the source's rawSchema and be a lowercase identifier (letters, digits and underscores), so a column such as cardId cannot be a key. |
aggregations | Aggregation[] | yes | At least one; see Aggregation fields. |
The name is derived, not sent: <team>.dynamic.<sourceAlias>__<keyColumns joined by _>, so (fraud, card_txns, [card]) becomes fraud.dynamic.card_txns__card and (fraud, card_txns, [card, merchant]) becomes fraud.dynamic.card_txns__card_merchant. Key order is part of the name, so [merchant, card] is a separate aggregate. Creating an aggregate that already exists returns 409; append to it instead.
The Hub works in this order:
- Validates the aggregation specs and compiles the
GroupBy. - Stores the conf in the Hub so eval can read it, and validates it with eval. A conf that eval rejects fails the whole call with
400and nothing is saved. - Saves the aggregate.
- Publishes it: syncs the
dynamicbranch, writes the conf to the KV metadata dataset, and starts a deploy. The deploy uploads history from the source table, starts the streaming job and keeps the upload scheduled daily.
The call returns once the deploy has been started, not when it finishes. See Behaviour and timing.
| Response field | Description |
|---|---|
name | The derived aggregate name. |
confHash | Hash of the current conf. Changes on every append. |
status | The aggregate's lifecycle status, as a number: 0 active, 1 pausing, 2 paused, 3 deleting. A new aggregate is 0. |
aggregations[].position | Order in the conf, starting at 0. Appended aggregations take the next positions. |
aggregations[].status | Always active today. |
aggregations[].outputColumns | The columns to read from the fetcher. |
Append aggregations
curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/aggregations"
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
-d '{
"aggregations": [
{
"name": "big_charges",
"operation": "count",
"expression": "amount",
"filter": "amount > 5000",
"windows": ["5m", "1h"]
}
]
}'The response is the updated aggregate, in the same shape as Create an aggregate.
Appending is the only way to change an aggregate's aggregations. The aggregate must be active: appending to one that is pausing, paused or being deleted returns 409.
- Existing aggregations never change, and an aggregation name cannot be reused within an aggregate. Reusing one returns
400. - New output columns are readable within about a minute; see Behaviour and timing for what they contain, because they start out incomplete.
- The streaming job applies them in place. The deploy the append starts runs the upload and leaves the running job alone.
Read an aggregate
curl "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card" -H "Authorization: Bearer $TOKEN"Returns the same shape as create, plus the daily deploy schedule:
| Response field | Description |
|---|---|
scheduleState | 1 active or 2 paused. Absent once the schedule is removed, on delete. |
nextScheduleTime | When the next daily deploy runs, while the schedule is active. |
GET /dynamic/v1/aggregates lists every aggregate, with its status but without the schedule fields:
{ "groups": [ { "name": "fraud.dynamic.card_txns__card", "team": "fraud", "...": "..." } ] }Re-publish an aggregate
curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/publish"
-H "Authorization: Bearer $TOKEN"Re-syncs the dynamic branch from what is saved, rewrites the conf to the KV metadata dataset and starts a deploy. Use it when publishing failed after an edit was saved (for example a transient KV or scheduler error). It is safe to repeat and does not restart a running streaming job. Only an active aggregate can be re-published; any other returns 409. The response is the aggregate, in the same shape as Create an aggregate.
Aggregate lifecycle
Pausing and deleting stop the aggregate's streaming job, and the job is stopped asynchronously. So both pass through a pending status first, until the job and any deploy runs have stopped. The Hub checks every 15 seconds or so.
status | Name | Meaning |
|---|---|---|
0 | ACTIVE | Deployed daily and streaming. Takes appends and re-publishes. |
1 | PAUSE_PENDING | Paused, waiting for the streaming job and deploy runs to stop. |
2 | PAUSED | No daily deploys and no streaming job. Fetched values are stale. |
3 | DELETE_PENDING | Deleted, waiting for the streaming job and deploy runs to stop; then the aggregate is removed and reads of it return 404. |
ACTIVE --pause--> PAUSE_PENDING --(job stopped)--> PAUSED --resume--> ACTIVE
ACTIVE, PAUSE_PENDING or PAUSED --delete--> DELETE_PENDING --(job stopped)--> removedOnly an ACTIVE aggregate takes appends and re-publishes. Resume a paused one before adding aggregations to it, so an aggregation is never saved where nothing computes it.
Pause an aggregate
curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/pause"
-H "Authorization: Bearer $TOKEN"Stops an active aggregate without removing it:
- The daily deploy schedule is held, so no more deploys run.
- Any deploy run in progress is cancelled.
- The streaming job is cancelled.
The response is the aggregate with status 1 (PAUSE_PENDING). It becomes 2 (PAUSED) once the job has stopped. Pausing an aggregate that is not active returns 409.
While paused, the fetcher keeps serving the values the aggregate last had, and they go stale. Use pause for an aggregate whose deploys keep failing, or one that is not needed for a while.
Resume an aggregate
curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/resume"
-H "Authorization: Bearer $TOKEN"Makes a paused aggregate active again. Resuming:
- rebuilds the aggregate with its team's current settings, so team changes synced while it was paused take effect. If the rebuilt aggregate fails validation, it stays paused and this returns
400; - starts a deploy run, which uploads history from the source table's last complete partition and then starts a new streaming job;
- reactivates the daily schedule from the next day. Days missed while paused are not run.
The response is the aggregate with status 0. Only a paused aggregate can be resumed. While it is still pausing, this returns 409; retry once it is paused.
The new streaming job does not reuse the old job's state, so after a resume, values follow the same timing as a newly created aggregate (see Behaviour and timing).
Delete an aggregate
curl -X DELETE "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card"
-H "Authorization: Bearer $TOKEN"Removes an aggregate, whether it is active, pausing or paused:
- The daily deploy schedule is removed.
- Any deploy run in progress is cancelled.
- The streaming job is cancelled.
The response is the aggregate with status 3 (DELETE_PENDING). Once the job has stopped, the aggregate and its aggregations are removed and the dynamic branch no longer holds it. Reads of it then return 404.
Creating an aggregate with the same name returns 409 while the old one is still being deleted. Once it is gone, the same name can be created again, and it starts from scratch.
After a delete, the fetcher keeps returning the aggregate's last values until they expire. Stop reading it from your rule engine before you delete it.
Fetcher API
Values are served by the fetcher service, not the Hub. Both endpoints take a JSON object with a requests array and return results in request order, so a rule engine can resolve many keys in one round trip.
Read whole aggregates: POST /v1/fetch/aggregations
Each request entry has one key set and several dynamic GroupBys, optionally limited to selected columns. This endpoint reads dynamic groups only, and any other GroupBy, including a hand-written PUSH one, is reported under errors. Use /v1/fetch/aggregates below for a mixed set.
curl -X POST "$FETCHER_URL/v1/fetch/aggregations"
-H "Content-Type: application/json"
-d '{
"requests": [
{
"keys": { "card": "4111111111111111" },
"groupBys": ["fraud.dynamic.card_txns__card", "fraud.dynamic.card_auths__card"],
"columns": { "fraud.dynamic.card_txns__card": ["txn_count_1h", "txn_count_30d"] }
}
]
}'{
"results": [
{
"keys": { "card": "4111111111111111" },
"values": {
"fraud.dynamic.card_txns__card": { "txn_count_1h": 3, "txn_count_30d": 17 },
"fraud.dynamic.card_auths__card": { "auth_count_5m": 0, "auth_count_1h": 4 }
},
"errors": {}
}
]
}| Field | Type | Required | Description |
|---|---|---|---|
requests[].keys | object | yes | A value for every key column of every listed GroupBy. Keys a GroupBy does not use are ignored. |
requests[].groupBys | string[] | yes | Dynamic GroupBy names to read with these keys. |
requests[].columns | object | no | Map from a GroupBy name to the output columns to return for it. GroupBys without an entry return every column. |
A GroupBy that cannot be read (unknown name, no serving info yet, a missing key, an unknown column) is reported under errors keyed by name, and the other GroupBys in the same request still return values. A malformed body returns 400 with {"errors": ["<message>"]}, and a failed KV read fails the whole call with 500 in the same shape.
The columns map only trims the response. All values for a key live in one KV record, so the read is the same either way.
Read selected columns of any GroupBy: POST /v1/fetch/aggregates
One GroupBy per request entry, so a rule engine can pull several cards and several aggregates in a single round trip. Dynamic and regular GroupBys can be mixed.
curl -X POST "$FETCHER_URL/v1/fetch/aggregates"
-H "Content-Type: application/json"
-d '{
"requests": [
{ "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "4111111111111111" }, "columns": ["txn_count_1h"] },
{ "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "5500000000000004" }, "columns": ["txn_count_1h"] }
]
}'{
"results": [
{ "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "4111111111111111" }, "values": { "txn_count_1h": 3 } },
{ "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "5500000000000004" }, "values": { "txn_count_1h": 0 } }
]
}| Field | Type | Required | Description |
|---|---|---|---|
requests[].groupBy | string | yes | Aggregate or GroupBy name. |
requests[].keys | object | yes | A value for every key column of that GroupBy. |
requests[].columns | string[] | no | Output columns to return. Omit to return every column. |
A failing entry carries error instead of values and does not fail the others:
{ "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "4111" }, "error": "Unknown columns txn_count_2h for fraud.dynamic.card_txns__card" }A malformed body returns 400 with {"error": "<message>"}, and a failure that affects the whole call returns 500 in the same shape.
Aggregation fields
Every entry in aggregations, whether creating an aggregate or appending to one.
| Field | Type | Required | Description |
|---|---|---|---|
name | string | yes | Identifier, unique within the aggregate and never reused. Lowercase letters, digits and underscores, starting with a letter; cannot be a key column. It is also the aggregation's input column, so the output columns are <name>_<operation>_<window>. |
operation | string | yes | One of count, sum, unique_count, approx_unique_count, min, max, average, last. |
expression | string | yes | Spark SQL over the source's rawSchema columns, e.g. amount or amount * 100. Required for every operation, and checked by eval. Events where it is NULL are not aggregated, so count counts non-null values. |
filter | string | no | Spark SQL predicate. Only events where it evaluates true are aggregated, e.g. country <> 'GB' or amount > 5000. |
windows | string[] | yes | At least one window length: a positive integer and one of m, h, d — e.g. 1m, 10m, 6h, 30d. Each window becomes one output column. Seconds and unwindowed (all-time) aggregations are not supported. |
Each window becomes <name>_<operation>_<window>, so { "name": "txn", "operation": "count", "windows": ["1m", "10m", "1h", "30d"] } produces txn_count_1m, txn_count_10m, txn_count_1h and txn_count_30d, while { "name": "spend", "operation": "sum", "windows": ["1h"] } produces spend_sum_1h.
For a filtered aggregation the predicate is applied to the expression (IF(<filter>, <expression>, NULL)), so a filter only ever narrows one aggregation and never affects the others.
Behaviour and timing
On create. The first deploy uploads history from the source table's last complete partition (yesterday, UTC) and starts the streaming job. On Kafka, the job starts reading from the beginning of today (UTC), so today's events are counted from midnight rather than from the latest offset. Pub/Sub does not support a start offset, so the job reads from wherever the subscription is. Values are readable from the fetcher once that first upload completes, usually a few minutes after the call returns. Until then, the fetcher reports the aggregate as an error because it has no serving info yet.
New keys. Windows of 2 days or less never depend on batch data, so a brand-new key reads real values for them straight away. Longer windows wait for the key's batch row, and dynamic aggregates limit that wait with first-seen-key grace, which defaults to 1 minute. The grace period starts when the streaming job starts or restarts. Once it has passed, a key with no batch row is treated as having no history and reads a streaming-only value instead of null. The trade-off is that just after a restart, a key whose batch row is still being read may briefly show a low value.
On append. The new columns are served within about a minute. The streaming job polls the conf from the KV metadata dataset (every 10 seconds by default), and the fetcher decodes with the latest conf rather than the upload-time serving info, so a column added seconds ago is readable before its upload finishes. A key's stored record only gains the new columns the next time the job writes that key. Until then they read as 0 or null, as described in 0 versus null below.
What an appended window contains. New aggregations count events from the moment the job applies them. Windows of 2 days or less are computed entirely by the streaming job, so they are complete one window length after the append. Longer windows also need batch history and fill in over three stages:
| Stage | What a window over 2 days contains |
|---|---|
| Right after the append | Only events since the append. |
| After the upload started by the append finishes and the job loads it (up to about 30 minutes later) | History up to midnight (UTC) plus events since the append. Events from midnight up to the append are still missing. |
| After the next daily upload | Complete. |
0 versus null. Counts (count, unique_count, approx_unique_count) over windows of 2 days or less come back as 0 rather than null when the streaming job has nothing for them, including for a key it has never seen. On a freshly appended window, 0 can also mean "not filled yet" until one window length has passed. For longer windows null means "not known yet", for example a key with no batch row during the grace period above. Other operations return null for "no value in this window". These rules apply to dynamic aggregates only.
Resolution. Dynamic aggregates choose their hop size from the window:
| Window | Hop |
|---|---|
| Under 12 minutes | 5 seconds |
| 12 minutes to under 1 hour | 1 minute |
| 1 hour to under 12 hours | 5 minutes |
| 12 hours to under 12 days | 1 hour |
| 12 days or more | 1 day |
A 1-minute window is therefore measured to within 5 seconds instead of being rounded up to a 5-minute hop, and appending a short window later never changes the layout the earlier windows were written with.
Errors
Hub failures return the status code with a JSON body carrying an error field. Its value is the failure's exception text, including a stack trace, so read it for diagnosis and branch on the status code rather than parsing it. (When the API is not mounted at all — no EVAL_URL or no online KV store — the paths are simply unrouted, so the 404 comes back without that body.)
| Status | Meaning | Typical cause |
|---|---|---|
400 | Bad request | A validation failure: an invalid window, an unsupported operation, an unknown team or source, a key column that is not in the source, a duplicate source alias, a reused aggregation name, or a name that breaks the identifier rule. Also a conf that eval rejects, a source table whose schema eval reports it could not read, a source registered before any team defaults are saved (without settingsTeam), a team created without an outputNamespace or named default, or an update or archive of a repo team. |
401 / 403 | Not authorized | A missing or invalid token, or a role below the one the call needs. |
404 | Not found | Unknown aggregate or team name, or the API is not enabled on this Hub. |
409 | Conflict | The team or aggregate name already exists, the aggregate's status does not allow the call, or a team being archived still owns aggregates. For example: appending to or re-publishing an aggregate that is not active, pausing one that is not active, resuming one that is not paused, or creating one whose name is still being deleted. |
500 | Server error | The Hub could not finish the work: an eval call that failed before saving, or a publishing failure after the edit was saved (KV write, branch sync, scheduler, deploy run). Retry the call if the aggregate does not show the change. If it does, use /publish, because retrying a saved create returns 409 and a saved append returns 400. |
Fetcher failures are mostly per request, as described in Fetcher API.
Limitations
- Aggregations are append-only. There is no API to change, remove or rename an existing aggregation or source. A whole aggregate can be deleted. Created teams can be updated and archived, but not renamed.
- A team name is
[a-z_][a-z0-9_]*, and the compute namespace derived from it folds underscores into dashes — sorisk,risk_and_riskall map to one namespace. The Hub reconciles such a namespace for the first team by name and logs the clash; give colliding teams distinct names. - There is no CLI for these calls. Use the Hub UI's Profiles area or the endpoints on this page; the CLI refuses to write to the
dynamicbranch. - Schedule changes through the schedule API are still rejected for dynamic confs. Use pause, resume and delete instead.
- A deleted aggregate's last values stay readable from the fetcher until they expire.
- A source whose stream cannot replay acknowledged messages (Pub/Sub, for example) undercounts windows of up to 2 days after a Flink restart until a full window has passed.
- A source table's columns are read once, at registration. Adding a column to the table does not make it available to aggregations.
- Fetching dynamic aggregates does not convert key values to the key column's type. A
longkey sent as a JSON number that fits in 32 bits, or as a string, fails with a per-request error.stringkey columns are not affected.
Related
- Serving Chronon Data for the fetcher libraries.
- Fetcher API for the non-dynamic fetch endpoints.
- Authentication for roles and tokens.