Dynamic Aggregations API

Dynamic aggregations are rolling aggregations that are defined at runtime instead of in a GroupBy you commit. They are built for rule engines that need a new counter or a new window during an incident: an analyst adds an aggregation through the Hub, and the running streaming job starts filling it without a git checkout, a recompile, or a Flink restart.

Under the hood an aggregate is an ordinary PUSH (GigaTile) GroupBy that the Hub compiles, validates, publishes and deploys for you. One aggregate exists per (team, source, key columns) triple, and aggregations are only ever appended to it.

Two services are involved:

  • The Hub owns teams, sources and aggregates, and does the writing and publishing.
  • The fetcher serves current values to your rule engine.

This page is the reference for both. The same calls back the Profiles area of the Hub UI (Profiles -> Sources, Aggregates, Fetch, and the API tab, which prints the matching curl for every action you take).

Wiring it up

  1. Register a source: a stream topic plus the table its events land in. Sources are shared by every team.
  2. Pick the owning team. Teams synced from teams.py appear in the Hub automatically; a team that is not in the repo can be created through the API.
  3. Create an aggregate for that team, source and key column set. The Hub derives its name, compiles the conf, uploads it so eval can check it, validates it, then syncs the branch, writes the conf to the KV metadata dataset and starts a deploy. The streaming job reads the conf from KV.
  4. Read values from the fetcher.
  5. Later, append aggregations to the same aggregate. The running job picks up the new conf and starts filling the new columns; nothing restarts.
  6. When an aggregate is no longer needed, pause it (stop its daily deploys and streaming job until you resume it) or delete it.

Before you start

The API is registered only when the Hub has both:

  • EVAL_URL (eval.url), used to read a source table's schema and to validate each conf, and
  • an online KV store, used to publish the conf for the streaming job and the fetcher.

Without them the routes are not mounted and every Hub call below returns 404. EVAL_AUTH_HEADER (eval.authHeader) is optional and is sent as the Authorization header on Hub-to-eval calls. When it is unset and the Hub runs on GCP, the Hub uses a Google identity token from the metadata server instead.

On GCP, the streaming job's service account also needs permission to create the aggregate's _PUSH Bigtable table. The default Dataproc role does not have it, and the failure is not surfaced as an API error.

Hub GET calls need the viewer role, and POST and DELETE calls need operator (see Authentication for roles and token acquisition). The fetcher does not authenticate requests itself, so the fetcher examples send no token; add whatever your gateway requires if it sits behind one.

export HUB_URL="https://zipline.example.com"
export TOKEN="$(zipline auth get-access-token)"   # or: gcloud auth print-identity-token
export FETCHER_URL="https://zipline-fetcher.example.com"

Every example below assumes those variables.

Aggregates live on a dedicated Hub branch named dynamic. That branch is written only by this API: upload, sync, schedule, unschedule, workflow-start and team-sync requests that target the dynamic branch are rejected with an error. So are upload, sync, schedule, unschedule and workflow-start requests for any conf named <team>.dynamic.*.

Hub API

Method Path Role Purpose
POST /dynamic/v1/sources operator Register a source
GET /dynamic/v1/sources viewer List sources with their columns
GET /dynamic/v1/teams viewer List teams that can own aggregates
POST /dynamic/v1/teams operator Create a team outside the repo
GET /dynamic/v1/teams/{name} viewer Read one team, with its own and resolved settings
PUT /dynamic/v1/teams/{name} operator Update a created team
POST /dynamic/v1/teams/{name}/archive operator Archive a created team
GET /dynamic/v1/team-defaults viewer Read the team defaults
PUT /dynamic/v1/team-defaults operator Save the team defaults
POST /dynamic/v1/aggregates operator Create an aggregate
GET /dynamic/v1/aggregates viewer List aggregates
GET /dynamic/v1/aggregates/{name} viewer Read one aggregate
POST /dynamic/v1/aggregates/{name}/aggregations operator Append aggregations
POST /dynamic/v1/aggregates/{name}/publish operator Re-publish the saved conf
POST /dynamic/v1/aggregates/{name}/pause operator Stop an aggregate's daily deploys and streaming job
POST /dynamic/v1/aggregates/{name}/resume operator Restart a paused aggregate
DELETE /dynamic/v1/aggregates/{name} operator Stop and remove an aggregate

Aggregate names are identifiers joined by dots and underscores, so they need no URL encoding.

Register a source

curl -X POST "$HUB_URL/dynamic/v1/sources" 
  -H "Authorization: Bearer $TOKEN" 
  -H "Content-Type: application/json" 
  -d '{
    "alias": "card_txns",
    "topic": "pubsub://card-transactions/serde=pubsub_schema/project=my-project/schemaId=card-txn-schema/subscription=card-txns-sub",
    "table": "dynamic_demo.card_transactions",
    "timeColumn": "ts"
  }'

Response:

{
  "alias": "card_txns",
  "topic": "pubsub://card-transactions/serde=pubsub_schema/project=my-project/schemaId=card-txn-schema/subscription=card-txns-sub",
  "table": "dynamic_demo.card_transactions",
  "timeColumn": "ts",
  "rawSchema": "event_id:string,card:string,amount:long,country:string,ts:long"
}
Field Type Required Description
alias string yes Short name, used in aggregate names. Lowercase letters, digits and underscores, starting with a letter; registered once — re-registering it returns 400.
topic string yes Stream URI, including its serde, e.g. pubsub://<topic>/serde=pubsub_schema/project=<project>/schemaId=<schema>/subscription=<subscription> or kafka://<topic>/serde=schema_registry/registry_host=<host>.
table string yes Table the events land in. Supplies the columns and the history for long windows.
timeColumn string yes Event time column, in epoch milliseconds.
settingsTeam string no Team whose catalog settings read the table's schema instead of the team defaults, for a table only that team's catalog can read. Not stored.

The Hub reads the table's columns and types at registration, with the catalog settings of the team defaults (or of settingsTeam), and stores them as rawSchema; aggregations and key columns must be drawn from those columns. A partition column named ds is dropped, because stream events never carry it. Supported column types are long, int, double, float, string, boolean and bytes; columns of any other type are left out of rawSchema and cannot be used.

Without saved team defaults or a settingsTeam, registration is rejected with 400. Sources belong to no team: every team can build aggregates on one, and each aggregate is validated with its own team's settings when it is saved.

Check the source before you register it, because it cannot be changed afterwards:

  • Every column in rawSchema must be present in the stream's events, or the streaming job fails to start. That includes table-only columns, and a partition column with a name other than ds.
  • The Hub only checks that topic and timeColumn are non-empty. A wrong stream URI or time column surfaces later, when an aggregate is validated or its streaming job starts.

List sources

curl "$HUB_URL/dynamic/v1/sources" -H "Authorization: Bearer $TOKEN"
{
  "sources": [
    {
      "alias": "card_txns",
      "topic": "pubsub://card-transactions/serde=pubsub_schema/project=my-project/schemaId=card-txn-schema/subscription=card-txns-sub",
      "table": "dynamic_demo.card_transactions",
      "timeColumn": "ts",
      "rawSchema": "event_id:string,card:string,amount:long,country:string,ts:long"
    }
  ]
}

List teams

curl "$HUB_URL/dynamic/v1/teams" -H "Authorization: Bearer $TOKEN"
{
  "teams": ["fraud"],
  "details": [
    {
      "name": "fraud",
      "origin": "repo",
      "outputNamespace": "dynamic_demo",
      "email": "fraud-oncall@example.com"
    }
  ]
}

origin is repo for a team synced from teams.py in your Chronon repo, and profiles for a team created with the call below. teams is kept for existing callers; details carries the same teams with their metadata.

Create a team

Only needed for teams that are not in the Chronon repo. A created team's settings are layered like a team in teams.py: its own settings over the team defaults, with the team's values winning.

curl -X POST "$HUB_URL/dynamic/v1/teams" 
  -H "Authorization: Bearer $TOKEN" 
  -H "Content-Type: application/json" 
  -d '{
    "name": "payments",
    "email": "payments@example.com",
    "description": "Payments risk rules",
    "outputNamespace": "dynamic_demo",
    "settings": {
      "env": {
        "common": { "CUSTOMER_ID": "payments" },
        "modeEnvironments": { "deploy": { "SPARK_CLUSTER_NAME": "payments-online" } }
      },
      "conf": { "common": { "spark.executor.memory": "4g" } }
    }
  }'
{
  "name": "payments",
  "origin": "profiles",
  "outputNamespace": "dynamic_demo",
  "email": "payments@example.com",
  "description": "Payments risk rules"
}
Field Type Required Description
name string yes Lowercase letters, digits and underscores. Becomes the first segment of the team's aggregate names. default is reserved for the team defaults.
email string no Owner contact.
description string no Shown in team lists.
outputNamespace string yes, unless baseTeam is given Namespace for the team's tables; letters, digits and underscores, optionally catalog.namespace. Defaults to baseTeam's when that is given.
settings object no The team's own settings: env, conf and clusterConf, each with common and per-mode maps (modeEnvironments, modeConfigs, modeClusterConfigs), as EnvironmentVariables, ConfigProperties and ClusterConfigProperties are written in teams.py.
baseTeam string no Team whose resolved settings become the new team's own, so its aggregates run where that team's do. Ignored when settings is given.

Without settings or baseTeam, the team has no settings of its own: it runs with the team defaults, or with no settings at all until defaults are saved. Nothing is copied from another team unless you name it in baseTeam.

A name already used by a repo team or a created team is rejected with 409; default, a missing outputNamespace or an unknown baseTeam with 400. If teams.py later adds a team with the same name, the team created here takes precedence for aggregates.

Read a team

curl "$HUB_URL/dynamic/v1/teams/payments" -H "Authorization: Bearer $TOKEN"
{
  "name": "payments",
  "origin": "profiles",
  "outputNamespace": "dynamic_demo",
  "email": "payments@example.com",
  "settings": {
    "env": { "common": { "CUSTOMER_ID": "payments" }, "modeEnvironments": { "deploy": { "SPARK_CLUSTER_NAME": "payments-online" } } }
  },
  "resolved": {
    "env": {
      "common": { "CUSTOMER_ID": "payments", "HUB_URL": "https://zipline.example.com" },
      "modeEnvironments": {
        "deploy": { "CUSTOMER_ID": "payments", "HUB_URL": "https://zipline.example.com", "SPARK_CLUSTER_NAME": "payments-online" }
      }
    }
  }
}

settings is what the team sets itself, and is only returned for created teams. resolved is what its aggregates run with: the team defaults merged with settings by the same rules teams.py compiles with — the team's values win, and each layer's common is folded into every mode. For a repo team, resolved is its compiled settings. An unknown team returns 404.

Update a team

curl -X PUT "$HUB_URL/dynamic/v1/teams/payments" 
  -H "Authorization: Bearer $TOKEN" 
  -H "Content-Type: application/json" 
  -d '{
    "email": "payments@example.com",
    "settings": { "env": { "common": { "CUSTOMER_ID": "payments" } } }
  }'

Returns the team as Read a team does.

Field Type Required Description
email string no Owner contact. Cleared when left out.
description string no Cleared when left out.
outputNamespace string no Kept when left out.
settings object no Replaces all of the team's own settings — anything not sent is removed. Kept as they are when left out.

Only created teams can be updated; a repo team is rejected with 400 and is changed by editing teams.py and syncing. The team's aggregates take the new settings the next time they are saved or published.

Archive a team

curl -X POST "$HUB_URL/dynamic/v1/teams/payments/archive" -H "Authorization: Bearer $TOKEN"

Archives a created team, so it is no longer listed and can no longer own aggregates. A team that still owns aggregates — paused ones and ones still being deleted included — is refused with 409, naming them; delete them first. A repo team is rejected with 400; remove it from teams.py instead.

Team defaults

The team defaults are the settings every created team inherits, like the default team in teams.py. They hold the shared environment, engine conf and cluster settings (catalogs, project, cluster), so a created team only needs to set what differs. Registering a source also reads its table with them.

curl "$HUB_URL/dynamic/v1/team-defaults" -H "Authorization: Bearer $TOKEN"
{
  "settings": {
    "env": { "common": { "HUB_URL": "https://zipline.example.com" } },
    "conf": { "common": { "spark.sql.catalog.spark_catalog.warehouse": "gs://my-warehouse/data/tables/" } }
  }
}

settings is missing until the defaults are first saved. To save them, send the whole set — it replaces the previous defaults:

curl -X PUT "$HUB_URL/dynamic/v1/team-defaults" 
  -H "Authorization: Bearer $TOKEN" 
  -H "Content-Type: application/json" 
  -d '{ "settings": { "env": { "common": { "HUB_URL": "https://zipline.example.com" } } } }'
{
  "settings": { "env": { "common": { "HUB_URL": "https://zipline.example.com" } } },
  "updatedTeams": 3,
  "affectedAggregates": 5
}

Saving recomputes the resolved settings of every created team in the same transaction. updatedTeams counts them; their affectedAggregates keep running with the settings they were built with until they are next saved or published. Repo teams are unaffected — their defaults come from teams.py.

Where a team lives

Both kinds of team are rows in the same zipline.teams table, keyed by (branch, name). A team synced from teams.py lives on the main branch; a team created here lives on the dynamic branch, as do the team defaults, in a row named default. That is why syncing the repo never removes a created team — a sync only reconciles its own branch — and why a dynamic row shadows a main row of the same name wherever the two could disagree.

There is one dynamic branch, and it is the only branch this API writes. You cannot branch from it, and the CLI is refused write access to it. Correspondingly, the UI can edit a team on the dynamic branch and shows a team on any other branch read-only, which is also how it treats confs: a teams.py team is changed by editing teams.py and syncing.

Compute namespaces

Creating a team also provisions the Kubernetes compute namespace its jobs run in, on deployments where the Hub is configured to do so (CRUCIBLE_TEAM_NAMESPACE_RECONCILE_ENABLED). The namespace is named from the team, labelled with it so cloud spend can be split per team, and given the engine service accounts. The call returns once that work is done, so a team you create is ready to run in.

Namespaces are only ever created, never deleted, so archiving a team leaves its namespace and anything still running in it alone.

Create an aggregate

curl -X POST "$HUB_URL/dynamic/v1/aggregates" 
  -H "Authorization: Bearer $TOKEN" 
  -H "Content-Type: application/json" 
  -d '{
    "team": "fraud",
    "sourceAlias": "card_txns",
    "keyColumns": ["card"],
    "aggregations": [
      {
        "name": "txn",
        "operation": "count",
        "expression": "amount",
        "windows": ["1m", "10m", "1h", "30d"]
      },
      {
        "name": "foreign",
        "operation": "count",
        "expression": "amount",
        "filter": "country <> '"'"'GB'"'"'",
        "windows": ["10m", "1h"]
      }
    ]
  }'

Response:

{
  "name": "fraud.dynamic.card_txns__card",
  "team": "fraud",
  "sourceAlias": "card_txns",
  "keyColumns": ["card"],
  "confHash": "3f9a1c",
  "aggregations": [
    {
      "name": "txn",
      "operation": "count",
      "expression": "amount",
      "windows": ["1m", "10m", "1h", "30d"],
      "position": 0,
      "status": "active",
      "outputColumns": ["txn_count_1m", "txn_count_10m", "txn_count_1h", "txn_count_30d"]
    },
    {
      "name": "foreign",
      "operation": "count",
      "expression": "amount",
      "filter": "country <> 'GB'",
      "windows": ["10m", "1h"],
      "position": 1,
      "status": "active",
      "outputColumns": ["foreign_count_10m", "foreign_count_1h"]
    }
  ]
}

The '"'"' sequence closes and reopens the shell's single-quoted string, so a SQL string literal such as 'GB' can appear inside the JSON body. The Hub UI prints bodies this way already, so a curl copied from the API tab can be pasted as is.

Field Type Required Description
team string yes Owning team, from /dynamic/v1/teams.
sourceAlias string yes A registered source alias.
keyColumns string[] yes Source columns to aggregate by, e.g. ["card"]. Each must exist in the source's rawSchema and be a lowercase identifier (letters, digits and underscores), so a column such as cardId cannot be a key.
aggregations Aggregation[] yes At least one; see Aggregation fields.

The name is derived, not sent: <team>.dynamic.<sourceAlias>__<keyColumns joined by _>, so (fraud, card_txns, [card]) becomes fraud.dynamic.card_txns__card and (fraud, card_txns, [card, merchant]) becomes fraud.dynamic.card_txns__card_merchant. Key order is part of the name, so [merchant, card] is a separate aggregate. Creating an aggregate that already exists returns 409; append to it instead.

The Hub works in this order:

  1. Validates the aggregation specs and compiles the GroupBy.
  2. Stores the conf in the Hub so eval can read it, and validates it with eval. A conf that eval rejects fails the whole call with 400 and nothing is saved.
  3. Saves the aggregate.
  4. Publishes it: syncs the dynamic branch, writes the conf to the KV metadata dataset, and starts a deploy. The deploy uploads history from the source table, starts the streaming job and keeps the upload scheduled daily.

The call returns once the deploy has been started, not when it finishes. See Behaviour and timing.

Response field Description
name The derived aggregate name.
confHash Hash of the current conf. Changes on every append.
status The aggregate's lifecycle status, as a number: 0 active, 1 pausing, 2 paused, 3 deleting. A new aggregate is 0.
aggregations[].position Order in the conf, starting at 0. Appended aggregations take the next positions.
aggregations[].status Always active today.
aggregations[].outputColumns The columns to read from the fetcher.

Append aggregations

curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/aggregations" 
  -H "Authorization: Bearer $TOKEN" 
  -H "Content-Type: application/json" 
  -d '{
    "aggregations": [
      {
        "name": "big_charges",
        "operation": "count",
        "expression": "amount",
        "filter": "amount > 5000",
        "windows": ["5m", "1h"]
      }
    ]
  }'

The response is the updated aggregate, in the same shape as Create an aggregate.

Appending is the only way to change an aggregate's aggregations. The aggregate must be active: appending to one that is pausing, paused or being deleted returns 409.

  • Existing aggregations never change, and an aggregation name cannot be reused within an aggregate. Reusing one returns 400.
  • New output columns are readable within about a minute; see Behaviour and timing for what they contain, because they start out incomplete.
  • The streaming job applies them in place. The deploy the append starts runs the upload and leaves the running job alone.

Read an aggregate

curl "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card" -H "Authorization: Bearer $TOKEN"

Returns the same shape as create, plus the daily deploy schedule:

Response field Description
scheduleState 1 active or 2 paused. Absent once the schedule is removed, on delete.
nextScheduleTime When the next daily deploy runs, while the schedule is active.

GET /dynamic/v1/aggregates lists every aggregate, with its status but without the schedule fields:

{ "groups": [ { "name": "fraud.dynamic.card_txns__card", "team": "fraud", "...": "..." } ] }

Re-publish an aggregate

curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/publish" 
  -H "Authorization: Bearer $TOKEN"

Re-syncs the dynamic branch from what is saved, rewrites the conf to the KV metadata dataset and starts a deploy. Use it when publishing failed after an edit was saved (for example a transient KV or scheduler error). It is safe to repeat and does not restart a running streaming job. Only an active aggregate can be re-published; any other returns 409. The response is the aggregate, in the same shape as Create an aggregate.

Aggregate lifecycle

Pausing and deleting stop the aggregate's streaming job, and the job is stopped asynchronously. So both pass through a pending status first, until the job and any deploy runs have stopped. The Hub checks every 15 seconds or so.

status Name Meaning
0 ACTIVE Deployed daily and streaming. Takes appends and re-publishes.
1 PAUSE_PENDING Paused, waiting for the streaming job and deploy runs to stop.
2 PAUSED No daily deploys and no streaming job. Fetched values are stale.
3 DELETE_PENDING Deleted, waiting for the streaming job and deploy runs to stop; then the aggregate is removed and reads of it return 404.
ACTIVE --pause--> PAUSE_PENDING --(job stopped)--> PAUSED --resume--> ACTIVE
ACTIVE, PAUSE_PENDING or PAUSED --delete--> DELETE_PENDING --(job stopped)--> removed

Only an ACTIVE aggregate takes appends and re-publishes. Resume a paused one before adding aggregations to it, so an aggregation is never saved where nothing computes it.

Pause an aggregate

curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/pause" 
  -H "Authorization: Bearer $TOKEN"

Stops an active aggregate without removing it:

  • The daily deploy schedule is held, so no more deploys run.
  • Any deploy run in progress is cancelled.
  • The streaming job is cancelled.

The response is the aggregate with status 1 (PAUSE_PENDING). It becomes 2 (PAUSED) once the job has stopped. Pausing an aggregate that is not active returns 409.

While paused, the fetcher keeps serving the values the aggregate last had, and they go stale. Use pause for an aggregate whose deploys keep failing, or one that is not needed for a while.

Resume an aggregate

curl -X POST "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card/resume" 
  -H "Authorization: Bearer $TOKEN"

Makes a paused aggregate active again. Resuming:

  • rebuilds the aggregate with its team's current settings, so team changes synced while it was paused take effect. If the rebuilt aggregate fails validation, it stays paused and this returns 400;
  • starts a deploy run, which uploads history from the source table's last complete partition and then starts a new streaming job;
  • reactivates the daily schedule from the next day. Days missed while paused are not run.

The response is the aggregate with status 0. Only a paused aggregate can be resumed. While it is still pausing, this returns 409; retry once it is paused.

The new streaming job does not reuse the old job's state, so after a resume, values follow the same timing as a newly created aggregate (see Behaviour and timing).

Delete an aggregate

curl -X DELETE "$HUB_URL/dynamic/v1/aggregates/fraud.dynamic.card_txns__card" 
  -H "Authorization: Bearer $TOKEN"

Removes an aggregate, whether it is active, pausing or paused:

  • The daily deploy schedule is removed.
  • Any deploy run in progress is cancelled.
  • The streaming job is cancelled.

The response is the aggregate with status 3 (DELETE_PENDING). Once the job has stopped, the aggregate and its aggregations are removed and the dynamic branch no longer holds it. Reads of it then return 404.

Creating an aggregate with the same name returns 409 while the old one is still being deleted. Once it is gone, the same name can be created again, and it starts from scratch.

After a delete, the fetcher keeps returning the aggregate's last values until they expire. Stop reading it from your rule engine before you delete it.

Fetcher API

Values are served by the fetcher service, not the Hub. Both endpoints take a JSON object with a requests array and return results in request order, so a rule engine can resolve many keys in one round trip.

Read whole aggregates: POST /v1/fetch/aggregations

Each request entry has one key set and several dynamic GroupBys, optionally limited to selected columns. This endpoint reads dynamic groups only, and any other GroupBy, including a hand-written PUSH one, is reported under errors. Use /v1/fetch/aggregates below for a mixed set.

curl -X POST "$FETCHER_URL/v1/fetch/aggregations" 
  -H "Content-Type: application/json" 
  -d '{
    "requests": [
      {
        "keys": { "card": "4111111111111111" },
        "groupBys": ["fraud.dynamic.card_txns__card", "fraud.dynamic.card_auths__card"],
        "columns": { "fraud.dynamic.card_txns__card": ["txn_count_1h", "txn_count_30d"] }
      }
    ]
  }'
{
  "results": [
    {
      "keys": { "card": "4111111111111111" },
      "values": {
        "fraud.dynamic.card_txns__card": { "txn_count_1h": 3, "txn_count_30d": 17 },
        "fraud.dynamic.card_auths__card": { "auth_count_5m": 0, "auth_count_1h": 4 }
      },
      "errors": {}
    }
  ]
}
Field Type Required Description
requests[].keys object yes A value for every key column of every listed GroupBy. Keys a GroupBy does not use are ignored.
requests[].groupBys string[] yes Dynamic GroupBy names to read with these keys.
requests[].columns object no Map from a GroupBy name to the output columns to return for it. GroupBys without an entry return every column.

A GroupBy that cannot be read (unknown name, no serving info yet, a missing key, an unknown column) is reported under errors keyed by name, and the other GroupBys in the same request still return values. A malformed body returns 400 with {"errors": ["<message>"]}, and a failed KV read fails the whole call with 500 in the same shape.

The columns map only trims the response. All values for a key live in one KV record, so the read is the same either way.

Read selected columns of any GroupBy: POST /v1/fetch/aggregates

One GroupBy per request entry, so a rule engine can pull several cards and several aggregates in a single round trip. Dynamic and regular GroupBys can be mixed.

curl -X POST "$FETCHER_URL/v1/fetch/aggregates" 
  -H "Content-Type: application/json" 
  -d '{
    "requests": [
      { "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "4111111111111111" }, "columns": ["txn_count_1h"] },
      { "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "5500000000000004" }, "columns": ["txn_count_1h"] }
    ]
  }'
{
  "results": [
    { "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "4111111111111111" }, "values": { "txn_count_1h": 3 } },
    { "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "5500000000000004" }, "values": { "txn_count_1h": 0 } }
  ]
}
Field Type Required Description
requests[].groupBy string yes Aggregate or GroupBy name.
requests[].keys object yes A value for every key column of that GroupBy.
requests[].columns string[] no Output columns to return. Omit to return every column.

A failing entry carries error instead of values and does not fail the others:

{ "groupBy": "fraud.dynamic.card_txns__card", "keys": { "card": "4111" }, "error": "Unknown columns txn_count_2h for fraud.dynamic.card_txns__card" }

A malformed body returns 400 with {"error": "<message>"}, and a failure that affects the whole call returns 500 in the same shape.

Aggregation fields

Every entry in aggregations, whether creating an aggregate or appending to one.

Field Type Required Description
name string yes Identifier, unique within the aggregate and never reused. Lowercase letters, digits and underscores, starting with a letter; cannot be a key column. It is also the aggregation's input column, so the output columns are <name>_<operation>_<window>.
operation string yes One of count, sum, unique_count, approx_unique_count, min, max, average, last.
expression string yes Spark SQL over the source's rawSchema columns, e.g. amount or amount * 100. Required for every operation, and checked by eval. Events where it is NULL are not aggregated, so count counts non-null values.
filter string no Spark SQL predicate. Only events where it evaluates true are aggregated, e.g. country <> 'GB' or amount > 5000.
windows string[] yes At least one window length: a positive integer and one of m, h, d — e.g. 1m, 10m, 6h, 30d. Each window becomes one output column. Seconds and unwindowed (all-time) aggregations are not supported.

Each window becomes <name>_<operation>_<window>, so { "name": "txn", "operation": "count", "windows": ["1m", "10m", "1h", "30d"] } produces txn_count_1m, txn_count_10m, txn_count_1h and txn_count_30d, while { "name": "spend", "operation": "sum", "windows": ["1h"] } produces spend_sum_1h.

For a filtered aggregation the predicate is applied to the expression (IF(<filter>, <expression>, NULL)), so a filter only ever narrows one aggregation and never affects the others.

Behaviour and timing

On create. The first deploy uploads history from the source table's last complete partition (yesterday, UTC) and starts the streaming job. On Kafka, the job starts reading from the beginning of today (UTC), so today's events are counted from midnight rather than from the latest offset. Pub/Sub does not support a start offset, so the job reads from wherever the subscription is. Values are readable from the fetcher once that first upload completes, usually a few minutes after the call returns. Until then, the fetcher reports the aggregate as an error because it has no serving info yet.

New keys. Windows of 2 days or less never depend on batch data, so a brand-new key reads real values for them straight away. Longer windows wait for the key's batch row, and dynamic aggregates limit that wait with first-seen-key grace, which defaults to 1 minute. The grace period starts when the streaming job starts or restarts. Once it has passed, a key with no batch row is treated as having no history and reads a streaming-only value instead of null. The trade-off is that just after a restart, a key whose batch row is still being read may briefly show a low value.

On append. The new columns are served within about a minute. The streaming job polls the conf from the KV metadata dataset (every 10 seconds by default), and the fetcher decodes with the latest conf rather than the upload-time serving info, so a column added seconds ago is readable before its upload finishes. A key's stored record only gains the new columns the next time the job writes that key. Until then they read as 0 or null, as described in 0 versus null below.

What an appended window contains. New aggregations count events from the moment the job applies them. Windows of 2 days or less are computed entirely by the streaming job, so they are complete one window length after the append. Longer windows also need batch history and fill in over three stages:

Stage What a window over 2 days contains
Right after the append Only events since the append.
After the upload started by the append finishes and the job loads it (up to about 30 minutes later) History up to midnight (UTC) plus events since the append. Events from midnight up to the append are still missing.
After the next daily upload Complete.

0 versus null. Counts (count, unique_count, approx_unique_count) over windows of 2 days or less come back as 0 rather than null when the streaming job has nothing for them, including for a key it has never seen. On a freshly appended window, 0 can also mean "not filled yet" until one window length has passed. For longer windows null means "not known yet", for example a key with no batch row during the grace period above. Other operations return null for "no value in this window". These rules apply to dynamic aggregates only.

Resolution. Dynamic aggregates choose their hop size from the window:

Window Hop
Under 12 minutes 5 seconds
12 minutes to under 1 hour 1 minute
1 hour to under 12 hours 5 minutes
12 hours to under 12 days 1 hour
12 days or more 1 day

A 1-minute window is therefore measured to within 5 seconds instead of being rounded up to a 5-minute hop, and appending a short window later never changes the layout the earlier windows were written with.

Errors

Hub failures return the status code with a JSON body carrying an error field. Its value is the failure's exception text, including a stack trace, so read it for diagnosis and branch on the status code rather than parsing it. (When the API is not mounted at all — no EVAL_URL or no online KV store — the paths are simply unrouted, so the 404 comes back without that body.)

Status Meaning Typical cause
400 Bad request A validation failure: an invalid window, an unsupported operation, an unknown team or source, a key column that is not in the source, a duplicate source alias, a reused aggregation name, or a name that breaks the identifier rule. Also a conf that eval rejects, a source table whose schema eval reports it could not read, a source registered before any team defaults are saved (without settingsTeam), a team created without an outputNamespace or named default, or an update or archive of a repo team.
401 / 403 Not authorized A missing or invalid token, or a role below the one the call needs.
404 Not found Unknown aggregate or team name, or the API is not enabled on this Hub.
409 Conflict The team or aggregate name already exists, the aggregate's status does not allow the call, or a team being archived still owns aggregates. For example: appending to or re-publishing an aggregate that is not active, pausing one that is not active, resuming one that is not paused, or creating one whose name is still being deleted.
500 Server error The Hub could not finish the work: an eval call that failed before saving, or a publishing failure after the edit was saved (KV write, branch sync, scheduler, deploy run). Retry the call if the aggregate does not show the change. If it does, use /publish, because retrying a saved create returns 409 and a saved append returns 400.

Fetcher failures are mostly per request, as described in Fetcher API.

Limitations

  • Aggregations are append-only. There is no API to change, remove or rename an existing aggregation or source. A whole aggregate can be deleted. Created teams can be updated and archived, but not renamed.
  • A team name is [a-z_][a-z0-9_]*, and the compute namespace derived from it folds underscores into dashes — so risk, risk_ and _risk all map to one namespace. The Hub reconciles such a namespace for the first team by name and logs the clash; give colliding teams distinct names.
  • There is no CLI for these calls. Use the Hub UI's Profiles area or the endpoints on this page; the CLI refuses to write to the dynamic branch.
  • Schedule changes through the schedule API are still rejected for dynamic confs. Use pause, resume and delete instead.
  • A deleted aggregate's last values stay readable from the fetcher until they expire.
  • A source whose stream cannot replay acknowledged messages (Pub/Sub, for example) undercounts windows of up to 2 days after a Flink restart until a full window has passed.
  • A source table's columns are read once, at registration. Adding a column to the table does not make it available to aggregations.
  • Fetching dynamic aggregates does not convert key values to the key column's type. A long key sent as a JSON number that fits in 32 bits, or as a string, fails with a per-request error. string key columns are not affected.
Edit this page