Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
# Changelog

## Unreleased — resourceType rename + queued time + ip dim + gauge fill

### Breaking

- Renamed the `resource` dimension to `resourceType` across the
events, gauges, and events-daily tables. All Metric column
constants, schema definitions, indexes, projections
(`p_by_resource*` → `p_by_resourceType*`), materialized-view
dimension list, and the `Metric::getResource()` getter (now
`Metric::getResourceType()`) follow the new name. Callers must
rename the `resource` tag key on writes and any queries that
filter/group by the old name. A one-time ClickHouse column rename
plus data backfill is required for existing deployments; that
migration lives in the cloud repo, not in this library.

### Added

- `Accumulator::collect()` accepts an optional `\DateTime $time`
argument. When provided, the row is written at that moment; when
omitted the ClickHouse adapter still stamps `now()`. Events with
the same (tenant, metric, tags) fold into the earliest supplied
time; gauges follow last-write-wins on both value and time.
- `Usage::addBatch()` payloads may carry a `time` field per row
(`DateTime|string`) — the ClickHouse adapter threads it through
`formatDateTime()`.
- New event-only `ip` dimension in `Metric::EVENT_COLUMNS` and
`Metric::getEventSchema()`. Stored as
`LowCardinality(Nullable(String))` in ClickHouse, indexed with a
`bloom_filter`, size 45 chars (IPv6 max). Gauges are unchanged.
`Metric::getIp()` is a typed accessor. The base-table primary key
is intentionally not extended with `ip`.
- Gauge time-series reads now carry the last observed value forward
across empty buckets inside the window (LOCF) instead of collapsing
to zero. Events keep zero-fill. Return shape is unchanged.

## 0.4.0 — Per-dimension ClickHouse projections and auto-routing

This release accelerates grouped reads against the events and gauges
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ $accumulator->collect('project_123', 'bandwidth', 5000, Usage::TYPE_EVENT, [
'method' => 'POST',
'status' => '201',
'service' => 'storage',
'resource' => 'bucket',
'resourceType' => 'bucket',
'resourceId' => 'abc123',
'resourceInternalId' => '42',
'teamId' => 'team_x',
Expand Down
33 changes: 14 additions & 19 deletions src/Usage/Accumulator.php
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,7 @@ class Accumulator
private Usage $usage;

/**
* In-memory buffer for metrics.
* Keyed by "{tenant}:{metric}:{type}:{tagsHash}" — events are summed, gauges
* use last-write-wins. Tenant is part of the key so metrics for different
* tenants never collapse into one entry.
*
* @var array<string, array{tenant: string, metric: string, value: int, type: string, tags: array<string, mixed>}>
* @var array<string, array{tenant: string, metric: string, value: int, type: string, tags: array<string, mixed>, time?: \DateTime}>
*/
private array $buffer = [];

Expand All @@ -40,18 +35,12 @@ public function __construct(Usage $usage)
/**
* Collect a metric into the in-memory buffer for deferred flushing.
*
* For event type: multiple collect() calls for the same metric are summed.
* For gauge type: last-write-wins semantics.
* No period fan-out — raw timestamps are used.
* Events fold additively; earliest non-null time wins on merge.
* Gauges use last-write-wins.
*
* @param string $tenant Tenant scope (shared-tables mode)
* @param string $metric Metric name
* @param int $value Value
* @param string $type Metric type: 'event' or 'gauge'
* @param array<string,mixed> $tags Optional tags
* @return self
* @param array<string,mixed> $tags
*/
public function collect(string $tenant, string $metric, int $value, string $type, array $tags = []): self
public function collect(string $tenant, string $metric, int $value, string $type, array $tags = [], ?\DateTime $time = null): self
{
// Compare against '' rather than empty(): the string "0" is a valid
// tenant/metric id but empty("0") is true in PHP.
Expand All @@ -77,17 +66,23 @@ public function collect(string $tenant, string $metric, int $value, string $type
$key = md5(json_encode([$tenant, $metric, $type, $canonicalTags], JSON_THROW_ON_ERROR));

if ($type === Usage::TYPE_EVENT && isset($this->buffer[$key])) {
// Additive: sum values for the same tenant + metric + tags combination
// earliest time wins on merge
$this->buffer[$key]['value'] += $value;
if ($time !== null && (!isset($this->buffer[$key]['time']) || $time < $this->buffer[$key]['time'])) {
$this->buffer[$key]['time'] = $time;
}
Comment thread
greptile-apps[bot] marked this conversation as resolved.
} else {
// New event entry, or gauge (last-write-wins)
$this->buffer[$key] = [
$entry = [
'tenant' => $tenant,
'metric' => $metric,
'value' => $value,
'type' => $type,
'tags' => $tags,
];
if ($time !== null) {
$entry['time'] = $time;
}
$this->buffer[$key] = $entry;
}

return $this;
Expand Down
2 changes: 1 addition & 1 deletion src/Usage/Adapter.php
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ abstract public function sum(string $tenant, array $queries = [], string $attrib
* the daily events table — gauges are never pre-aggregated.
*
* @param string $tenant Tenant scope (shared-tables mode)
* @param array<\Utopia\Query\Query> $queries Filters (metric, time range, resource, etc.)
* @param array<\Utopia\Query\Query> $queries Filters (metric, time range, resourceType, etc.)
* @return array<Metric>
*/
abstract public function findDaily(string $tenant, array $queries = []): array;
Expand Down
Loading
Loading