zeekayandHanzo Dev f37c8d89da sightings: a dimension row is written once per window, not once per batch ...

The ingest writers issue one INSERT round-trip per table, and the receivers
call them SYNCHRONOUSLY on the connection handler — so round-trips per batch
are intake latency, not just database load. datastorelogs took five per batch,
datastoretraces six. Four and five of those wrote the same dimension rows over
and over.

Measured on the live plane, from the 14-minute window this writer last held
production traffic:

	event.log_attribute     21,475 rows written →    52 distinct     413x
	event.log_key            3,902 rows written →     5 distinct     780x
	event.log_resource_key  17,572 rows written →    17 distinct   1,034x
	event.log_resource       1,302 rows written →    65 distinct      20x

44,251 dimension rows carrying 139 rows of information, in front of the socket
that drains the fleet's logs. In that window this path sustained ~390 event.log
rows/min against a ~5,500 rows/min baseline.

pkg/sightings remembers what has already been written and drops the repeats.
Steady state falls from five round-trips per batch to ONE — the fact table
alone — proven end-to-end in TestSteadyStateCostsOneRoundTrip against a conn
that counts prepared INSERTs.

Three invariants, each with a test, because each is a way to lose data rather
than merely re-send it:

  - Only DIMENSION rows are suppressed. event.log and event.span are facts. And
    event.trace is a PARTIAL contribution folded by an AggregatingMergeTree —
    suppressing a repeat there would quietly subtract spans from a trace's own
    span count, so it has no Set at all.
  - A row is recorded only after its write SUCCEEDS. Remembering a failed
    INSERT would drop that sighting permanently; sightings.Write enforces the
    ordering so no call site can get it wrong.
  - One Set per TABLE, never per row type. log_key and log_resource_key are
    both keyRow; a shared Set would let a key seen as a tag suppress the same
    name as a resource key.

Keys carry the 1800s resource bucket, so a sighting refreshes once per window
instead of never: log_attribute.unix_milli and log_resource.seen_at_ts_bucket_start
stay accurate to one bucket. operationRow is keyed WITHOUT its time, which is
the latest span start and therefore differs every batch — (service, name) is
the identity that repeats.

Bounded by two map generations rather than an LRU: forgetting early costs a
redundant write, never a lost row, so the cheap structure is the correct one.

Co-authored-by: Hanzo Dev <[email protected]>

2026-08-01 19:53:38 -07:00

..

sightings_test.go

sightings: a dimension row is written once per window, not once per batch

2026-08-01 19:53:38 -07:00

sightings.go

sightings: a dimension row is written once per window, not once per batch

2026-08-01 19:53:38 -07:00