Beacon ingestion with Valkey Router and Momento Cache

Live events concentrate activity across large populations of TVs, browsers and mobile devices. Those devices send beacons as people watch, interact and experience changes in playback. The resulting traffic can surge suddenly, and its profile is difficult to predict. Critical internal services need to keep processing event data while the audience changes the volume and pace of ingestion.

A beacon carries device telemetry with selected signals for downstream analysis. The event data it carries might support quality-of-experience monitoring, anti-piracy detection or personalization. Selecting those signals matters to ingestion as well as analysis. Every added field increases the payload, and those extra bytes multiply across the beacon traffic that the system must receive and hold.

Beacon ingestion therefore needs a protective boundary between device traffic and internal processing. In this architecture, Momento Valkey Router receives beacons through a gateway tier, manages device connections and batches data for forwarding. Shared, open-source Valkey collects related events and buffers them until downstream services can process them. Together, the gateway and Valkey turn an unpredictable beacon firehose into organized data that internal services can consume at a manageable rate.

Valkey Router and shared Valkey collection form a protective ingestion boundary between device traffic and downstream services. Kafka, ClickHouse and S3 illustrate downstream choices.

Protect internal services from device traffic

The gateway separates the audience’s connections from the connections that internal infrastructure must support. Without that boundary, a growing device population can increase database connection pressure as well as beacon volume. These are distinct demands. An internal service might have enough processing capacity for the events while struggling to manage the connections delivering them.

Valkey Router handles device connections and reuses a pool of connections to Valkey. Pooling prevents each device connection from becoming a separate Valkey connection. The gateway can receive traffic from many devices while controlling the number of database connections. Momento supplies this gateway layer alongside Valkey’s open-source storage foundation, placing device-facing capabilities ahead of shared event collection.

The gateway also controls access to that collection. Authentication and access checks determine whether incoming traffic is allowed to reach internal infrastructure. HTTPS and TLS termination protect device-to-gateway communication, and custom certificates let the endpoint use the certificates required by the deployment. These capabilities put connection and access handling at the boundary where device traffic enters the system.

Throttling limits the forwarding rate when incoming traffic would otherwise overwhelm the next stage. The gateway controls ingestion before excess traffic reaches the database and consumers. Connection pooling and throttling address different parts of the same protection. One controls database connections, while the other controls the rate of data reaching internal services.

Share forwarding overhead across small beacons

Managing connections still leaves a large number of small payloads to forward. Sending every beacon separately repeats communication and request-handling overhead. When that repeated overhead consumes a substantial part of ingestion capacity, the system spends effort moving small pieces of telemetry instead of collecting their event data efficiently.

Valkey Router batches compatible beacons locally before forwarding them. Several beacons share the overhead of a batch, reducing the repeated forwarding cost per beacon. Their event data remains available for collection. Batching changes how data is transferred without requiring the application to remove the observations it needs for analysis.

Batching introduces a short wait at the gateway because data must accumulate before it can be sent together. The gateway bounds that wait and the amount of data it gathers. A count trigger sends a batch when enough beacons have accumulated. A timer sends the accumulated data when the waiting interval ends. During busy periods, the count trigger can fill batches quickly. During quieter periods, the timer lets small batches progress instead of leaving beacons waiting for traffic that may not arrive soon.

A size bound can also limit how much data a batch contains. This matters because beacon sizes can vary with the selected signals. A batch with an acceptable beacon count could still contain too many bytes if those beacons have larger payloads. Count, time and size bounds serve complementary purposes. They limit local accumulation while balancing forwarding efficiency against the delay introduced by waiting.

Each gateway assembles its own batches from the traffic it receives. This keeps batching close to device ingestion, where small payloads first accumulate. The local batch holds data briefly to improve transfer efficiency. Shared Valkey collection serves the longer-lived need to hold events for downstream processing. Efficient batches from separate gateways must also reach the right collection point, because each gateway may receive only part of a partition’s activity.

Bring related events to the same collection point

A gateway tier distributes ingestion across multiple instances, but related beacons can reach different gateways. A consumer processing an application-defined partition needs its related events together, even when they come from different devices. Leaving those events fragmented across gateway-local collections would push the task of gathering them onto every downstream consumer.

The architecture collates related events by deriving a grouping key deterministically from beacon properties. Every gateway applies the same grouping rule. Beacons with the same grouping property produce the same grouping key, giving their related activity a consistent logical address. The gateways use that address to route event data to the same Valkey shard. The shared collection brings related events together in a coherent data stream for downstream consumption.

Consider two devices sending beacons assigned to partition S42. One beacon reaches one Valkey Router instance and the other reaches another. Both carry the partition identifier, and both gateways derive the grouping key partition:S42 from it. Each gateway uses that logical address to route the related event data to the selected Valkey shard. The partition’s events meet in shared collection even though different devices and gateways handled the beacons.

The selected shard is a portion of shared Valkey collection that holds multiple keyed groups. The grouping key keeps the partition’s events associated alongside data for other partitions. A consumer can process the collected partition activity without gathering fragments from individual gateways. This separates the gateway that receives a beacon from the collection where its related events belong, allowing the ingestion tier to distribute device traffic while preserving the relationships downstream analysis needs.

Each gateway buffers events locally for efficient forwarding in batches. Matching partition keys bring related events from different gateways to the same Valkey collection point. The arrows follow the three marked S42 events; other partitions and shard placements are illustrative.

Give events room to wait for consumers

Organized events can still arrive faster than downstream services process them. A traffic burst increases ingestion immediately, while a consumer may continue at its previous processing rate. Consumer slowdowns can create the same imbalance even when incoming traffic remains steady. Shared buffering gives the architecture somewhere to hold that difference.

Valkey holds collected event data while consumers process it. During a burst, the buffer grows as ingestion outpaces consumption. When traffic subsides or consumer capacity increases, consumers can catch up and the backlog can shrink. The shared buffer lets the gateway continue receiving traffic through a temporary mismatch in rates, within the capacity available to hold the data.

That capacity is finite. The amount of data waiting depends on both incoming volume and how long it waits for downstream processing. Larger beacon payloads or heavier traffic increase the data entering the buffer. Slower consumers extend the waiting duration. Either change can increase the space required, and both can occur during the same live event.

Headroom accommodates those departures from normal operation. It provides space for traffic spikes, periods of slower consumption and the interval before autoscaling adds capacity. Autoscaling takes time to respond to increased demand. Existing capacity must hold the accumulating data during that response interval, so scaling complements headroom rather than replacing it.

This buffer serves a different purpose from gateway batching. The gateway waits briefly to forward small beacons efficiently. Valkey holds shared, collected events while downstream processing catches up. Keeping those responsibilities distinct explains why efficient forwarding alone cannot absorb a sustained difference between ingestion and consumption.

Hand collected events to downstream services

At the downstream boundary, internal services receive event data that has been collected, grouped and buffered. The ingestion architecture has already handled device connections and brought related events into shared collection. Consumers can use that data for the processing their applications require.

Destinations can include Kafka for event distribution, ClickHouse for analytics and S3 for object storage. These services have different downstream roles, but each can begin from the organized event data available at the handoff. The ingestion boundary’s contribution is to prepare that data while accommodating the device traffic that produces it.

Make live-event traffic manageable

The gateway and shared Valkey buffer protect internal processing through cooperating responsibilities. Valkey Router handles device connections and controls ingestion. Batching shares forwarding overhead across small beacons. Deterministic grouping keys bring related events from different gateways into coherent data streams. Shared buffering gives those events room to wait through temporary differences between ingestion and consumption.

Together, these mechanisms let internal services process organized event data while the boundary handles unpredictable device traffic. Their combined effect depends on sufficient capacity for the traffic and its waiting duration. The architecture makes that relationship explicit, connecting efficient ingestion with the headroom needed to support real-time processing during live events.

If you are designing this boundary, bring your expected beacon traffic, payload sizes, grouping requirements and downstream processing needs to Momento. We can help you explore how Valkey Router and shared Valkey collection fit your workload and topology.