Helicone System Design

Hi @Lakee, Jawn is an ECS service. It has a few functions atm:

• Kafka consumer
• REST API for external use
• REST API for some internal endpoints

Kafka is manage hosted on Upstash cloud. Jawn consumes from Kafka, processes the logs, inserts into Postgres + Clickhouse.

Gotcha! So I am guessing the workers push the events to kafka?

If so, are they pushed as batches or a single event at a time

just wondering since ik kafka works well for batches, but I am not so how you can batch with CF workers

Hi,

Yes, workers push events to Kafka via HTTP since Cloudflare workers don't support TCP.

Events are pushed one at a time because each request arrives individually. We can't batch on the worker side because workers are serverless, and storing them in memory isn't feasible. Pushing them to Kafka immediately is more reliable; if a worker fails, we'd lose any stored events.

Kafka handles individual logs well, and we consume them in batches. This has significantly reduced our database load and stopped it from dying.

@Lakee