sqlflow

sqlflow is an open-source stream processing engine from Turbolytics that uses SQL to define pipelines and DuckDB to execute them. It reads continuously arriving events, transforms or aggregates them, and writes the results to another system. The engine runs on a single node in one container without a JVM, coordinator, or required external state store. It is aimed at streaming workloads that need a practical path from a topic, WebSocket, or webhook to a database or object store without introducing a distributed processing cluster. Turbolytics reports a published Kafka-to-sink benchmark of 1,057,360 messages per second with a 303 MiB container peak. A separate steady-stream observation kept the working set between 25 and 29 MiB for an hour.

A pipeline is configuration with SQL in it, which makes the transformation easy to keep in version control and review with the rest of an application's code. Developers can run that SQL against fixture data before the pipeline touches a live broker, then start the same configuration against production sources. This gives teams a testable workflow for work such as rolling counts from Kafka, minute windows over a public firehose, or landing a stream as Parquet or Iceberg. Prometheus metrics ship with the binary, so the operational view doesn't require a separate add-on. Keeping a pipeline in a file also gives it the usual software workflow. A team can review a change in a pull request, keep it beside application code, regenerate related declarations, and roll it back without reconstructing settings from a web console.

The current source catalog covers Kafka, WebSockets, and HTTP webhooks. Stable native sinks include Kafka, ClickHouse, Iceberg, and PostgreSQL. sqlflow can also use DuckDB's attach mechanism to write to MotherDuck, DuckLake, and S3-compatible object storage as Parquet. The connector page labels maturity directly, separating stable connectors from experimental MySQL and SQLite support and naming missing sources such as Pulsar and Redis Streams. That candor is useful when a team is checking whether the engine fits an existing stack rather than assuming every logo has equal support.

Under the hood, DuckDB executes the SQL and Apache Arrow provides a columnar in-memory format between stages. sqlflow supports stream-oriented operations such as filtering, aggregation, and tumbling windows, and can write closed windows to a sink in batches. Its Bluesky demonstration reads the Jetstream WebSocket, groups posts into one-minute windows by language, upserts the closed results into PostgreSQL, and serves analytic queries over HTTP. The page has no separate backend, and its charts query that stack directly. The complete pipeline and analytics service run in less than 256 MB of memory together in that demonstration, beside a small PostgreSQL instance, showing the kind of compact deployment the project is designed to support.

sqlflow fits data engineers, backend developers, and small platform teams that have a stream and a SQL-expressible transformation but don't need a many-worker cluster. It can run on a laptop during development, beside a collector, on an edge machine, or on a modest server close to the data. The strongest use cases are aggregation, filtering, windowing, and moving results into a durable destination. Teams whose work is mostly per-message reshaping may prefer a mapping-focused tool, while workloads that require broad event-time semantics, watermarks, or horizontal scale are better served by a distributed engine.

The project's boundaries are unusually clear. sqlflow is not a data warehouse, so results are queried in the sink rather than in the processor. It isn't a general-purpose ETL platform for transformations that don't fit SQL, and it isn't a hosted control plane. A single node also means there is no coordinator or shard rebalancing across a fleet. Those limits are part of the design, not missing marketing details. In exchange, operators get a much smaller system whose pipelines can be seeded, replayed, and asserted on like other software. This makes it easier to place beside an existing application or at the edge, though the operator remains responsible for the machine and the availability of each external source and sink.

sqlflow is MIT licensed and self-hosted. The published path to trying it is a Docker image, and the quickstart validates a configuration against fixture data before connecting it to Kafka. There is no SaaS account, subscription, or paid product tier. Turbolytics does offer consulting for architecture reviews, migrations, support, and hands-on engineering, but the engine itself remains available as open-source software. The submitted directory URL pointed to the Bluesky demonstration, while the product homepage is the better canonical destination for evaluating the actual engine.