PostHog Moved Its Warehouse Off DuckDB After Five Months and 2 PB of Data. Here Is Why That Is Not a Reason for You to Leave.
On November 2, PostHog published a post titled "Why we're moving our warehouse off DuckDB." Five months earlier the company had rebuilt its data warehouse on DuckDB instead of ClickHouse, with one instance per customer. Now it is moving to its own version of Trino and a new open table format it calls HogLake. If you use DuckDB for your own analytics, the headline reads like a warning. The details say something narrower.
What PostHog hit
The team says it still likes DuckDB but underestimated how large and complicated customer workloads would get. The first limit is structural: DuckDB is a single-node engine, so the worker has to be sized before each query runs. That forced very large instances and made CPU and memory needs hard to predict.
The second set of problems came from DuckLake, the table format and catalog layered on top. PostHog ran all its data in one DuckLake, about 2 PB and roughly 1 trillion rows, with 15.3 million snapshots. Single-table writes at that scale could take more than three minutes because each commit re-read statistics for the whole catalog, and over 99% of the stats rows belonged to deleted tables. Cleanup made writes 30 to 50 times faster, but the rows kept coming back. Fixes they needed moved slowly upstream, so they maintained a fork, which cut into the point of using a community project.
A third issue was the Postgres wire protocol, which was hard to scale compared to HTTP because of reconnects, idle connections and client-specific compatibility problems.
Notice the scale
Every limit above starts to bite at a size most of us do not approach. 2 PB is about two million gigabytes. A solo operator's analytics, even for a product with real traffic, is typically gigabytes, maybe a few terabytes. PostHog was running a multi-tenant warehouse where each customer's workload could be wildly different from the next, and their goal was to "work at any scale, at a fair price," in their words.
That is a product requirement you do not have. You have one dataset, one workload, and a lot of control over both.
What I would take from it
Two lessons carry down to my size. The first is that a table format with a catalog adds metadata work that grows with the number of snapshots and tables, so if you adopt one, watch commit time as the number of tables and deletions grows. The second is that being tied to a single-node engine means you size for your biggest query, and you should know what that is before it surprises you.
The part I would copy is the habit PostHog already had: an open format underneath, so that moving engines does not mean rewriting data. They were able to move because their data was not trapped in a proprietary store. If your analytics sit in Parquet files in a bucket, you can switch engines later without a migration project. That is the real insurance.
Where this could be wrong
I do not have PostHog's numbers beyond what they published, and a few of their problems, such as the snapshot count, might arrive earlier for an unusual workload at smaller scale. A product with heavy write churn and many small tables could hit catalog overhead sooner than my gigabyte intuition suggests. I am also reading one company's account of its own decision, and the post is five minutes long, so it leaves out whatever the team does not want to share.
It is also fair to say the move shows how quickly the right choice changes. Five months is not a long time for a rebuild to be reversed.
What I'd actually do
Keep using DuckDB if it is working. Store the data in an open format such as Parquet, and write down the one query that would hurt most on a bigger dataset so you know your actual ceiling. Check the commit time on any catalog layer once a quarter. Do not migrate because a company operating at a scale you will not reach did. If you ever hit their limits, you will have a good problem, and the same open files that made their move possible will make yours possible too.
Author
Lukas
@lukcombinator