Coder Legion - Recent posts tagged clickhouse

Deploying SigNoz in 2026: A Survival Guide to ClickHouse v25+ and OTel Gotchas

Mon, 28 Sep 2026 10:42:43 +0000

Hello, fellow developers! ‍

When building a modern B2B SaaS ecosystem, bulletproof observability isn't a luxury—it's a strict survival requirement. If you are handling distributed cron jobs, processing thousands of webhook deliveries, or managing isolated services, you need to know exactly what happens under the hood when things go wrong.

While architecting the infrastructure for Verne Software, I decided to take complete control of our telemetry data by self-hosting SigNoz. The official repository makes it look incredibly easy: just run a docker-compose up script and you are supposedly good to go. You think: "Great, a 10-minute setup and I can get back to writing business logic." Well, not quite.

The reality is that official documentation often lags behind major architectural shifts in underlying Docker images. What was supposed to be a quick deployment turned into a deep dive into ClickHouse configurations, OpenTelemetry bugs, and networking quirks.

In this article, I want to share my recent battle scars. We will go through the specific traps I encountered while trying to get the modern SigNoz stack running in an isolated Docker network—and exactly how to fix them.

The Target Architecture (The "Happy Path")

Before we dive into the errors, let's look at the final, working setup. The core principle here is strict network isolation and a highly deterministic boot sequence.

Network Isolation

The entire SigNoz stack lives inside an isolated Docker network (verne_observability). The SigNoz API and UI container is additionally connected to an internal network (verne_internal). This allows our core API to proxy dashboard requests without exposing the database or internal collector ports to the outside world.

The Boot Sequence

In the modern stack, you can no longer start everything at once. Order matters immensely. The services must wait for their neighbors to be completely ready:

signoz_init_clickhouse: A one-shot container that downloads essential UDF binaries (like histogramQuantile).
signoz_zookeeper: The coordinator for ClickHouse.
signoz_clickhouse: The main datastore.
signoz_telemetrystore_migrator: A crucial bootstrap step that handles schemas.
signoz_otel_collector & signoz: The final data consumers.

Here is how you enforce this dependency chain in your docker-compose.yml to prevent race conditions:

signoz_telemetrystore_migrator:
    # ...
    depends_on:
      signoz_clickhouse:
        condition: service_healthy # Wait until DB can accept migrations

  signoz_otel_collector:
    # ...
    depends_on:
      signoz_telemetrystore_migrator:
        condition: service_completed_successfully # Start ONLY after schema is created

Trap #1: The ClickHouse v25+ Configuration Shift

The Symptom: You mount your custom clickhouse-config.xml exactly as older tutorials suggest, but ClickHouse refuses to start, throwing errors about missing paths or invalid settings at the top level.

The Reality: With ClickHouse 25.5.6, the configuration paradigm has shifted. Replacing the main config.xml outright is a recipe for disaster. Furthermore, certain settings have been strictly recategorized.

For instance, you might see this exact crash loop: Code: 137. A setting 'log_queries' appeared at top level in config. But it is user-level setting that should be located in users.xml inside <profiles> section.

The Fix: Stop mounting over the default config. Instead, utilize the config.d/ and users.d/ directories for your overrides.

If you want to enable query logging, you must define it specifically inside a user profile. Create a users.xml file and mount it to /etc/clickhouse-server/users.d/users.xml:

<clickhouse>
    <profiles>
        <default>
            <log_queries>1</log_queries>
        </default>
    </profiles>
</clickhouse>

Keep your cluster topologies and UDF paths in config.d/, and your profiles and passwords in users.d/. Respecting this separation will save you hours of debugging.

Trap #2: The Invisible Migrations & Missing Databases

The Symptom: You fire up your stack, and the OpenTelemetry Collector immediately crash-loops, screaming: Database signoz_traces does not exist.

The Reality: In older iterations, you could often get away with starting the datastore and the collector simultaneously. The schemas were either auto-created on the fly or bundled differently. Today, schema creation is strictly decoupled from the application logic. If you just start ClickHouse and the OTel Collector, the necessary databases (signoz_traces, signoz_metrics, signoz_logs) simply will not exist.

The Fix: You need to embrace the one-shot container pattern. Specifically, you must introduce ephemeral containers that do the dirty work before the main services are allowed to boot.

First, as mentioned in our boot sequence, signoz_init_clickhouse must run to download essential User Defined Functions (like the histogramQuantile binary) directly into the ClickHouse user_scripts directory.

Second, and most importantly, you need the signoz_telemetrystore_migrator. This container handles the complex bootstrap process and synchronizes the schemas. Your collector should only run after this migrator exits with a success code.

# The crucial pre-flight check for the collector
  signoz_otel_collector:
    image: signoz/signoz-otel-collector:${OTELCOL_TAG:-v0.111.10}
    command:
      - -c
      - |
        /signoz-otel-collector migrate sync check &&
        /signoz-otel-collector --config=/etc/otel-collector-config.yaml
    depends_on:
      signoz_telemetrystore_migrator:
        condition: service_completed_successfully

By explicitly chaining migrate sync check in the command and waiting for the migrator container, you eliminate the race condition completely.

Trap #3: Taming the OpenTelemetry Collector

The Symptom: The Collector fails to start due to configuration parsing errors, missing image tags, or deprecated flags.

The Reality: The OpenTelemetry ecosystem moves incredibly fast, and SigNoz's custom collector is no exception. Between versions, breaking changes are silently introduced, turning perfectly good configuration files into landmines.

The Fixes: Here are the three specific collector headaches you need to watch out for:

Image Tagging Roulette: If you are pinning versions in your .env file (which you absolutely should be doing in production), pay attention to the exact naming convention. Dropping the v prefix (e.g., trying to pull 0.144.2 instead of v0.144.2) will result in an image pull failure. It sounds trivial, but it's a remarkably common stumbling block when updating the stack.
The Renamed Exporter: The telemetry pipeline configuration has evolved. If you are migrating an older otel-collector-config.yaml, you must update the metrics exporter name. The legacy clickhousemetricswrite has been deprecated and replaced. You now need to use signozclickhousemetrics to route your data correctly.
Feature Gates: Clean up your CLI commands. Deprecated flags (like -pkg.translator.prometheus.NormalizeName) that used to just trigger harmless CLI warnings will now cause fatal container crashes because those features have been stabilized and the flags removed. Keep your command array strictly to the bare minimum required to pass the config file.

Trap #4: Pragmatic Security (The Passwordless Docker Network)

The Symptom: You try to follow security best practices by adding authentication to every component, but the stack becomes a house of cards. Specifically, Bitnami’s ZooKeeper refuses to start, and the OpenTelemetry Collector fails to connect to ClickHouse despite "correct" credentials in the DSN.

The Reality: We often fight two battles here. First, modern Bitnami images (often used for ZooKeeper) are secure by default and will simply exit if no authentication is configured. Second, OpenTelemetry configuration files are notoriously finicky with environment variable interpolation in DSN strings—one misplaced character in a password can break the entire pipeline.

The Fix: Embrace network-level isolation as your primary defense. In the official SigNoz approach, and the one we adopted for Verne Software, ClickHouse runs without a password internally. Security is enforced by the fact that the verne_observability network is not accessible from the outside world.

To get the stack running smoothly, apply these two adjustments:

For ZooKeeper: Explicitly allow anonymous logins in your docker-compose.yml. This is acceptable because ZooKeeper is buried deep within your private network.

For ClickHouse: Use a clean, passwordless DSN in your OTel and SigNoz API configurations. It eliminates the "interpolation hell" and simplifies the handshake between services.

signoz_zookeeper:
    image: bitnami/zookeeper:3.9.2
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes # Required for the stack to boot without complex SASL setup
    networks:
      - verne_observability

By relying on the Docker network boundary, you reduce configuration complexity while maintaining a solid security posture for your internal telemetry.

Conclusion: The Real Cost of Self-Hosting

After navigating these four traps, my docker-compose ps finally shows a beautiful sight: six containers humming in perfect harmony. From the one-shot UDF initializer to the perfectly synchronized telemetry migrator, the stack is now a reliable engine for our observability needs.

However, the main takeaway from this journey is clear: infrastructure is a full-time job. While owning your telemetry data is empowering, maintaining the underlying "plumbing"—the databases, collectors, and ever-shifting configurations—is a massive tax on engineering time.

This experience is exactly what shapes our philosophy at Verne Software. We believe developers should spend their "innovation tokens" on building core business logic, not on taming infrastructure monsters. It is why we are building the Nautilus ecosystem—to provide powerful "as-a-Service" tools that just work.

If you’re interested in seeing how we’re applying these lessons to simplify developer workflows, we’ve just launched the beta version of our platform at vernesoft.com. You can also dive into our growing documentation at docs.vernesoft.com to see what we’re building next.

Stay pragmatic, and may your logs always be searchable!

OpenSearch, Loki, Quickwit and ClickHouse: Which Platform Should You Choose for Logs and Observability in 2026?

Thu, 23 Jul 2026 10:00:00 +0000

Log management and observability are no longer a matter of “storing text and searching it quickly.” In 2026, the real challenge is economic and architectural: ingesting growing volumes, retaining data for months, correlating logs with metrics and traces, and responding during an incident without costs spiraling out of control.

For years, Elasticsearch was the dominant option. Then came OpenSearch, Loki, Quickwit and, with significant momentum, ClickHouse as an analytical backend. All of them can solve part of the problem, but they do not optimize for the same things. The senior decision should not be “which one is faster” in the abstract, but which questions the team will ask, what retention it needs and how much complexity it can operate.

The Historical Standard: Elasticsearch and OpenSearch

OpenSearch began as an open source fork of Elasticsearch and keeps the mental model most familiar to many teams: indices, shards, replicas, mappings, Query DSL, aggregations and a mature full-text experience. If your organization already comes from Elasticsearch, it is the most direct transition.

Architecturally, it is built on Apache Lucene. Each index is divided into shards; each shard maintains segments and inverted indices; the cluster coordinates writes, replicas, distributed searches and aggregations. This provides a great deal of power, but also a clear operational footprint: JVM, heap sizing, merges, hot/warm/cold tiers and mapping tuning.

For logs, OpenSearch shines when you need expressive text search, rich filters, aggregations, mature dashboards, existing tooling or use cases close to SIEM. The cost appears as volume grows: indexing a lot of content makes it fast to search, but consumes storage and memory.

In 2026, the 3.x branch also matters. OpenSearch 3.0 introduced relevant changes such as Lucene 10 and JVM 21, along with performance improvements and vector search capabilities. That signals project vitality, but it also means upgrades need more careful planning than in a purely stateless stack.

Loki: Observability Centered on Efficiency

Grafana Loki takes a different philosophy: do not index the full content of logs, only the labels. In practice, it is closer to “Prometheus for logs” than to Elasticsearch: queries by labels and subsequent filtering over lines with LogQL.

Its storage is based on compressed chunks and a small index. Chunks usually live in object storage such as S3, GCS, Azure Blob or MinIO, while the index points to streams defined by label combinations. That decision reduces cost, but shifts responsibility to label design.

The practical rule is simple: labels must have controlled cardinality. cluster, namespace, app, environment or service usually work well. request_id, user_id or trace_id as labels can break the model because they multiply streams, hurt compaction and grow the index. For those values, it is better to filter log content or rely on traces.

Loki shines in Kubernetes because the label model fits pods, namespaces and services. Its Grafana integration is also excellent: log exploration, jumps from metrics, alerts with LogQL and correlation with Tempo or Prometheus.

Its weakness appears when you want to treat logs as an arbitrary search corpus. A query like “find this fragment across all logs from the last six months” can be much more expensive than in a system with a full-text index. Loki is designed for queries bounded by labels and time, not to replace a general-purpose search engine.

In 2026, Loki remains highly active. The official documentation lists recent 3.x versions, including the 3.7 branch, and Grafana continues to position it as a horizontally scalable, highly available and cost-efficient log aggregation system. Grafana’s acquisition of Logline further reinforces the focus on difficult searches within logs, although it does not change the central principle: label design remains critical.

When Loki Shines

Loki is the pragmatic choice for Kubernetes teams that already live in Grafana, need to retain volume at low cost and query by service, namespace, cluster, environment and time window. It is less convincing for global full-text search or deep forensic analysis.

Quickwit: Cloud-Native Search on Object Storage

Quickwit is a distributed search platform written in Rust, designed for append-only workloads such as logs and traces. Its thesis is attractive: keep powerful search, but decouple compute and storage by using object storage as the main layer.

Instead of depending on hot local disks, Quickwit writes indices and splits to S3-compatible storage, Azure Blob, Google Cloud Storage or MinIO. Indexers process data, searchers query segments and the metastore coordinates state. The model uses inverted indices optimized for immutable data and remote storage: closer to OpenSearch in search than Loki, but with cloud-native economics.

Quickwit is especially interesting for teams that want to replace part of an expensive Elasticsearch/OpenSearch cluster, already have logs in S3 or can move them there, and need text search with a better operational cost profile. It also has Grafana integration and APIs that are partially compatible with Elasticsearch/OpenSearch, which helps with some existing workflows.

The critical nuance for 2026 is its project status. Datadog acquired Quickwit in 2024. The acquisition did not mean the open source repository disappeared: the quickwit-oss/quickwit repository remains public and shows recent activity, including 2026 tags and discussions. The Quickwit Grafana plugin also publishes recent compatibility with Grafana 12.1+ and 13. That said, the product risk is different from Loki or OpenSearch: strategic direction is influenced by Datadog, and part of the effort may be oriented toward internal or commercial use cases.

The honest reading is this: Quickwit should not be dismissed as “dead,” because according to the reviewed sources it is not archived. But it should be evaluated with more diligence: release cadence, critical issues, roadmap, real API compatibility, operational ease and dependency on the maintainer team. For a core observability platform, that evaluation matters as much as the benchmark.

ClickHouse: The Analytical Engine of New Observability

ClickHouse was born as a columnar analytical database, not as a log platform. That is exactly why it has become relevant to observability. Many incidents are not solved by searching for an exact string, but by grouping by service, calculating percentiles, exploring cardinalities, joining logs with traces and querying billions of events over a time range.

ClickHouse’s model is columnar. Data is stored by columns, compresses extremely well and is queried with vectorized execution, ordered primary indices, data skipping indexes and engines such as MergeTree. For observability, this makes it possible to store wide events and read only the columns needed.

The tradeoff is that ClickHouse is not Elasticsearch. Full-text search exists and has improved, but its strength remains structured and semi-structured analysis. If your queries are “give me all 500 errors by version, region and endpoint in the last 15 minutes,” ClickHouse is a very good fit. If your main query is “find any line containing a rare phrase across six months of unstructured text,” you probably want another component or a hybrid strategy.

ClickHouse has moved decisively toward observability. The acquisition of HyperDX and the launch of ClickStack change its positioning: it is no longer only “use ClickHouse as the backend and build the UI yourself,” but a stack with OpenTelemetry, HyperDX as the interface and ClickHouse as the engine. It shines in high-volume platforms, structured logs, traces, derived metrics and low-latency aggregate queries, although it requires understanding modeling, TTLs, compression and ingestion costs.

Quick Comparison

Characteristic	OpenSearch	Loki	Quickwit	ClickHouse
Main model	Distributed search on Lucene	Logs by streams and labels	Cloud-native distributed search	Columnar analytical database
Storage model	Shards, Lucene segments, hot/warm/cold tiers	Compressed chunks + small index	Indices/splits in object storage	Compressed columns in MergeTree
Index	Inverted, very complete	Labels, not full content	Inverted, optimized for object storage	Primary index, data skipping, columns
Full-text search	Excellent	Limited and dependent on filters	Very good	Good, not its main strength
Relative cost	High	Low	Low-medium	Low-medium
Query latency	Low on well-sized hot indices	Low if bounded by labels; worse on broad scans	Low-medium depending on cache and object storage	Very low on well-modeled aggregations
Scaling	Mature, but operationally involved	Horizontal and economical	Decouples compute and storage	Excellent for analytics and ingestion
Kubernetes	Good	Excellent	Good	Good, better with ClickStack/OTel
Grafana	Yes	Native	Yes	Yes
Elasticsearch compatibility	High in OpenSearch	No	Partial	No
Best use case	Complex search and compatibility	Low-cost Kubernetes logs	Efficient search on S3	Unified analytical observability
Main risk	Cost and complexity	Label cardinality	Ecosystem and governance after acquisition	Modeling and SQL/operations learning curve

Which Should You Choose in 2026?

The answer depends less on technology fashion and more on the team’s profile.

If You Are a Platform Team with Elasticsearch Heritage

OpenSearch is the conservative option if you already have automation, alerts and operational knowledge around Elasticsearch/OpenSearch. Do not migrate “as is” expecting costs to drop: review mappings, retention, ILM/ISM, shards, indexed fields and the separation between hot data and archive.

If You Are a Kubernetes Team Centered on Grafana

Loki is usually the first option. It is efficient, integrates very well with Grafana and reduces the cost of operational logs. The key is to govern labels from day one: allowed taxonomy, cardinality limits and context-driven queries before text searches.

If You Want to Reduce Cost Versus OpenSearch Without Losing Search

Quickwit deserves a serious PoC when Loki falls short on search and OpenSearch is expensive. The decision must include continuity: repository activity, releases, clients, community support, Grafana integration and Datadog’s position.

If You Want a Unified Observability Platform

ClickHouse is probably the strongest bet for consolidating logs, traces, derived metrics and events in a shared analytical database. ClickStack and HyperDX reduce the need to build the entire experience from scratch, but adoption depends on maturity in SQL, modeling and analytical database operations.

If You Have Security, Audit or SIEM Requirements

OpenSearch remains more natural for textual investigation and security tooling. ClickHouse can work for security analytics at scale, but it requires modeling. Loki is rarely the first choice for SIEM. Quickwit can be interesting, with a smaller ecosystem.

If Budget Is the Bottleneck

Do not choose only by storage cost per TB. Calculate total cost: ingestion, queries, retention, operations, backups, upgrades, training and incident time. Loki and ClickHouse usually win on raw cost; Quickwit can be highly competitive on object storage; OpenSearch can be reasonable if it is limited to data that is truly searchable and hot.

How to Frame a Decision PoC

A useful test is not installing four Helm charts and looking at a demo. Define real questions from past incidents and run them against representative data:

The last 7 days of production logs, with real cardinality.
A hot 2-hour window with high volume.
A rare text search over a broad period.
An aggregation by service, endpoint, version and region.
A correlation flow from alert to logs and traces.

Measure sustained ingestion, storage cost, p50/p95 latency, operational complexity and team experience. Include an uncomfortable scenario: a failed node, slow object storage, unexpected cardinality or a badly written query.

Conclusion

The era when Elasticsearch was the only viable option for logs is over. In 2026, there are four clear paths.

OpenSearch is the mature choice for full-text search and compatibility. Loki is efficient for operational logs in Kubernetes. Quickwit offers modern search on object storage, with mandatory governance evaluation after the Datadog acquisition. ClickHouse is consolidating as an analytical engine for unified observability, especially with ClickStack and HyperDX.

The best platform is not the one that wins every benchmark. It is the one that answers your production questions with the lowest sustainable cost and the least operational friction for your team.

CTA: Decide with Data, Not Preferences

Before migrating or standardizing, run a two-week PoC with real data, real queries and an honest estimate of total cost. It should reveal which data belongs in search, which belongs in analytics and which belongs in cheap storage.

Sources

OpenSearch 3.0: https://opensearch.org/blog/unveiling-opensearch-3-0/
OpenSearch 3.0, Lucene 10 and JVM 21: https://opensearch.org/blog/opensearch-3-0-what-to-expect/
Grafana Loki, documentation: https://grafana.com/docs/loki/latest/
Grafana Loki, releases 3.x: https://grafana.com/docs/loki/latest/release-notes/
Grafana Loki, labels and cardinality: https://grafana.com/blog/how-labels-in-loki-can-make-log-queries-faster-and-easier/
Datadog acquires Quickwit: https://www.datadoghq.com/blog/datadog-acquires-quickwit/
Quickwit OSS: https://github.com/quickwit-oss/quickwit
Quickwit datasource for Grafana: https://github.com/quickwit-oss/quickwit-datasource
ClickStack: https://clickhouse.com/clickstack
ClickStack/HyperDX in ClickHouse Cloud: https://clickhouse.com/docs/use-cases/observability/clickstack/overview
ClickHouse acquires HyperDX: https://clickhouse.com/blog/202504-newsletter
ClickHouse and observability 2026: https://clickhouse.com/resources/engineering/what-is-observability

ClickHouse system tables ate my disk (and the fix)

Thu, 09 Jul 2026 15:14:44 +0000

TL;DR.

A monitoring alert said I was dropping check results. The disk was 100% full. My actual data was 20 MB. ClickHouse had quietly written 12 GB of logs about itself, mostly system.text_log and system.trace_log. Those same logs also burn CPU at idle. The fix is a few lines of ClickHouse config that disable the noisy logs and slow the metrics collector. Full config is below.

The alert

One morning a critical alert fired: UptimepageResultsLost. Its description is blunt: storage write failures or dropped results, checks run but results are not persisting. Then it did something worse than fire once. It cleared, fired again, cleared, and fired again, over and over.

I run Uptimepage, an uptime monitoring service. "Results not persisting" means the one thing customers pay for, recording whether their sites are up, might be failing. So it had my full attention.

The good news first: no data was lost. The write path retries, and every failed write was caught by a retry. I keep a counter for results that actually get dropped, and it stayed at zero the whole time. But something was clearly wrong underneath.

The real signal: a full disk

The first real signal came from Postgres, which had crashed and restarted a couple of minutes earlier:

FATAL: could not write lock file "postmaster.pid": No space left on device

The disk was 100% full. Postgres could not write its lock file, crashed, and recovered on its own through WAL replay. ClickHouse, where I store raw check results, was rejecting inserts with its own version of the same complaint:

Code: 243. DB::Exception: Cannot reserve 1.00 MiB, not enough space:
While executing WaitForAsyncInsert. (NOT_ENOUGH_SPACE)

So the alert was a symptom. The real problem was a full disk. That reframes the question: I do not store much, so what filled it?

Leak one: old Docker images

docker system df gave the first half of the answer:

88 Docker images, only two of them in use. Every deploy pulls a fresh image, and nothing pruned the old ones, so they piled up for weeks. A docker image prune -af reclaimed about 10 GB and took the disk off the ceiling.

That stopped the bleeding. But 13.8 GB of volumes is a lot for a service whose data I thought was tiny. That number turned out to be the real story.

Leak two: ClickHouse logging about itself

I went into ClickHouse and asked the obvious question, which table is big:

SELECT database, formatReadableSize(sum(bytes_on_disk)) AS size, sum(rows) AS rows
FROM system.parts
WHERE active
GROUP BY database
ORDER BY sum(bytes_on_disk) DESC

The answer stopped me:

Database	Size	Rows
`system`	11.8 GiB	577,340,927
`monitor`	19.7 MiB	1,813,884

monitor is my data: every check result and rollup I keep. About 20 MB. The system database, ClickHouse's own diagnostic tables, was 11.8 GiB. Nearly all of the storage was ClickHouse logging about itself.

Breaking system down by table:

Two tables did most of the damage. system.text_log (5.29 GiB) is a copy of the server's own log output written into a table. system.trace_log (3.02 GiB) is the query profiler, which samples running queries. Both are handy when you are actively debugging ClickHouse. Neither is worth multiple gigabytes when I am not. And text_log is off by default, so something in my setup had switched it on.

The fix

Two parts: reclaim the space now, and stop it coming back.

Reclaim now. The system log tables are throwaway diagnostics, not real data. Truncate them:

TRUNCATE TABLE IF EXISTS system.text_log;
TRUNCATE TABLE IF EXISTS system.trace_log;
-- and the rest of the system.*_log tables

That gave back 12 GB at once and took the disk from 74% down to 41%.

Stop the regrowth. Truncating is a one-time cleanup. Without a cap, the logs fill right back up. The durable fix is ClickHouse config, added under /etc/clickhouse-server/config.d/. I watch ClickHouse through Grafana, not these tables. So I disable almost all of them and keep only query_log and part_log, both bounded, for the rare hands-on debugging session:

<clickhouse>
    <!-- Sample async metrics every 60s, not every second. -->
    <asynchronous_metrics_update_period_s>60</asynchronous_metrics_update_period_s>

    <!-- Disable the log tables. remove="1" on an absent table is a no-op,
         so this list is safe to paste as-is across ClickHouse versions. -->
    <asynchronous_metric_log remove="1"/>
    <asynchronous_insert_log remove="1"/>
    <backup_log remove="1"/>
    <error_log remove="1"/>
    <crash_log remove="1"/>
    <metric_log remove="1"/>
    <query_metric_log remove="1"/>
    <query_thread_log remove="1"/>
    <query_views_log remove="1"/>
    <session_log remove="1"/>
    <text_log remove="1"/>
    <trace_log remove="1"/>
    <opentelemetry_span_log remove="1"/>
    <zookeeper_log remove="1"/>
    <processors_profile_log remove="1"/>
    <latency_log remove="1"/>
    <background_schedule_pool_log remove="1"/>

    <!-- Keep query_log and part_log, bounded, for on-hand debugging. -->
    <query_log>
        <ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
        <max_size_rows>1048576</max_size_rows>
    </query_log>
    <part_log>
        <ttl>event_date + INTERVAL 3 DAY DELETE</ttl>
        <max_size_rows>1048576</max_size_rows>
    </part_log>
</clickhouse>

remove="1" disables a system log completely. The <ttl> element on query_log and part_log adds a TTL and keeps the table's default partitioning, so you do not have to restate the whole engine. ClickHouse picks up the change after a restart. Altinity's "system tables ate my disk" note covers the same ground and is worth a read. If you would rather keep the diagnostics, give every log a short <ttl> instead of disabling it.

Bonus: lower CPU, not just disk

Those log tables are written constantly, and on a small server that steady write load shows up as CPU. A long-running ClickHouse issue tracks the server using noticeable CPU at zero load, #60016. People there report dropping from 40 to 70% CPU down to about 1.5% after disabling the logs and slowing the async-metrics collector. So those two settings, the asynchronous_metrics_update_period_s line and the remove="1" block, pay off twice: less disk and less CPU.

The real lesson: alert on the cause, not the effect

Here is the part that stings. I had an alert for "results are being dropped." I did not have an alert for "the disk is filling up." So a full disk is a slow, predictable problem that builds over days. But it only reached me as a sudden downstream symptom, after Postgres had already crashed once.

A downstream alert like "results lost" is not a substitute for watching the resource that actually runs out. I added the missing one: a plain host disk-space alert on node_filesystem_avail_bytes, firing at 80% and 90% used, well before anything starts failing. That is the alert that should have caught this on day one.

Key takeaways

ClickHouse system log tables are unbounded by default and can dwarf your real data. Mine were 11.8 GB against 20 MB of actual data.

system.text_log and system.trace_log are the usual offenders. text_log is off by default, so check whether something enabled it.

Cap them in config: remove="1" to disable a log, or <ttl> plus <max_size_rows> to bound the ones you keep. Truncate to reclaim space right away.

It is not just disk. The same logs burn CPU at idle on small servers. Disabling them, plus asynchronous_metrics_update_period_s = 60, took reporters in ClickHouse issue #60016 from 40 to 70% CPU down to about 1.5%.

Anything that pulls artifacts on a schedule, Docker images in my case, needs matching cleanup or it becomes a slow disk leak.

Alert on the cause (disk space), not only the effect (dropped writes). The cause gives you days of warning; the effect gives you minutes.

This happened on Uptimepage, the uptime monitor I run and dogfood.

Postgres vs ClickHouse? I use both. 4 tricks from the split.

Thu, 09 Jul 2026 05:57:37 +0000

TL;DR. My uptime monitor keeps config and incidents in Postgres and check results in ClickHouse. The split is one rule: does a row ever change? Config gets edited, so it wants Postgres transactions and constraints. A check result is written once and never touched again, so it goes to ClickHouse, where the right codec, a careful sort key, and a per-row TTL make billions of rows cheap. Four tricks and a bonus below, useful even if you only ever run one database.

Every monitor I run writes a row every 20 seconds, from every region, and never stops. One monitor is about 4,300 rows a day, per region. Multiply by every monitor on the platform and the rows only ever go up.

That stream would slowly crush a normal Postgres table, and watching it is the whole product. So the check results do not live in Postgres. They live in ClickHouse. Everything else, the monitors and incidents and teams, lives in Postgres. The interesting part is the line between them.

People ask why not one database. "Postgres vs ClickHouse" is the wrong question, because the two are not fighting over the same job. Here is the line I draw, and the one rule under all of it: split your data by how it is written, not by which engine is faster.

1. Does the row change? That picks the database

Forget benchmarks for a second. One question sorts a table into one store or the other: after you write a row, will you ever change it?

one monitor, two stores:

  you edit a monitor  ->  Postgres   (targets, incidents, team)
                          the row changes, has constraints, lives in a transaction

  a probe checks it   ->  ClickHouse (check_results)
                          one row, written once, never updated

A monitor is a row that changes. You toggle it off, edit the URL, change the interval. So it lives in Postgres:

CREATE TABLE targets (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name          TEXT NOT NULL,
    check_spec    JSONB NOT NULL,
    interval_secs INTEGER NOT NULL CHECK (interval_secs >= 10),
    enabled       BOOLEAN NOT NULL DEFAULT true,
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

That CHECK means a 3-second interval can never reach the table, no matter which part of the app tried to write it. This is Postgres doing the thing it is best at: a small set of rows that must stay correct while many callers change them at once.

A check result is the opposite. It is written once when a probe finishes, and then it never changes. Nothing ever updates it, so it does not need any of that:

CREATE TABLE check_results (
    org_id      UUID,
    target_id   UUID,
    region      LowCardinality(String),
    timestamp   DateTime('UTC') CODEC(DoubleDelta, ZSTD(1)),
    status      Enum8('up' = 1, 'down' = 2, 'degraded' = 3, 'error' = 4),
    duration_ms UInt32 CODEC(T64, ZSTD(1))
) ENGINE = MergeTree

No updated_at, no foreign key, no transaction. Just an append-only stream that grows forever. That is the shape ClickHouse is built for and the shape that slowly hurts Postgres.

Notice the types too. status is an Enum8, one byte on disk, not the string "up". region is LowCardinality, stored once in a dictionary and referenced by a small id instead of repeating the text on every row. On a table that only ever grows, a byte saved per row is a byte saved times billions.

2. The right codec turns a day of timestamps into almost nothing

Once the results are in a column store, the type on each column is most of the compression, and the default setting wastes a lot of space.

Look at that timestamp again. A monitor checks every 20 seconds, so for one monitor a day of timestamps is a run of 20, 20, 20, 20, over and over. DoubleDelta stores the change in the change. For a steady interval the first delta is a constant 20 and the second delta is zero, so each row after the first packs down to about a bit. The codec is not a ClickHouse invention: it comes from Facebook's Gorilla time-series paper, built for exactly this, measurements taken at a steady rate.

timestamp   DateTime('UTC')  CODEC(DoubleDelta, ZSTD(1)),
duration_ms UInt32           CODEC(T64, ZSTD(1)),
dns_ms      Nullable(UInt16) CODEC(T64, ZSTD(1)),

I also store the timestamp at whole seconds on purpose. The smallest interval is 20 seconds, so no two checks for one monitor ever land in the same second, and the sub-second detail lives in duration_ms where it belongs. Whole seconds that step by a fixed amount compress far harder than milliseconds would.

The latency columns get T64 instead, because they are small integers that stay in a small range, a different shape from a steady clock. T64 crops the unused high bits off a block of values, so a UInt32 that never climbs past a few thousand milliseconds stops paying for all 32 bits. That is the trick worth copying, and it is not the specific codec names. It is that a timestamp ticking by a fixed step and a latency staying in a small range are two different shapes, and telling the database which is which does the work.

3. Put the tenant in the sort key, not the partition

I got this one wrong first, so learn it from my mistake instead of your own.

This is a multi-tenant app, so every row carries an org_id and almost every query filters by it. My first schema partitioned by org, one slot per customer, because it felt tidy. ClickHouse turned that into a flood of tiny parts, the merges could not keep up, and startup got slower the more customers I had. I ripped it out. A partition is not a folder for tidiness. It is a physical unit ClickHouse merges and expires, and you want few large ones, not many small ones.

Here is what it should be:

) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)

org_id leads the ORDER BY, so a per-org query walks the sorted primary index and reads only that org's slice. But it stays out of PARTITION BY. The rule I follow now: partition by something low-cardinality that you also delete by, here the day, and put the high-cardinality tenant key in the sort order. Sort key answers "find this org fast". Partition answers "drop old data cheaply", which is the next trick.

4. Make retention a number in a row, not a schema change

Different plans keep history for different lengths of time. The clumsy way is a migration or a cleanup job per plan. The clean way is to make the retention window a column:

ttl_days UInt16 DEFAULT 30 CODEC(ZSTD(1))
) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (org_id, target_id, region, timestamp)
TTL timestamp + toIntervalDay(ttl_days)

Each row carries its own ttl_days, stamped from the org's plan at the moment it is written. A free plan keeps 30 days, a paid plan keeps more, and changing that needs no ALTER, no migration, no backfill. The next write just stores a different number. The daily partitions line up with the TTL, so ClickHouse expires old data by dropping whole parts, close to free, and a whole column of the same 30 compresses away to nothing. The flexibility costs no space.

One more piece makes reading that history cheap. A materialized view rolls raw checks into per-minute and per-hour summaries as they land:

CREATE MATERIALIZED VIEW check_results_1m
ENGINE = AggregatingMergeTree
ORDER BY (org_id, target_id, region, minute)
AS SELECT
    org_id, target_id, region,
    toStartOfMinute(timestamp)  AS minute,
    countState()                AS total_checks,
    countIfState(status = 'up') AS up_checks,
    avgState(duration_ms)       AS avg_duration_ms
FROM check_results
GROUP BY org_id, target_id, region, minute;

So a dashboard showing 30 days across a thousand monitors reads a few thousand minute-buckets, not millions of raw rows. Recent views read the minute rollup, long history reads an hour rollup, and the raw rows underneath expire on their own TTL. The firehose is there when you need to drill into one bad minute, and left alone the rest of the time.

One more, because everyone hits it: make inserts idempotent

Here is the trick I wish someone had told me first. My agents batch check results and send them to ClickHouse. A network blip after the server commits but before my side gets the ack means the batch retries and sends the exact same block again. Without protection, that double-counts every row in it, and your uptime numbers quietly drift.

ClickHouse has a fix built in, but for a plain (non-Replicated) MergeTree it is off until you turn it on:

SETTINGS index_granularity = 8192,
         non_replicated_deduplication_window = 1000;

The server hashes each inserted block and remembers the last 1,000 hashes. A retry sends an identical block, the hash matches, and the server drops it instead of appending it again. The retry becomes safe to do blindly, so the client stays simple: send, and if unsure, send again. Make the window bigger than the most blocks a single retry could resend, and you stop having to worry about duplicate writes.

So, one database or two?

For almost every app, one. If your data fits in Postgres and stays fast, a second database is a cost you should not pay: two schemas, two clients, two things to back up and think about.

Reach for the second store only when one table stops looking like the rest of your tables. For me that table was check_results. It is written once and never edited, it grows without end, and every question I ask it is a summary over a time range. That is a different shape from my monitors and my incidents, so it wanted a different database.

The rule I would give my past self: do not split by "which database is faster". Split by how the data is written. Rows that change and must stay correct want Postgres. An append-only stream you only ever summarize wants a column store like ClickHouse. Most of the tricks above are that one idea pushed down into the schema.

Uptimepage is open source, AGPL-3.0, and both schemas are in the repo: github.com/uptimepage/uptimepage. The probe that writes those rows is its own post, and the wider build story, one binary and two databases, is here. Or start free on the hosted tier and point a check at something.

php_clickhouse 0.8.1: Three Releases Later, Stable

Sat, 09 May 2026 17:17:45 +0000

The launch post for php_clickhouse 0.6.0 covered the framing: native binary protocol, soft fork of the stalled SeasClick, modern ClickHouse types, 30-40% faster than HTTP at high throughput. That post landed April 25, 2026. Today (May 1, 2026) the current tag is 0.8.1, and I'm calling the extension stable.

The six days in between were a focused quality cycle, not a feature sprint. Three buckets:

Performance. Insert and write paths build native ClickHouse columns one at a time directly from row-major input. Peak intermediate PHP memory dropped from N_rows × N_cols zvals to one column.
Security. Strict full-consumption parsers across Map, narrow-int, Int128 / UInt128, geo, DateTime64, Time64, hex literals, and typed parameters. Wrong-type input throws instead of corrupting memory or coercing silently to zero. Recursive type-conversion gained a depth cap so adversarial server schemas can't blow the stack.
Stability. Per-Client state moved from file-scope std::map banks onto the zend_object itself. Unblocks ZTS, plugs leaks on bailout, fixes a refcount bug on the progress callback. Insert path recovers the native handle on every server-side rejection point so a thrown insert no longer wedges the connection.

Three releases (0.7.0, 0.8.0, 0.8.1) closed the API gap with the most-used HTTP client, refactored the extension's state model, hardened the insert surface, and surfaced one upstream UB fix that has since merged into clickhouse-cpp.

Here's the work.

0.7.0: Closing the Ergonomics Gap with smi2/phpClickHouse

The native binary protocol gives you 30-40% throughput. Most teams won't trade a familiar API for that, so the native client has to match the ergonomic surface of the most-used PHP HTTP client (smi2/phpClickHouse). 0.7.0 is the release that actually does that.

What landed:

setSettings(array) for client-wide ClickHouse settings (max_execution_time, max_memory_usage, async_insert). Per-call settings as a 5th array argument on select() / insert() / execute() / writeStart(). Per-call overrides global.
Server-side typed parameters via the {name:Type} placeholder syntax. Routed through Query::SetParam so the server quotes and parses according to the declared type. Plain {name} placeholders keep their existing client-side identifier-substitution behavior. Arrays format as ClickHouse array literals so Array(UInt32), Array(String) round-trip cleanly.
setProgressCallback(?callable) invoked for every Progress packet during a query (rows, bytes, total_rows, written_rows, written_bytes).
getStatistics() returning rows_read, bytes_read, total_rows, written_rows, written_bytes, blocks, rows_before_limit, applied_limit, elapsed_ms from the last completed query. Reset at the start of each query.
Structured ClickHouseException: server_code (e.g. 159 for TIMEOUT_EXCEEDED), server_name (DB::Exception), query_id. Populated on server errors and on any throw with a query-id context.
insertAssoc(table, rows) derives the column list from the keys of the first row.
SQL helpers: databaseSize(), tablesSize(), partitions(), showTables(), showCreateTable(), getServerUptime(). Each validates identifiers against the safe-character set.
Sub-second timeouts via connect_timeout_ms, receive_timeout_ms, send_timeout_ms config keys. Override the existing seconds-based keys when present.
Per-client query log accumulator: enableLogQueries(bool) toggles, getLogQueries() returns and clears. Each entry carries sql, query_id, elapsed_ms, rows_read, bytes_read, error_code, error_message.

The other under-the-hood change in 0.7.0 was migrating to a stub-driven arginfo workflow (clickhouse.stub.php → generated clickhouse_arginfo.h). Method parameter and return types are now declared at the engine boundary and visible to Reflection, IDEs, and static analyzers. Behavior is unchanged for correctly-typed callers; wrong-type callers now hit ZPP at the boundary instead of a custom thrown exception inside the method body.

None of 0.7.0 is novel on its own. The point is that without these the native client made you pay an ergonomics tax to get the speed. 0.7.0 settles that tab.

0.8.0: Per-Object State, ZTS, and Streaming

The 0.6.0 / 0.7.0 surface stored per-Client state in seven file-scope std::map<int, ...> banks keyed on Z_OBJ_HANDLE: the Client*, the in-flight insert Block, the ClientStats, the global settings, the progress and profile callbacks, the log toggle, the query log buffer.

That works, and it has three durability problems baked in:

No ZTS support. Threaded SAPIs share that file-scope state across threads. The 0.6.0 code gated MINIT with a hard error when --enable-zts was on. ClickHouse from RoadRunner / FrankenPHP / Swoole / php-pm was a non-starter.
Leaks on bailout. PHP's userspace __destruct doesn't run on fatal errors, so the map entries (and the underlying Client* and any half-open insert stream) leaked.
Refcount bug on the progress callback. A struct copy of the registered callable went stale when the calling scope went out of scope, and the next progress packet hit a freed zval.

0.8.0 moved the per-Client state onto the zend_object itself via custom create_object / free_obj handlers. The seven file-scope maps disappear entirely. ZTS gating at MINIT was deleted in the same release.

The refactor unblocks three things at once:

Threaded SAPIs. No global state to thread-isolate, so ZTS Linux is a first-class target now. CI grew a linux-zts job (PHP 8.4 ZTS built from source).
Cleanup on bailout. free_obj runs unconditionally, including on fatal errors. The Client* and any half-open insert stream get torn down properly.
The progress-callback fix lands. setProgressCallback now uses ZVAL_COPY instead of a struct copy, so the callable doesn't get freed out from under the next packet.

A Windows config.w32 shipped in the same release, rewritten from a 9-line warning stub to a full Windows build script that mirrors config.m4's source list and flags. Optional --enable-clickhouse-openssl plumbing is mirrored via CHECK_LIB("libssl.lib", ...). CI exercises Windows as a build + extension-load smoke test (no live ClickHouse on Windows yet).

Streaming reads

0.8.0 introduced two new read paths for result sets that don't fit comfortably in a single PHP array:

$it = $ch->selectStream("SELECT id, payload FROM events WHERE day = today()");
foreach ($it as $row) {
    process($row);
}

selectStream() returns a ClickHouseRowIterator (Iterator + Countable) that walks blocks lazily. The iterator survives unset($client) because blocks own their column data via shared_ptr.

For unbounded streams where you don't want to count or rewind:

$ch->selectStreamCallback(
    "SELECT id, body FROM events_unbounded",
    fn(array $row) => writeToS3($row),
);

The callback fires once per row as blocks arrive, never accumulating the full result.

The plain select() path is unchanged and remains the faster choice when you actually want a full PHP array. The streaming variants exist for the row-millions case where you don't.

Geo, LowCardinality(Nullable), and the Map matrix

The type surface expanded too:

Geo types Point, Ring, Polygon, MultiPolygon round-trip via ColumnGeo. Point as [Float64, Float64], the others as nested arrays.
LowCardinality(Nullable(String)) and LowCardinality(Nullable(FixedString)) round-trip on read and write.
The insert path now accepts any Map(K, V) over scalar K and V (String, all signed/unsigned integer widths, Float32/64, UUID) plus LowCardinality(String) keys and values. The read path mirrors the same matrix except for LowCardinality keys (vendor gap). Previously only five hardcoded combinations worked.
SimpleAggregateFunction(f, T) reads transparently as T.

Geo support unblocks one of the two large reasons people stayed on the HTTP client. The other was streaming.

Other 0.8.0 surfaces worth naming

selectStatement() returns a ClickHouseStatement result wrapper: Iterator, Countable, ArrayAccess, JsonSerializable, plus fetchOne() / fetchKeyPair() / fetchColumn() / toArray() / statistics(). Read-only (offsetSet / offsetUnset throw). Carries a per-call stats snapshot so it survives the client running other queries afterwards.
setVerbose(bool|callable) for protocol-level lifecycle tracing. Pass true for JSON lines on STDERR, or a callable invoked with ($eventName, $context). Events: select_start, data_block, select_finish, execute_start, execute_finish, server_exception. No-op when off, so the hot path stays cheap on production.
DDL helpers: isExists(), showDatabases(), showProcesslist(), getServerVersion(), tableSize(), truncateTable(), dropPartition(). All identifier args validated; dropPartition SQL-escapes the partition value.
Client introspection: resetConnection(), getServerInfo() (name, version, revision, timezone, display_name), getCurrentEndpoint() (host/port of the active endpoint when an endpoints[] pool is in use), setProfileCallback(), ping_before_query config key.
query_id echoed through getStatistics() so callers can correlate a stats snapshot to a server-side query in system.query_log.
smi2-style sugar: setSettings() returns $this for chaining, setSetting(key, value) for the single-key form, setDatabase(string) issues USE and updates the cached default used by databaseSize() / showTables(), getter aliases (getServerCode(), getServerName(), getQueryId()) on ClickHouseException.

IPv4 / IPv6 crash, fixed

This one's worth calling out as a bug-of-the-release. clickhouse-cpp v2.6.1 made ColumnIPv4 / ColumnIPv6 siblings of (not subclasses of) ColumnUInt32 / ColumnFixedString. The 0.6.0 / 0.7.0 read paths were doing As<ColumnUInt32>() / As<ColumnFixedString>() on IP columns, which now returned null instead of dispatching. The next dereference segfaulted the worker.

Fixed by switching to ColumnIPv*::AsString(row) for canonical dotted-quad / ::1 form. If you hit a crash on IP column reads pre-0.8.0, this is why.

Distribution: pre-built binaries via PIE

Binaries for Linux glibc (x86_64 + arm64) and macOS (x86_64 + arm64) are now available. On a supported platform the install collapses to one line:

pie install iliaal/php_clickhouse

No vendored clickhouse-cpp build, no abseil compile, no five-minute make. TLS still requires the source build (pie install iliaal/php_clickhouse --enable-clickhouse-openssl), but that's a smaller set of users.

0.8.1: The Insert Path That Recovers

0.8.0 was the architecture release. 0.8.1 was the hardening pass: nine rounds of reviewer-driven fixes, mostly on the insert and write surface plus the type-conversion boundary.

The headline bug:

ClickHouseException: cannot execute query while inserting

If a server-side insert rejection (missing table, bad column, CHECK constraint, schema drift) threw out of BeginInsert / SendInsertBlock / EndInsert, the vendored client's inserting_ flag stayed set. Subsequent select / execute on the same handle threw the message above until the caller manually called resetConnection().

0.8.1 wraps every server-side rejection point in a connection-reset-then-rethrow. Same handle stays usable.

Destructor cleanup mirrors the same dirty/clean recovery split: an in-flight streaming insert with sent blocks is dropped via ResetConnection on unset() rather than committed via EndInsert. Clean sessions still EndInsert. Avoids partial commits on script bailout.

Memory: column-at-a-time insert

Pre-0.8.1, insert() and write() materialized a full column-major PHP zval matrix from the user's row-major input before building the native ClickHouse columns. For a 1M-row × 30-column insert that's 30M zvals sitting in PHP memory while the column build runs.

0.8.1 builds native columns one at a time directly from the row-major input. Peak intermediate PHP memory drops from N_rows × N_cols to one column.

insertAssoc() benefited from the same change: no more positional copy of input rows. The column gatherer reads each column directly from the original associative rows, and key validation uses zend_hash_exists against the first row's HashTable instead of allocating a new std::string for every row key.

Strict parsers across the type surface

Map, narrow-int (Int8 / Int16 / Int32 / their unsigned siblings), Int128 / UInt128, geo, DateTime64, Time64 insert paths now use full-consumption strict parsers. Non-numeric strings, fractional doubles, non-finite floats, and out-of-range values throw instead of silently coercing to 0 / 0.0 inside the column.

UInt64 inserts gained a shared strict_zval_u64 parser that accepts decimal and hex strings above ZEND_LONG_MAX on both the scalar and Map(*, UInt64) paths. Reads continue to surface upper-half values as decimal strings.

The class of bug strict parsing eliminates is the worst kind of insert bug: the string "foo" lands in an Int32 column as 0, no error, no audit trail. Now it throws.

Validation and reentry

A few smaller fixes worth naming:

write() rejects rows narrower or wider than the writeStart column count. The previous path took the first row's element count as authoritative, so [1] against writeStart(t, ['a','b']) landed 1 into column a with b defaulted server-side.
insert() rejects rows with extra positional or named cells. A row like [1, 99] against a single-column table previously landed as 1 with 99 lost.
A failed later write() no longer commits previously sent blocks. The catch path tracks whether any block has been sent in the current writeStart() session and chooses ResetConnection (discard) over EndInsert (commit) on a dirty session.
insertAssoc() rejects integer-keyed later rows and any key-set drift from the first row. The first row defines the column set; every later row must match.
Enum8 / Enum16 inserts reject undeclared integers, NULL on non-Nullable columns, and unknown string names.
Single-token placeholder validator: {name} placeholders accept exactly one identifier and reject comma-separated lists. Comma-list callers must use array form.
Same-client reentry guard: a userland progress / profile callback that fires another query on the same handle now throws cleanly instead of crashing the worker on the next ReceiveData.
Recursive type-conversion depth cap (32) keeps deeply nested structures (Array(Array(...)), Map(K, Tuple(...))) from blowing the stack.

23 new PHPTs (072–094) pin all of the above.

Upstream: One Fix Merged Back to clickhouse-cpp

The ASan job added in 0.8.0 caught a latent UB in the vendored library that nobody had been hitting in production, but UBSan flagged on every empty LowCardinality(String) value:

runtime error: null pointer passed as argument 2,
  which is declared to never be null

ColumnStringBlock::AppendUnsafe was calling memcpy(pos, str.data(), str.size()) unconditionally. When str was constructed from an empty std::string, str.data() is allowed to be NULL, and libc declares memcpy's second argument with __attribute__((nonnull)) regardless of the size. Every libc no-ops memcpy(_, NULL, 0) in practice, so the bug was benign on real workloads, but the false-positive UBSan trip was noising the extension's ASan job and obscuring real findings.

Patch: guard the memcpy with if (str.size() > 0). Submitted upstream as clickhouse-cpp#489, merged 2026-04-27. The local patch in lib/clickhouse-cpp/LOCAL_PATCHES.md will drop the next time the vendored library bumps.

What's Still Missing

Two limitations carry forward from clickhouse-cpp v2.6.1:

SELECT ... WITH TOTALS and SETTINGS extremes=1 throw unimplemented 7 from the cpp layer. The vendored library does not dispatch the Totals / Extremes packet types (upstream issue #297). getTotals() / getExtremes() are deferred.
Map(LowCardinality(K), V) reads are not yet decoded by the vendored library (writes succeed). showProcesslist() selects a fixed projection of standard columns to avoid the unsupported Map columns (ProfileEvents, Settings, used_*).

If either blocks your workload, file an issue at github.com/iliaal/php_clickhouse with the schema and a minimal repro. Both are upstream and tracked.

The repo is at github.com/iliaal/php_clickhouse. Install via PIE: pie install iliaal/php_clickhouse (add --enable-clickhouse-openssl for TLS). The original launch post that framed the fork story sits at ilia.ws.