The MirrorMaker That Lies to Kafka (and Gets Away With It)

Cross-cluster Kafka replication is one of the most CPU- and network-expensive workloads in a large streaming platform. Apache Kafka's built-in MirrorMaker decompresses every batch, deserializes every record, then re-batches and re-compresses on the destination - burning cycles to reproduce bytes that already existed on the source. At Pinterest's scale, those cycles added up to entire fleets of machines doing nothing but encoding and decoding.

For years, we ran a "shallow" MirrorMaker that forwarded entire compressed RecordBatches verbatim, preserving compression, timestamps, and CRCs, and only rewriting the few header fields that must change on the destination. The catch? It required substantial changes to Kafka broker internals. Every Kafka upgrade meant weeks of re-porting a moving target.

Enter NeoShallowMirror, our new shallow MirrorMaker. It tells Kafka a very careful lie: that it is producing and consuming individual records one at a time. In reality, whole compressed RecordBatches are shuttled from source to destination untouched, and Kafka's internal accounting is kept perfectly consistent with the fiction. The result: we cut MirrorMaker CPU by an order of magnitude and turned Kafka upgrades from weeks of painful re-porting into a simple dependency bump.

Key takeaways for attendees:

• The Shallow Trick: A deep dive into Kafka internals - what's safe to rewrite before the CRC field, what isn't, and how to fuse a whole batch into a single synthetic Record without breaking validation or offsets.
• The Dependency Trick: A repeatable patching pattern you can reuse the next time the right fix lives inside a third-party dependency you'd rather not fork.
• Performance Wins: How bypassing serialization and deserialization at scale translates into much smaller footprints and more stable mirror clusters.