BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//sf26-agenda//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
BEGIN:VEVENT
UID:SESS-15@sf26-agenda
DTSTAMP:20261020T151709Z
DTSTART:20261105T193000Z
DTEND:20261105T201500Z
SUMMARY:The MirrorMaker That Lies to Kafka (and Gets Away With It)
LOCATION:Room 2000
DESCRIPTION:Vahid Hashemian\n\nCross-cluster Kafka replication is one of th
 e most CPU- and network-expensive workloads in a large streaming platform.
  Apache Kafka's built-in MirrorMaker decompresses every batch\, deserializ
 es every record\, then re-batches and re-compresses on the destination - b
 urning cycles to reproduce bytes that already existed on the source. At Pi
 nterest's scale\, those cycles added up to entire fleets of machines doing
  nothing but encoding and decoding. For years\, we ran a "shallow" MirrorM
 aker that forwarded entire compressed RecordBatches verbatim\, preserving 
 compression\, timestamps\, and CRCs\, and only rewriting the few header fi
 elds that must change on the destination. The catch? It required substanti
 al changes to Kafka broker internals. Every Kafka upgrade meant weeks of r
 e-porting a moving target. Enter NeoShallowMirror\, our new shallow Mirror
 Maker. It tells Kafka a very careful lie: that it is producing and consumi
 ng individual records one at a time. In reality\, whole compressed RecordB
 atches are shuttled from source to destination untouched\, and Kafka's int
 ernal accounting is kept perfectly consistent with the fiction. The result
 : we cut MirrorMaker CPU by an order of magnitude and turned Kafka upgrade
 s from weeks of painful re-porting into a simple dependency bump. Key take
 aways for attendees: The Shallow Trick: A deep dive into Kafka internals -
  what's safe to rewrite before the CRC field\, what isn't\, and how to fus
 e a whole batch into a single synthetic Record without breaking validation
  or offsets. The Dependency Trick: A repeatable patching pattern you can r
 euse the next time the right fix lives inside a third-party dependency you
 'd rather not fork. Performance Wins: How bypassing serialization and dese
 rialization at scale translates into much smaller footprints and more stab
 le mirror clusters.
URL:https://sf26.rmoff.net/s/the-mirrormaker-that-lies-to-kafka-and-gets-aw
 ay-with-it-15
END:VEVENT
END:VCALENDAR