Understanding Puffin File Format for Advanced Query Optimization in Apache Iceberg

Apache Iceberg manifests store essential statistics such as min/max values and null counts, but as tables scale, query engines increasingly require richer metadata to make efficient optimization decisions. Puffin was introduced in Iceberg to address this gap by providing a file format for storing advanced statistics, sketches, and secondary indexes that do not naturally fit within manifests.

In this session, we will explore the Puffin file format under the hood, including its structure, blob-based design, and snapshot-aware metadata model. We'll examine the two Puffin blob types supported today - Theta Sketches for approximate distinct counts and Deletion Vectors for row-level deletes introduced in Iceberg v3, and discuss how they enable smarter query planning, more efficient delete handling, and metadata-driven optimizations without rewriting data files.

We will also look at how Puffin is already being adopted in production, with systems such as AWS Glue and Impala producing and consuming Puffin metadata to power smarter optimizers and table operations. Finally, we will discuss why Puffin is becoming an increasingly important building block in Iceberg's evolution and how it lays the foundation for future metadata-driven innovations.

Attendees will leave with:

• An understanding of the Puffin file format and where it fits within Iceberg's metadata architecture.
• Knowledge of how Theta Sketches and Deletion Vectors are stored and utilized by query engines.
• Real-world examples of how Puffin metadata is being used to improve query planning and table maintenance.
• Insight into why Puffin is becoming a key component of future Iceberg optimization strategies.