Overview

The second experiment explored whether frequently occurring byte values inside GGUF files could be represented using four-bit (nibble) symbols instead of full bytes.

Rather than compressing repeated sequences, the compressor builds a dictionary of common byte values and packs two mapped values into a single byte whenever possible.

The experiment remains fully lossless.

Motivation

Byte frequency analysis showed that many byte values appeared repeatedly throughout small samples.

Instead of storing every byte directly, the most common values were assigned four-bit identifiers.

Example:

Nibble 0 → 00
Nibble 1 → 10
Nibble 2 → FF

Whenever a byte existed inside the dictionary, only its four-bit identifier was written.

Unknown bytes were emitted as escape sequences.

Compression Pipeline

GGUF File
    ↓
Read Raw Bytes
    ↓
Build Byte Frequency Table
    ↓
Create Nibble Dictionary
    ↓
Pack Two Nibbles Per Byte
    ↓
Store Escape Bytes
    ↓
Write BFGN1 Format

Initial Results

Original Bytes : 5000
Mapped Bytes   : 4283
Escaped Bytes  : 717

Compressed Size: 3254
Saved          : 34.92%