Learning Goals

• Understand prefix-free codes and their properties.
• Learn Huffman's greedy algorithm for optimal codes.
• Analyze correctness using Greedy Choice and Optimal Substructure.
• Trace Huffman tree construction.

The Problem

• We want to store a file with $N$ characters.
• Standard ASCII: 8 bits per char.
• Total size: $8N$ bits.
• Can we do better if some chars are more frequent than others?

Example

• File: 'AAAAABBBCCD'
• A: 5, B: 3, C: 2, D: 1.
• Total chars: 11.
• Fixed-length (2 bits needed for 4 symbols):
• Cost: $11 \times 2 = 22$ bits.

Variable-Length Codes

• Idea: Short codes for frequent chars.
• A: 0 (1 bit)
• B: 10 (2 bits)
• C: 110 (3 bits)
• D: 111 (3 bits)
• Cost: $5(1) + 3(2) + 2(3) + 1(3) = 5 + 6 + 6 + 3 = 20$ bits.
• Savings: 10% (Simple example to show concept).

Ambiguity

• If A=0, B=1, C=01...
• String '01': Is it 'AB' or 'C'?
• We need Prefix Codes.
• Definition: No codeword is a prefix of another codeword.

Binary Tree Representation

• We can represent prefix codes as a Binary Tree.
• Leaves = Characters.
• Left edge = 0.
• Right edge = 1.
• Path from root to leaf = Code.

Cost of Tree

• For a tree $T$, cost $B(T) = \sum f(c) \times depth(c)$.
• Where $f(c)$ is frequency.
• Goal: Finding $T$ that minimizes $B(T)$.

Greedy Intuition

• High frequency chars should be near the root (short depth).
• Low frequency chars should be deep.
• Who should be deepest?
• The two rarest characters!

Huffman(C)

n = |C|
Q = C  // Min-Priority Queue keyed by frequency
for i = 1 to n-1
  z = Allocate-Node()
  x = Extract-Min(Q)
  y = Extract-Min(Q)
  z.left = x
  z.right = y
  z.freq = x.freq + y.freq
  Insert(Q, z)
return Extract-Min(Q) // Root

Step-by-Step Construction

• Chars: a:45, b:13, c:12, d:16, e:9, f:5.
• Q: [(f,5), (e,9), (c,12), (b,13), (d,16), (a,45)].
• 1. Extract f:5, e:9. Merge -> Node(14).
• Q: [(c,12), (b,13), (14), (d,16), (a,45)].

Step 2

• Extract c:12, b:13. Merge -> Node(25).
• Q: [(14), (d,16), (25), (a,45)].

Step 3

• Extract (14), d:16. Merge -> Node(30).
• Q: [(25), (30), (a,45)].

Step 4

• Extract (25), (30). Merge -> Node(55).
• Q: [(a,45), (55)].

Step 5

• Extract a:45, (55). Merge -> Root(100).
• Done.

Analysis

• We perform $n-1$ merges.
• Each step: 2 Extract-Min, 1 Insert.
• Using a Binary Heap: $O(\log n)$ per op.
• Total: $O(n \log n)$.

Lemma 1: Interpretation

• The two rarest characters $x, y$ must be siblings at the maximum depth in some optimal tree.
• Proof: Suppose they are not.
• Swap them with the nodes at max depth.
• Since $f(x), f(y)$ are minimal, swapping them deeper does not increase total cost.
• Thus there is an optimal tree where they are siblings.

Lemma 2: Optimal Substructure

• If we merge $x, y$ into $z$.
• $Cost(Tree) = Cost(Tree') + f(x) + f(y)$.
• Since cost contribution is additive.
• Solving for reduced alphabet $C'$ yields optimal for $C$.

Full Binary Tree

• Every internal node has exactly 2 children.
• An optimal prefix code is always a Full Tree.
• Why? If a node had 1 child, we could replace the node with the child and shorten codes.

Entropy

• Shannon's Entropy.
• Huffman coding approaches the theoretical limit $H = -\sum p_i \log_2 p_i$.
• It is optimal for symbol-by-symbol coding.

How to Decode

• Stream: 1011001...
• Start at Root.
• If 1: Go Right.
• If 0: Go Left.
• Leaf Reached? Output char, Reset to Root.
• Repeat.

Encoding/Decoding Table

• Need to store the tree or table in the file header.
• Otherwise decoder doesn't know the mapping.
• Overhead is small for large files.

Fibonacci Frequencies

• What if freqs are Fibonacci numbers: 1, 1, 2, 3, 5, 8...
• Trace Huffman.
• 1,1 -> 2.
• 2,2 -> 4.
• 3,4 -> 7.
• 5,7 -> 12.
• 8,12 -> 20.
• Result: A comb-shaped tree. Deepest char gets $n-1$ bits.

Wrap up

• Huffman is the quintessential Greedy Algorithm.
• Used in ZIP, JPEG, MP3.
• Correctness proved via Exchange Argument.
• Greedy is powerful!

Watch Out For...

• Forgetting that Huffman codes are prefix-free (no code is prefix of another).
• Confusing code length with code value (length matters for cost).
• Not using a min-heap (using max-heap gives wrong result).
• Forgetting to store the tree/table with the encoded file.

Huffman Tree Builder

🚀 Interactive Demo: huffman_demo.html

Activity 1: Build Huffman Tree

• ❓ Problem:
• Build a Huffman tree for: A:8, B:3, C:1, D:1, E:2
• Take 3 minutes to trace the algorithm.

Solution 1

• 💡 Answer:
• Merge order: (C:1,D:1)→2, (2,E:2)→4, (B:3,4)→7, (7,A:8)→15.
• Codes: A=1, B=01, E=001, C=0000, D=0001.

Activity 2: Calculate Expected Length

• ❓ Problem:
• Given: A:50%, B:25%, C:15%, D:10%. What is the expected code length?
• Take 2 minutes to solve this.

Solution 2

• 💡 Answer:
• A:1 bit, B:2 bits, C:3 bits, D:3 bits.
• Expected = $0.5(1) + 0.25(2) + 0.15(3) + 0.1(3) = 1.75$ bits.

Activity 3: Decode Huffman String

• ❓ Problem:
• Given tree with A=0, B=10, C=11. Decode: 01011100
• Take 2 minutes to solve this.

Solution 3

• 💡 Answer:
• 0|10|11|10|0 → A B C B A

Activity 4: Fixed vs Variable Comparison

• ❓ Problem:
• 5 chars with freqs 100, 50, 30, 15, 5. Fixed vs Huffman savings?
• Take 2 minutes to solve this.

Solution 4

• 💡 Answer:
• Fixed: 3 bits × 200 = 600 bits.
• Huffman: Merge 5+15→20, 20+30→50, 50+50→100, 100+100→200.
• Cost: 100(1)+50(2)+30(3)+15(4)+5(4) = 370 bits.
• Savings ≈ 38%.

Lecture Summary

• Prefix codes ensure unambiguous decoding (no code is prefix of another).
• Huffman's algorithm greedily merges the two lowest-frequency nodes.
• Time complexity: $O(n \log n)$ using a min-priority queue.
• Optimal for symbol-by-symbol encoding (approaches Shannon entropy).
• Used in real-world compression: ZIP, JPEG, MP3.

All Interactive Demos

🚀 Interactive Demo: huffman_demo.html

Supplementary Resources

🚀 Interactive Demo: worked_examples.html
🚀 Interactive Demo: reading_practice.html
🚀 Interactive Demo: handout.html