🖨️ Printing Instructions: Press Ctrl/Cmd + P and select "Save as PDF".

Asymptotic Notations

Big-O, Omega, Theta

Learning Goals

Master Big-O, Omega, Theta.
Understand growth rates.
Prove bounds algebraically.

Introduction & Motivation

How do we measure algorithm speed?

We want to compare algorithms, not computers.
Wall-clock time depends on:
Hardware (CPU, RAM, Cache)
Compiler / Interpreter
System Load
Input data specifics
We need a MACHINE-INDEPENDENT metric.

Counting Operations

Instead of seconds, we count 'Elementary Operations'.
Assignments: $x \leftarrow 5$
Arithmetic: $a + b$
Comparisons: $a < b$
Array Access: $A[i]$
We assume these take constant time $O(1)$.

Example: Linear Search

Algorithm:

function LinearSearch(A, x)
  for i = 0 to n-1 do
    if A[i] == x return i
  return -1

Best case: 1 comparison (First element).
Worst case: $n$ comparisons (Last element or Not found).
Average case: $n/2$ comparisons.
We care most about the WORST CASE.

Input Size (n)

We express runtime $T(n)$ as a function of input size $n$.
For sorting: $n$ = number of items.
For graphs: $V$ vertices, $E$ edges.
For primality test: $b$ bits (number of digits).
We study how $T(n)$ grows as $n \to \infty$.

Big-O Notation ($O$)

The Intuition of Big-O

Big-O is an UPPER BOUND.
Like saying "I have less than $100 in my pocket."
If an algorithm is $O(n^2)$, it is essentially "At most quadratic".
It suppresses constant factors and lower-order terms.

Formal Definition of Big-O

Definition:
$f(n) \in O(g(n))$ if there exist positive constants $c$ and $n_0$ such that:
$$0 \le f(n) \le c \cdot g(n) \quad \forall n \ge n_0$$
This means for large enough $n$, $f(n)$ is bounded above by a scaled version of $g(n)$.

Visualizing Big-O

Graphing $f(n)$ and $c \cdot g(n)$:
At first, $f(n)$ might be larger.
But after the crossing point $n_0$, $c \cdot g(n)$ stays above $f(n)$ forever.
Crucial: We can pick a HUGE $c$ if needed.

Interactive: Big-O Definition

Interactive Demo: big_o_interactive.html

Example 1: Linear Function

Claim: $3n + 10 \in O(n)$.
We need to find $c, n_0$.
Set $n_0 = 10$. Then for $n \ge 10$, $10 \le n$.
$3n + 10 \le 3n + n = 4n$.
So pick $c = 4, n_0 = 10$.
Condition holds: $3n+10 \le 4n$ for $n \ge 10$.

Example 2: Another Proof

Claim: $100n + 5 \in O(n^2)$.
Is this true?
Yes! $O$ is an upper bound, not necessarily tight.
$100n + 5 \le 100n + n = 101n$ (for $n \ge 5$).
And $101n \le 101n^2$ since $n \le n^2$.
So $100n + 5 \le 101n^2$. ($c=101, n_0=5$).

Big-O Checklist

1. Drop lower order terms.
$3n^2 + 10n + 5 \to 3n^2$
2. Drop constant coefficients.
$3n^2 \to n^2$
Result: $O(n^2)$.

Why do we drop constants?

Constants depend on hardware/implementation.
In assembly: `ADD` might be 1 cycle.
In Python: `+` might be 100 cycles.
But $n^2$ operations will always overwhelm $n$ operations eventually, regardless of the constant factor.

Common Misunderstanding

False: $f(n) = O(g(n))$ implies equality.
True: It is set membership $f(n) \in O(g(n))$.
We write $=$ by convention (abuse of notation).
Read it as "f(n) is Order g(n)".

Big-Omega Notation ($\Omega$)

The Intuition of Big-Omega

Big-Omega is a LOWER BOUND.
Like saying "I have at least $5 in my pocket."
Useful for proving hardness.
"Sorting takes $\Omega(n \log n)$" means no comparison sort can ever be faster than that.

Formal Definition of Big-Omega

Definition:
$f(n) \in \Omega(g(n))$ if there exist positive constants $c, n_0$ such that:
$$0 \le c \cdot g(n) \le f(n) \quad \forall n \ge n_0$$
Note the inequality direction flip.
$f(n)$ dominates $g(n)$.

Example: Big-Omega

Claim: $n^2 \in \Omega(n)$.
Proof: We need $c \cdot n \le n^2$.
Divide by $n$: $c \le n$.
Pick $c=1$. Condition holds for all $n \ge 1$.
So $n^2 \in \Omega(n)$. (Quadratic is 'at least' linear).

Another Example

Claim: $3n + 5 \in \Omega(n)$.
Proof:
$3n + 5 \ge 3n$.
So pick $c=3, n_0=1$.
Condition $3n \le 3n+5$ holds.

Big-Theta Notation ($\Theta$)

The Intuition of Big-Theta

Big-Theta is a TIGHT BOUND.
Sandwiched between Upper and Lower bounds.
"I have exactly $5 (give or take cents)."
Most precise description of an algorithm's behavior.

Formal Definition of Big-Theta

Definition:
$f(n) \in \Theta(g(n))$ if:
$$f(n) \in O(g(n)) \quad \text{AND} \quad f(n) \in \Omega(g(n))$$
Equivalently: $\exists c_1, c_2, n_0$ such that:
$$c_1 g(n) \le f(n) \le c_2 g(n) \quad \forall n \ge n_0$$

Example: Quadratic

Let $f(n) = \frac{1}{2}n^2 - 3n$.
Show $f(n) \in \Theta(n^2)$.
1. Upper bound: $\frac{1}{2}n^2 - 3n \le \frac{1}{2}n^2$. ($c_2=1/2$).
2. Lower bound: For large $n$, $\frac{1}{2}n^2 - 3n \ge c_1 n^2$?
Yes, for $n \ge 7$, $\frac{1}{2}n^2 - 3n \ge \frac{1}{14}n^2$ (roughly).
Or simply: limit is constant.

The Limit Method

Using Calculus Limits

Often easier than finding $c, n_0$.
Compute $L = \lim_{n \to \infty} \frac{f(n)}{g(n)}$.
If $L = 0$: $f(n) \in O(g(n))$ but not $\Theta$. (Little-o)
If $0 < L < \infty$: $f(n) \in \Theta(g(n))$.
If $L = \infty$: $f(n) \in \Omega(g(n))$ but not $\Theta$. (Little-omega)

Example: Limits

Compare $f(n) = 3n^2 + n$ and $g(n) = n^2$.
$$L = \lim_{n \to \infty} \frac{3n^2 + n}{n^2}$$
$$L = \lim_{n \to \infty} (3 + \frac{1}{n}) = 3$$
Since $0 < 3 < \infty$, $f(n) \in \Theta(n^2)$.

Example: Polynomials

Compare $n^3$ and $n^2$.
$$L = \lim_{n \to \infty} \frac{n^3}{n^2} = \lim_{n \to \infty} n = \infty$$
So $n^3$ grows strictly faster.
$n^3 \in \Omega(n^2)$.

L'Hopital's Rule

Useful for logs and exponentials.
If $\lim f(n)/g(n)$ is $\frac{\infty}{\infty}$, take derivatives.
Compare $\ln n$ vs $n$.
$$\lim \frac{\ln n}{n} = \lim \frac{1/n}{1} = 0$$
So $\ln n \in O(n)$ (strictly slower).

Common Dominance Hierarchy

The Hierarchy (Memorize This)

$$1 \prec \log n \prec \sqrt{n} \prec n \prec n \log n \prec n^2 \prec n^3 \prec 2^n \prec n!$$
Each function is Little-o of the next.
$O(1)$: Constant time (Index array).
$O(\log n)$: Logarithmic (Binary Search).
$O(n)$: Linear (Loop).

Interactive: Growth Rates

Interactive Demo: growth_rates_demo.html

Exponential Growth

$2^n$: Exponential.
Add 1 to input $\to$ double the time.
Impossible for large $n$. (The "Intractable" barrier).
Example: Brute force password cracking.

Properties of Asymptotic Notation

Transitivity

If $f \in O(g)$ and $g \in O(h)$, then $f \in O(h)$.
Example: If $f(n) \le n$ and $n \le n^2$, then $f(n) \le n^2$.
Holds for $\Omega$ and $\Theta$ too.

Additivity

If $f \in O(g)$, then $f+g \in O(g)$.
Also: $O(f + g) = O(\max(f, g))$.
Example: $n^2 + n = O(n^2)$.

Multiplication by Constant

If $f \in O(g)$, then $k \cdot f \in O(g)$.
This is why we ignore coefficients.
$1000n^2$ is effectively the same class as $n^2$.

Practice Problems

Problem 1

Is $2^{n+1} \in O(2^n)$?
Answer: YES.
$2^{n+1} = 2 \cdot 2^n$.
Since constants don't matter, $O(2 \cdot 2^n) = O(2^n)$.

Problem 2

Is $2^{2n} \in O(2^n)$?
Answer: NO.
$2^{2n} = (2^n)^2 = 4^n$.
Limit $\frac{4^n}{2^n} = 2^n \to \infty$.

Stirling's Approximation

How fast does $n!$ grow?
$$n! \approx \sqrt{2\pi n} \left(\frac{n}{e}\right)^n$$
Useful for analyzing $\log(n!)$:
Taking logs: $\ln(n!) \approx n \ln n - n$.
Consequence: $\log(n!) \in \Theta(n \log n)$.

Problem 3

Is $\log(n!) \in \Theta(n \log n)$?
Answer: YES.
Stirling's Approximation.
Or simple bound: $n! \le n^n → \log n! \le n \log n$.
$n! \ge (n/2)^{n/2} → \log n! \ge \frac{n}{2} \log \frac{n}{2} \approx \Theta(n \log n)$.

Problem 4

Order these functions by growth rate:
$n^{1.5}, n \log n, n!, 100n, 2^{\log n}$
Note: $2^{\log n} = n$.
Order: $2^{\log n} = n$ and $100n$ (Tied $\Theta(n)$) $\prec n \log n \prec n^{1.5} \prec n!$

Little-o and Little-omega

Strict Bounds

Little-o ($o$): Strictly less than.
Definition: $f(n) \in o(g(n))$ if limit is 0.
Example: $n \in o(n^2)$.
But $2n \notin o(n)$.

Analogy with Real Numbers

$f \in O(g) \approx f \le g$
$f \in \Omega(g) \approx f \ge g$
$f \in \Theta(g) \approx f = g$
$f \in o(g) \approx f < g$
$f \in \omega(g) \approx f > g$

Practical Importance

Why do we care?
If you have $N=10^6$ items:
$O(n)$ takes 1ms.
$O(n^2)$ takes 1000 seconds (~16 mins).
$O(2^n)$ takes longer than universe age.

Space Complexity

We can use the same notation for Memory.
Array of size $n$: $O(n)$ space.
2D Matrix: $O(n^2)$ space.
Recursive stack depth: e.g., $O(\log n)$ for Quicksort.

Analyzing Code Structures

Nested Loops

python:

for i in range(n):
  for j in range(n):
    print(i, j)

Inner loop runs $n$ times.
Outer loop runs $n$ times.
Total: $n \times n = n^2$.

Dependent Inner Loop

python:

for i in range(n):
  for j in range(i):
    print(i, j)

Inner runs $0, 1, 2, ..., n-1$ times.
Sum: $\sum_{i=0}^{n-1} i = \frac{n(n-1)}{2}$.
Class: $\Theta(n^2)$ still.

Consecutive Loops

python:

for i in range(n): ...
for j in range(n): ...

Total: $n + n = 2n$.
Class: $\Theta(n)$.
Key: Add consecutive, Multiply nested.

Logarithmic Loops

python:

i = 1
while i < n:
  i = i * 2

i takes values $1, 2, 4, 8, ...$
Runs $k$ steps where $2^k \approx n$.
$k = \log_2 n$.
Class: $\Theta(\log n)$.

Common Pitfalls & Anti-Patterns

Watch Out For...

Thinking O(g(n)) means 'equals'. It means 'subset of'.
Confusing worst-case (a scenario) with Big-O (a bound, usually on worst-case).

Interactive Practice

Activity 1: Basic Polynomial

❓ Problem:
Prove $3n^2 - 100n + 6 \in O(n^2)$.
Take 2 minutes to solve this.

Solution 1

💡 Answer:
Choose $n_0$ such that $-100n$ is overwhelmed... $3n^2 + 6 \le 4n^2$ for $n \ge 3$.

Activity 2: Big-Omega

❓ Problem:
Prove $n^{1.1} \in \Omega(n)$.
Take 2 minutes to solve this.

Solution 2

💡 Answer:
Use limits or definition. $n^{1.1}/n = n^{0.1} \to \infty$.

Activity 3: Limits

❓ Problem:
Compare $n \log n$ and $n^{1.5}$.
Take 2 minutes to solve this.

Solution 3

💡 Answer:
Limit is 0. So $n \log n \in o(n^{1.5})$.

Activity 4: Little-o

❓ Problem:
Is $2n \in o(n)$?
Take 2 minutes to solve this.

Solution 4

💡 Answer:
Limit is 2. Not 0. So False.

Activity 5: Exponential

❓ Problem:
Is $2^{n+1} \in O(2^n)$?
Take 2 minutes to solve this.

Solution 5

💡 Answer:
Yes. $2^{n+1} = 2 \cdot 2^n$. Constant factor 2.

Activity 6: Factorial

❓ Problem:
Compare $n!$ and $2^n$.
Take 2 minutes to solve this.

Solution 6

💡 Answer:
$n!$ grows much faster. $n! \in \Omega(2^n)$.

Supplementary Resources

Interactive Demo: worked_examples.html

Interactive Demo: reading_practice.html

Interactive Demo: handout.html