Lecture 08: Sorting Lower Bounds

COSC 311 Algorithms, Fall 2022

$ \def\compare{ {\mathrm{compare}} } \def\swap{ {\mathrm{swap}} } \def\sort{ {\mathrm{sort}} } \def\insert{ {\mathrm{insert}} } \def\true{ {\mathrm{true}} } \def\false{ {\mathrm{false}} } \def\BubbleSort{ {\mathrm{BubbleSort}} } \def\SelectionSort{ {\mathrm{SelectionSort}} } \def\Merge{ {\mathrm{Merge}} } \def\MergeSort{ {\mathrm{MergeSort}} } \def\QuickSort{ {\mathrm{QuickSort}} } \def\Split{ {\mathrm{Split}} } $

Announcements

1. Homework 2 Draft Posted
2. Homework Late Days

Overview

1. QuickSort Again
2. Sorting Lower Bound

Last Time

QuickSort(a, i, j):
  if j - i <= 1 then
    return
  endif
  p <- GetPivot(a, i, j) # select a pivot
  k <- Split(a, i, j, p)
  QuickSort(a, i, k-1)
  QuickSort(a, k+1, j)

Split Before

Split(a, i, j, p):
  left <- i, right <- j
  while left < right do
    if a[left] > p and a[right] <= p then
       swap(a, left, right)
       left++, right--
    else
      if a[left] <= p then left++
      if a[right] > p then right--
    endif
  endwhile
  return right

What is the problem?

Split Updated

Split(a, i, j, pIndex):
  pivot <- a[pIndex]
  swap(a, pIndex, j);        # move pivot to last index
  small <- i - 1;
  for cur in range i..j do
    if a[cur] <= pivot then
      small <- small + 1
      swap(a, small, cur)
    endif
  endfor
  return small

Invariants:

1. elements $\leq$ pivot are at indices [i..small]
2. elements $>$ pivot are at indices [small+1..cur]

So far

1. $O(n^2)$ sorting: SelectionSort, InsertionSort, BubbleSort
2. $O(n^2)$ worst-case, $O(n \log n)$ average case: QuickSort
3. $O(n \log n)$ worst-case: MergeSort

Question. Can we do better?

Lecture Ticket

Sorting arrays of size $n$ requires $\Omega(n)$ operations.

Why?

Today’s Lower Bound

Any algorithm that accesses and modifies arrays using only $\compare$ and $\swap$ operations requires $\Omega(n \log n)$ comparisons.

• MergeSort is (asymptoticaly) optimal?

Ingredients of Lower Bound

Consider. Arrays are permutations of $1,2,\ldots,n$

Main idea. $a$ and $b$ are distinct arrays and $A$ is an algorithm

1. $A$ distinguishes $a$ and $b$ if $A(a)$ and $A(b)$ both make a call to $\compare(\cdot, i, j)$ with $\compare(a, i, j) \neq \compare(a, i, j)$

2. if $A$ does not distinguish $a$ and $b$, then it does not sort both $a$ and $b$

3. If $A$ performs too few $\compare$ operations, then it cannot distinguish all arrays of size $n$

   • $\implies A$ does not sort all arrays of size $n$

Decision Trees I

Consider:

1. Fixed sorting algorithm $A$

    InsertionSort(a):
      for i = 2 to n do
        j <- i
        while j > 1 and compare(a, j-1, j) do
           swap(a, j, j-1)
        endwhile
      endfor
   

2. Fixed set of inputs of size $n$

   • $S_n =$ permutations of $1,2,\ldots,n$
   • How many arrays in $S_n$?

Decision Trees II

Follow execution of $A$ on all inputs from $S_n$

Define a binary tree:

1. each node corresponds to a single $\compare$ operation
2. each node has two children corresponding to two possible outcomes of $\compare$

Form this tree for all comparisons made on all inputs in $S_n$

• label each node with inputs consistent with all $\compare$ outcomes

Decision Tree Example, n = 3

  for i = 2 to n do
  | j <- i
  | while j > 1 and compare(a, j-1, j) do
  | | swap(a, j, j-1)

Decision Tree Depth

Question. What does the depth of decision tree correspond to in terms of execution of $A$?

Indistinguishability

Question. If $A$ distinguishes $a$ and $b$, what can we say about nodes labeled with $a$ and $b$?

Indistinguishability Claim

Claim. If $A$ does not distinguish $a$ and $b$ with $a \neq b$, then $A$ does not sort both $a$ and $b$.

Why?

Indistinguishability Consequence

Consequence. If $A$ sorts all arrays of size $n$, then every leaf of $A$’s decision tree is labeled with a single permutation array.

Why?

How Big is Decision Tree?

How many leaves must a correct decision tree have?

How deep must decion tree be?

Putting it All Together

1. Consider any sorting algorithm $A$
2. Fix inputs $S_n = $ permutations of size $n$
3. Decision tree must have at least $\mid S_n \mid = n!$ leaves
4. Decision tree must have depth at least $\log n!$
5. $A$ must perform at least $\log n!$ comparisons

Claim. $\log n! = \Omega(n \log n)$

Conclusion

Theorem. Any algorithm that sorts all permutations of size $n$ using only $\compare$ and $\swap$ operations requires $\Omega(n \log n)$ comparisons.

Next Time

When can we sort faster than $O(n \log n)$?