Lecture 05: Sorting by Divide and Conquer

COSC 311 Algorithms, Fall 2022

$ \def\compare{ {\mathrm{compare}} } \def\swap{ {\mathrm{swap}} } \def\sort{ {\mathrm{sort}} } \def\insert{ {\mathrm{insert}} } \def\true{ {\mathrm{true}} } \def\false{ {\mathrm{false}} } \def\BubbleSort{ {\mathrm{BubbleSort}} } \def\SelectionSort{ {\mathrm{SelectionSort}} } \def\Merge{ {\mathrm{Merge}} } \def\MergeSort{ {\mathrm{MergeSort}} } $

Overview

  1. Efficiency of Sorting
  2. Review of Big O
  3. Sorting by Divide and Conquer: MergeSort

Previously

Careful analysis of correctness of SelectionSort

01  SelectionSort(a):
02    n <- size(a)
03    for j = 1 to n - 1 do
04      min <- j
05      for i = j+1 to n do
06        if compare(a, min, i) 
07          min <- i
08        endif
09      endfor
10      swap(a, j, min)
11    endfor

Today

Focus on efficiency

  • How many elementary operations does a procedure perform?

Questions about SelectionSort

01  SelectionSort(a):
02    n <- size(a)
03    for j = 1 to n - 1 do
04      min <- j
05      for i = j+1 to n do
06        if compare(a, min, i) 
07          min <- i
08        endif
09      endfor
10      swap(a, j, min)
11    endfor

How many comparisons?

How many swaps?

How many “elementary” operations (including arithmetic)?

Abstracting Away Unknowns

Uncertain Parameters:

  1. precise running times of elementary operations
  2. overhead associated with executing algorithm

Assume:

  1. elementary operations take constant (but unknown) time
  2. overhead of starting computation is negligible for large inputs

Focus on asymptotic growth of # of operations as function of input size

Big O Notation

Qualitative measure of function growth:

  1. ignore constant factors
  2. ignore values on small inputs

Formally: $f$, $g$ functions, write $f = O(g)$ if

  • exists $C > 0$ and $N \geq 1$ such that
    • $f(n) \leq C g(n)$ for all $n \geq N$

Big O in Pictures

Properties of O

  1. all constants are $O(1)$
  2. if $f(n) \leq g(n)$ for all $n$, then $f = O(g)$
    • if $a \leq b$ then $n^a = O(n^b)$
    • if $a < b$ then $n^b \neq O(n^a)$
  3. if $f = O(g)$, then $a \cdot f = O(g)$
  4. if $f = O(g)$ and $g = O(h)$, then $f = O(h)$
  5. if $f, g = O(h)$, then $f + g = O(h)$
  6. if $f_1 = O(g_1)$ and $f_2 = O(g_2)$ then $f_1 \cdot f_2 = O(g_1 \cdot g_2)$

Running time of SelectionSort in O?

01  SelectionSort(a):
02    n <- size(a)
03    for j = 1 to n - 1 do
04      min <- j
05      for i = j+1 to n do
06        if compare(a, min, i) 
07          min <- i
08        endif
09      endfor
10      swap(a, j, min)
11    endfor

Faster Sorting?

Divide and Conquer

Divide and Conquer Strategy

Given a computational task:

  1. Divide the task into sub-tasks
    • smaller instances of same task
  2. Solve the sub-tasks recursively
  3. Combine solutions to solve original task

Question

How could divide the sorting task into sub-tasks?

Sorting by D&C

  1. Divide by index
    • MergeSort
  2. Divide by value
    • QuickSort
  3. Divide by bit representation
    • RadixSort

Dividing by Index: MergeSort

As before, $a$ an array of size $n$

  • access/assignment by index (not just compare/swap)

MergeSort algorithm:

  1. divide $a$ into halves $m = (n+1) / 2$
  2. sort $a[1..m-1]$ recursively
  3. sort $a[m..n]$ recursively
  4. merge $a[1..m-1]$ and $a[m..n]$ to form sorted array

Pseudocode

# sort values of a between indices i and j-1
MergeSort(a, i, j):
  if j - i = 1 then
    return
  endif
  m <- (i + j) / 2
  MergeSort(a,i,m)
  MergeSort(a,m,j)
  Merge(a,i,m,j)

Illustration of MergeSort

Merge Procedure

Correctness of MergeSort

Establish two claims:

Claim 1 (merge). If $a[i..m-1]$ and $a[m..j]$ are sorted, then after $\Merge(a, i, m, j)$, $a[i..j]$ is sorted.

  • Argued on lecture ticket!

Claim 2. For any indices $i < j$, after calling $\MergeSort(a, i, j)$, $a[i..j]$ is sorted.

  • How to argue this? (assume Claim 1 is true…)

Pseudocode Again

00  # sort values of a between indices i and j-1
01  MergeSort(a, i, j):
02    if j - i = 1 then
03      return
04    endif
05    m <- (i + j) / 2
06    MergeSort(a,i,m)
07    MergeSort(a,m,j)
08    Merge(a,i,m,j)

Inductive Claim

Consider $\MergeSort(a, i, j)$, define $k = j - i$ to be size

$P(k)$: for every $k’ \leq k$, $\MergeSort(a, i, j)$ with size $k’$ succeeds

Base case $k = 1$:

Inductive step $P(k) \implies P(k+1)$:

Question

How efficient is MergeSort?

00  # sort values of a between indices i and j-1
01  MergeSort(a, i, j):
02    if j - i = 1 then
03      return
04    endif
05    m <- (i + j) / 2
06    MergeSort(a,i,m)
07    MergeSort(a,m,j)
08    Merge(a,i,m,j)

Analyzing Running Time

00  # sort values of a between indices i and j-1
01  MergeSort(a, i, j):
02    if j - i = 1 then
03      return
04    endif
05    m <- (i + j) / 2
06    MergeSort(a,i,m)
07    MergeSort(a,m,j)
08    Merge(a,i,m,j)

Observation 1. Let $k = j - i$ be the size of the method call $\MergeSort(a, i, j)$. Then running time is $O(k) + $ running time of recursive calls on lines 6-7.

Observation 2. Recursive calls have size $k / 2$.

Combining Observations

Recall Logarithms (base 2)

Define $\log$ by

  • $\log a = b \iff 2^b = a$

Another way

  • $\log a$ is # times $a$ can be divided by $2$ to get (at most) $1$.

Facts.

  1. For every constant $c > 0$, $\log n = O(n^c)$.
  2. $\log n \neq O(1)$.

A Final Calculation

  • Running time $T(n)$ of $\MergeSort(a, 1, n+1)$:

    \[\begin{align*} T(n) &= 2 T(n/2) + O(n)\\ &= 4 T(\frac n 4) + 2 O(\frac n 2) + O(n)\\ &= 8 T(\frac n 8) + 4 O(\frac n 4) + 2 O(\frac n 2) + O(n)\\ &\vdots\\ &= n T(1) + \frac n 2 O(2) + \cdots + 8 O(\frac n 8) + 4 O(\frac n 4) + 2 O(\frac n 2) + O(n)\\ &= O(n) + O(n) + \cdots + O(n) + O(n) + O(n)\\ &{}\\ &= \end{align*}\]

Coming Up

  1. More Divide and Conquer Sorting!
  2. Can we sort even faster?!