Further complexity rules

• Big Oh is reflexive: $f(n) = O(f(n))$
  Prove by setting $c$ = 1 and $n_0$ = 0
• Big Oh is transitive: $f(n) = O(g(n))$ and $g(n) = O(h(n)) \rightarrow f(n) = O(h(n))$
  Determine $c$ and $n_0$ that makes this work!
• Drop low order term: \[f(n) = O(g(n)) \rightarrow f(n) + g(n) = \theta(g(n))\]
• Drop the leading constant: \[f(n) = O(g(n)) \mbox{ and } h(n) = O(k(n)) \rightarrow f(n) \times h(n) = O(g(n) \times k(n))\]
• Generally, \[\sum_{i = 0}^{k} a_i n^i = \theta(n^k)\]

Further complexity rules

• Two useful rules for logarithms: 1 = $n^0$ = $o(\log n)$, $\log n$ = $o(n^r)$ for all positive $r$. So $\log n$ lies somewhere between constant time and $n^r$ for any $r$.

Asymptotic Dominance

Recall \[\begin{array}{lcl} f(n) = o(g(n)) & \mbox{iff} & \lim_{n \rightarrow \infty} \frac{f(n)}{g(n)} = 0 \\ f(n) = \omega(g(n)) & \mbox{iff} & \lim_{n \rightarrow \infty} \frac{f(n)}{g(n)} = \infty \\ f(n) = \theta(g(n)) & \mbox{iff} & \lim_{n \rightarrow \infty} \frac{f(n)}{g(n)} = r > 0 \end{array}\]

• We say $f(n)$ dominates $g(n)$, written $f(n) >> g(n)$ if $g(n) = O(f(n))$
We can therefore state a dominance ranking of complexity classes which commonly arise in algorithm analysis: \[n^n >> n! >> 2^n >> n^3 >> n^2 >> n \log n >> n >> \log n >> 1\] You must remember these – listing them in order should be second nature

Asymptotic Dominance

Including some more obsecure classes of complexity \[\begin{array}{l} n^n >> n! >> c^n >> n^3 >> n^2 >> n^{1+\varepsilon} >> \\ n \log n >> n >> \sqrt{n} >> \log^2 n >> \log n >> \\ \frac{\log n}{\log\log n} >> \log\log n >> \alpha(n) >> 1\end{array}\]

Asymptotic Dominance in Action

On a computer executing 1 instruction per nanosecond, running time for varying input size:

The big match up...

IBM Roadrunner $1.026$ quadrillion calculations per second ($1.026 \times 10^15$) Running an Ο($n^2$) algorithm

The big match up...

2.3 million times slower, stacked on top of each other would be 27km high!
But we’re going to beat Roadrunner with one iPhone using a better algorithm, Apple iPhone 450 million calculations per second Running an Ο($n \log n$) algorithm

The big match up...

Binary Search

• Binary search is a very useful method for searching an ordered list
  Given an array: a[1:n] where $a[1] \leq a[2] \leq ... \leq a[n]$
• We wish to find the element $x$ in the list
  i.e. we require the index: $i$ where $a[i-1] < x \leq a[i]$
• Q: Why can’t we just look for $a[i] = x$?
• Strategy:
  1. Examine element $a[m]$ where $\displaystyle m = \frac{n}{2}$
  2. If $x = a[m]$, stop – we’re done
  3. If $x > a[m]$, then throw away bottom half of list, recursively search remaining half, otherwise vice versa

Binary Search

• For each comparison, we half the number of potential items
• Q: Complexity of binary search (number of comparisons):
• Best case?
• We find it at the middle element: $T(n) = \theta(1)$
• Worst case?
• We have to keep halving the search space until only one item remains: $T(n) = \theta(\log(n))$
• Implications:
  Find any name in the Manhattan phone book (1 million names) in 20 comparisons
• Never lose at “20 questions” again – binary search the dictionary

Basic Algorithm Analysis

• To summarize, our simplifications are:
  • Real computer to RAM model
  • Running time measured by number of instructions
  • Group inputs by size and focus on worst case
  • Asymptotic analysis ignores constant factors/low order terms
• Result of these is that asymptotic complexity can be found directly from pseudocode
• Employ some simple rules for worst case analysis of simple structures

Basic Algorithm Analysis

• Sequenced statements:
• If $T(I_i)$ is worst-case execution time of statement $I_i$, then for sequenced statements: \[T(I_1; I_2) = T(I_1) + T(I_2)\]

Basic Algorithm Analysis

• Choice statements:
• So we can’t get an exact answer → we can’t say for certain what $T(I)$ will equal
• Q: Suggestions?
• We’re interested in worst case, so we take the slowest
• Because answer is inexact we use Big Oh, we’re saying running time is bounded from above by this worst case time: \[T(\mbox{If } C \mbox{ then } I_1 \mbox{ else } I_2) = O(T(C) + max\{T(I_1), T(I_2)\})\]

Basic Algorithm Analysis

• Iteration statements:
• Q: What information are we missing to give an answer?
• We need to know how many times the loop will iterate Assuming we know this (= $n$), time is: \[T(\mbox{while } C \mbox{ do } I) = O \left(\sum_{i = 1}^{n} T(C) + T(I) \right)\]
• Iteration statements (loops) are more complex than sequential or choice
• Require use of sums

Algorithm analysis example

• Analyse running time as function of $n$.
• Q: How many times do we increment s? What is the asymptotic complexity?
• Example 1
• We go round the loop $n$ times, so $s$ is incremented $n$ times: $f(n) = \theta(n)$
• Expressed as a sum: $\displaystyle\sum_{i = 1}^{n} 1 = n$

Algorithm analysis example

• Analyse running time as function of n
• Q: How many times do we increment s? What is the asymptotic complexity?
• Example 2
• Same as before the inner for-loop: $\displaystyle\sum_{j = 1}^{n} 1 = n$
• The outer for-loop: $\displaystyle\sum_{i = 1}^{n} n = n^2$ and complexity is: $f(n) = \theta(n^2)$

Analyze running time

• Q: How many times do we increment $s$? What is the asymptotic complexity?
• Example 3
• Same as before the inner for-loop: $\displaystyle\sum_{j = 1}^{i} 1 = i$
• The outer for-loop: $\displaystyle\sum_{i = 1}^{n} i = \frac{n(n+1)}{2}$.
• We do less work than before, however, the asymptotically two algorithms have the same running time: $f(n) = \theta(n^2)$

Tower of Hanoi

Move $n$ discs from A to B using C as a spare observing following rules:

• Only one disc can be moved at a time
• Discs can only be placed on top of larger discs

Recursive Algorithm:Classic example

• To move $n$ discs from A to B:
• First move $n-1$ discs from A to C, then move 1 from A to B, finally move $n-1$ discs back to B

Tower of Hanoi recursive algorithm

• From our example, we would call function with: ToH(8,A,B,C)
• If we measure time in terms of how many moves we have to make. What is $T(n)$?
• Base case is T(1) = 1, in general $\displaystyle T(n) = \left\{\begin{array}{ll} 1 & \mbox{ if } n = 1 \\ 2 T(n-1) + 1 & \mbox{ if } n > 1\end{array}\right.$

Recursive Algorithm Analysis

T(n)

Recursive Algorithm Analysis

Recursive Algorithm Analysis

Recursive Algorithm Analysis

Recursive Algorithm Analysis

Recursive Algorithm Analysis

Recursive Algorithm Analysis

Recursive Algorithm Analysis

• The standard identity for geometric series with the ratio $r$ is $\displaystyle\sum_{i = 0}^{n} r^i = \frac{1 - r^{n+1}}{1 - r}.$
• Hence, Tower of Hanoi is exponential complexity: $T(n) = \theta(2^n)$.