Asymptotic notations

Let us consider the array sorting problem. Although it is not a decision problem, it is a sufficiently simple example. The sorting problem is sub-exponential: it can be solved in polynomial time (by many known algorithms and implicitly, by a Turing Machine).

Let us consider one such algorithm - MergeSort, and show that the above claim is true.

MergeSort proceeds in three steps:

• split the array into two parts;
• recursively sort each of the part;
• merge the sorted parts into a fully-sorted array;

For now, let us focus solely on the merge procedure presented in the previous lecture. There are many technical issues that prevent us from immediately computing an execution time:

• some instructions have complexity which depends on the machine and implementation:
  • malloc may depend on the standard C library implementation
  • we do not know how many execution steps control instructions (if, while) take
• it is difficult to combine instruction times into a single time function

We can address these issues if we introduce a notation which allows us:

• to express execution time in terms of asymptotic growth instead of a particular time function;
• to compute asymptotic growth notations for an algorithm, by combining notations for algorithm parts using algebraic operators (e.g. addition, multiplication);

Definition ($ \Theta$ (theta) notation):

Let $ g : \mathbb{R} \rightarrow \mathbb{R}$ . Then $ \Theta(g(n))$ is the class of functions:

$ \Theta(g(n)) = \left \{ f : \mathbb{R} \rightarrow \mathbb{R} \left\lvert \begin{array}{ll} \exists c_1,c_2 \in \mathbb{R}^+ \\ \exists n_0 \in \mathbb{N} \end{array}, \forall n \geq n_0, c_1g(n) \leq f(n) \leq c_2g(n) \right. \right \}$

Thus, $ \Theta(f(n))$ is the class of all functions with the same asymptotic growth as $ f(n)$ . We can easily observe that, for all continuous $ g,f \in Hom(\mathbb{R},\mathbb{R})$ such that $ g \in \Theta(f(n))$ , we have $ $$\lim_{n\to\infty} \frac{f(n)}{g(n)} = c$ , where $ c \neq 0$ .

There is an infinite number of classes $ \Theta(f(n))$ , one for each function $ f$ . However, if $ g(n) \in \Theta(f(n))$ , then $ \Theta(g(n)) = \Theta(f(n))$ .

It makes sense to consider classes which describe functions with inferior/superior asymptotic growth:

Definition ($ O, \Omega$ notations):

Let $ f : \mathbb{R} \rightarrow \mathbb{R}$ . Then:

$ O(g(n)) = \left\{ f:\mathbb{R} \rightarrow \mathbb{R} \left\lvert \begin{array}{ll} \exists c \in \mathbb{R}^+ \\ \exists n_0 \in \mathbb{N} \end{array}, \forall n \geq n_0, 0 \leq f(n) \leq c*g(n) \right. \right\}$

$ \Omega(g(n)) = \left\{ f:\mathbb{R} \rightarrow \mathbb{R} \left\lvert \begin{array}{ll} \exists c \in \mathbb{R}^+ \\ \exists n_0 \in \mathbb{N} \end{array}, \forall n \geq n_0, 0 \leq c*g(n) \leq f(n) \right. \right\}$

Note that $ g \in O(f(n)) \Longrightarrow O(g(n)) \subseteq O(f(n))$ , while $ g \in \Omega(f(n)) \Longrightarrow \Omega(g(n)) \subseteq \Omega(f(n))$ . Finally, $ \Omega(f(n)) \cap O(f(n)) = \Theta(f(n))$ . Each of the above propositions can be easily proved using the respective definitions of the notations.

$ O$ and $ \Omega$ offer relaxed bounds for asymptotic function growth. Thus, $ g \in O(f(n))$ should be read as: The function $ g$ grows asymptotically at most as much as $ f$ . It makes sense to also consider tight bounds:

Definition ($ o,\omega$ notations):

$ o(g(n)) = \left\{ f:\mathbb{R} \rightarrow \mathbb{R} \left\lvert \begin{array}{ll} \forall c \in \mathbb{R}^+ \\ \exists n_0 \in \mathbb{N} \end{array}, \forall n \geq n_0, 0 \leq f(n) \leq c*g(n) \right. \right\}$

$ \omega(g(n)) = \left\{ f:\mathbb{R} \rightarrow \mathbb{R} \left\lvert \begin{array}{ll} \forall c \in \mathbb{R}^+ \\ \exists n_0 \in \mathbb{N} \end{array}, \forall n \geq n_0, 0 \leq c*g(n) \leq f(n) \right. \right\}$

Thus, $ g \in o(f(n))$ should be read: $ g$ grows asymptotically strictly less than $ f$ . We have $ o(f(n)) \cap \omega(f(n)) = \emptyset$ , $ O(f(n)) \cap \Omega(f(n)) = \Theta(f(n))$ .

If $ f(n) \in \Omega(n^2)$ and $ g(n) \in O(n^3)$ then $ \displaystyle \frac{f(n)}{g(n)} \in \ldots$

If $ f(n) \in o(n^2)$ and $ g(n) \in \Theta(n^3)$ then $ f(n) \cdot g(n) \in \ldots$

If $ f(n) \in \Theta(n^3)$ and $ g(n) \in o(n^2)$ then $ \displaystyle \frac{f(n)}{g(n)} \in \ldots$

Asymptotic notations are defined over functions of reals (i.e. $ \mathbb{H}om(\mathbb{R},\mathbb{R})$ ) while execution times are functions over naturals (i.e. $ \mathbb{H}om(\mathbb{N},\mathbb{N})$ ). This may cause confusion when analysing notations. For instance: if $ f(n)\in \Theta(n^3)$ and $ g(n) \in o(n^2)$ then $ \frac{f(n)}{g(n)}\in o(n)$ only if $ f,g$ are execution times. This does not hold for functions over reals.

In general, when working with asymptotic notations, we must pay attention to the particular context. At the lecture, unless otherwise stated, we will assume functions are defined over reals.

Prove or disprove the following implications:

$ f(n)=O(\log n) \Rightarrow 2^{f(n)}=O(n)$

$ f(n)=O(n^2)$ and $ g(n)=O(n) \Rightarrow f(g(n))=O(n^3)$

$ f(n)=O(n)$ and $ g(n)=1+\sqrt{f(n)} \Rightarrow g(n)=\Omega(\log n)$

This section follows closely Lecture 2 from [1]. Quite often, asymptotic notations are used to refer to arbitrary functions having certain properties related to their order of growth. For instance, in:

$ \lceil f(x) \rceil = f(x) + O(1)$

applying “rounding” to $ f(x)$ , may be expressed as the original $ f(x)$ to which we add a function bounded by a constant. Similarly:

$ \displaystyle \frac{1}{1-x} =1+x+x^2+x^3+\Omega(x^4)$ , for $ -1 < x < 1$

The above notation allows us to “formally disregard” the terms from the expansion, by replacing them with an asymptotic notation which characterises their order of growth. One should make a distinction between the usage of asymptotic notations in arithmetic expressions, such as the ones previously illustrated, and equations. Consider the followind example:

$ f(x)=O(1/x)$

which should be read: there exists a function $ h \in O(1/x)$ such that $ f(x)=h(x)$ . Similarly:

$ f(x)=O(\log x)+O(1/x)$

should be read: there exists functions $ h \in O(1/x)$ and $ w \in O(\log x)$ such that $ f(x)=w(x)+h(x)$ . In equations such as:

$ O(x)=O(\log x)+O(1/x)$

the equality is not symmetric, and should be read from left to right: for any function $ f \in O(x)$ , there exist functions $ h \in O(1/x)$ and $ w \in O(\log x)$ such that $ f(x)=w(x)+h(x)$ . In order to avoid mistakes, the following algorithmic rule should be applied. When reading an equation of the form:

$ left = right$

• each occurrence of an asymptotic notation in the left should be replaced by an unuversally quantified function belonging to the corresponding class.
• each occurrence of an asymptotic notation in right should be replaced by an existentially quantified function from the corresponding class.