%\documentstyle[10pt,twoside]{article}
%\documentstyle[twoside]{article}
\documentclass[twoside]{article}

\usepackage{epsfig}

\setlength{\oddsidemargin}{0 in}
\setlength{\evensidemargin}{0 in}
\setlength{\topmargin}{-0.6 in}
\setlength{\textwidth}{6.7 in}
\setlength{\textheight}{8.5 in}
\setlength{\headsep}{0.75 in}
\setlength{\parindent}{0 in}
\setlength{\parskip}{0.1 in}
\usepackage{amsmath,amssymb,enumerate,algorithm,ifthen,algorithmic}

% The following commands sets up the lecnum (lecture number)
% counter and make various numbering schemes work relative
% to the lecture number. Don't edit them!
%
\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}

\newcommand{\bx}{{\bf x}}
\newcommand{\bb}{{\bf b}}
\newcommand{\bc}{{\bf c}}
\newcommand{\bA}{{\bf A}}
\newcommand{\bO}{{\bf 0}}



\newtheorem{theorem}{Theorem} 
\newtheorem{lemma}{Lemma} 
\newtheorem{claim}{Claim} 
\newtheorem{proposition}{Proposition} 
\newtheorem{prob}{Problem} 
\newtheorem{corollary}{Corollary} 
\newtheorem{question}{Question} 
\newtheorem{conjecture}{Conjecture} 
\newtheorem{example}{Example} 
\newtheorem{definition}{Definition} 
\newtheorem{remarka}{Remark} 

\def\P{\mathop{\rm P}\nolimits}
\def\NP{\mathop{\rm NP}\nolimits}
\def\DTIME{\mathop{\rm DTIME}\nolimits}
\def\BPTIME{\mathop{\rm BPTIME}\nolimits}
\def\ZPTIME{\mathop{\rm ZPTIME}\nolimits}
\def\polylog{\mathop{\rm polylog}\nolimits}

\newenvironment{remark}{\begin{remarka}\rm}{\end{remarka}} 
\newenvironment{proof}{{\bf Proof.}}{\hfill\rule{2mm}{2mm}} 
\newenvironment{pproof}[1]{\noindent{\textbf{Proof of #1.}}}{\hfill\rule{2mm}{2mm}} 
\newcommand{\calI}{{\cal I}}
\newcommand{\calT}{{\cal T}}
\newcommand{\calP}{{\cal P}}
\newcommand{\opt}{\mbox{\sc opt}}
\newcommand{\OPT}{\mbox{\sc OPT}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\ZZ}{\mathbb{Z}}



%
% The following macro is used to generate the header.
%
\newcommand{\lecture}[5]{
   \pagestyle{myheadings}
   \thispagestyle{plain}
   \newpage
   \setcounter{lecnum}{#1}
   \setcounter{page}{1}
   \noindent
   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 6.28in { {\bf CMPUT 675: Approximation Algorithms
                        \hfill Fall 2014} }
       \vspace{4mm}
       \hbox to 6.28in { {\Large \hfill Lecture #1 (#2): #3 \hfill} }
       \vspace{2mm}
       \hbox to 6.28in { {\it Lecturer: #4 \hfill Scribe: #5} }
      \vspace{2mm}}
   }
   \end{center}
   \markboth{Lecture #1: #3}{Lecture #1: #3}
   \vspace*{4mm}
}

%
% Convention for citations is authors' initials followed by the year.
% For example, to cite a paper by Leighton and Maggs you would type
% \cite{LM89}, and to cite a paper by Strassen you would type \cite{S69}.
% (To avoid bibliography problems, for now we redefine the \cite command.)
%
\renewcommand{\cite}[1]{[#1]}

\input{epsf}

%Use this command for a figure; it puts a figure in wherever you want it.
%usage: \fig{NUMBER}{FIGURE-SIZE}{CAPTION}{FILENAME}
\newcommand{\fig}[4]{
			\vspace{0.2 in}
			\setlength{\epsfxsize}{#2}
			\centerline{\epsfbox{#4}}
			\begin{center}
			Figure \thelecnum.#1:~#3
			\end{center}
	}

% Use these for theorems, lemmas, proofs, etc.

% Some useful equation alignment commands, borrowed from TeX
\makeatletter
\def\eqalign#1{\,\vcenter{\openup\jot\m@th
  \ialign{\strut\hfil$\displaystyle{##}$&$\displaystyle{{}##}$\hfil
      \crcr#1\crcr}}\,}
\def\eqalignno#1{\displ@y \tabskip\@centering
  \halign to\displaywidth{\hfil$\displaystyle{##}$\tabskip\z@skip
    &$\displaystyle{{}##}$\hfil\tabskip\@centering
    &\llap{$##$}\tabskip\z@skip\crcr
    #1\crcr}}
\def\leqalignno#1{\displ@y \tabskip\@centering
  \halign to\displaywidth{\hfil$\displaystyle{##}$\tabskip\z@skip
    &$\displaystyle{{}##}$\hfil\tabskip\@centering
    &\kern-\displaywidth\rlap{$##$}\tabskip\displaywidth\crcr
    #1\crcr}}
\makeatother

% **** IF YOU WANT TO DEFINE ADDITIONAL MACROS FOR YOURSELF, PUT THEM HERE:

% NOW START EDITING
\begin{document}
%FILL IN THE RIGHT INFO.
%\lecture{**LECTURE-NUMBER**}{**DATE**}{**LECTURER**}{**SCRIBE**}
\lecture{9}{Sep 22}{Linear Programming}{Zachary Friggstad}{Zachary Friggstad}

% **** YOUR NOTES GO HERE:

\section{Linear Programming Introduction}

By now, you should realize the importance of efficiently computable {\em lower bounds} and {\em upper bounds} in approximation algorithms.
Often, an approximation algorithm is devised by finding a related structure that we can efficiently compute and describing how
to transform it into a feasible solution without inflating the cost too much.

Here are some examples we have seen:
\begin{itemize}
\item In \textsc{Vertex Cover}, we compute a maximal matching. Taking both endpoints of each of these edges is a 2-approximation.
\item In the \textsc{Traveling Salesman Problem}, we used two lower bounds in Christofides' algorithm: the cost of a minimum spanning tree
and twice the cost of a min-cost matching on any even-size subset of vertices.
\item In \textsc{Minimizing Makespan on Identical Parallel Machines}, one lower bound was the efficiently-computable solution to the scaled running times.
This was close to the true optimum essentially because we did not scale the running times by much.
\end{itemize}

There is certainly some art to this: we have to understand the problem structure well enough to identify efficiently computable structures that a) provide a bound on the optimum solution value
and b) can be transformed into a feasible solution with comparable cost.

Linear programming is a versatile tool for obtaining such lower bounds. At a high level, we use linear programs (LPs) by first modelling the problem at hand by an integer program
and then dropping the integrality requirements to obtain a linear program. Since this is a relaxation of the original integer programming model, the optimum linear programming solution
immediately yields a lower or upper bound (depending on whether we are minimizing or maximizing the objective). The upshot is that we can efficiently solve linear programs in polynomial time.
In general, the variables in an optimum LP solution will be fractional and the challenge is to round them to feasible integer solutions while not losing much in the objective.

\subsection{Intro by Example: \textsc{Minimum Weight Vertex Cover}}\label{sec:vc}
Suppose $G = (V,E)$ is a graph and each $v \in V$ has a {\em cost} $c(v) \geq 0$. The goal is to find a vertex cover $U \subseteq V$ with minimum {\em cost}, not minimum size.
This captures the classic \textsc{Vertex Cover} problem from the first lecture by using $c(v) = 1$ for each vertex $v$.

Here is an integer programming formulation of the problem. We introduce a variable $x_v$ for each $v \in V$. The idea is that setting $x_v = 1$ means we use $v$ in the vertex cover
and setting $x_v = 0$ means we do not use $v$ in the vertex cover.

\[
\begin{array}{rrcll}
{\rm minimize}: & \displaystyle \sum_{v \in V} c(v) \cdot x_v & & & \\~\\
{\rm subject~to}: & x_u + x_v & \geq & 1 & \text{for each edge } (u,v) \in E \\~\\
& x_v & \in  & \{0,1\} & \text{for each vertex } v \in V
\end{array}
\]

This is an integer program.  The last line says each $x_v$ value should be either 0 or 1 which, as mentioned, corresponds to excluding or including $v$ in the vertex cover.
The second line asserts that for each edge $(u,v) \in E$, at least one of $x_u$ or $x_v$ should be 1. This is equivalent to saying at least one of $u$ or $v$ should be in the vertex cover.
Finally, the first line says we should minimize the cost of the vertices $v$ whose $x_v$ value is set to 1. This is the same as minimizing the cost of the vertex cover.

Of course, this has not bought us much. Solving integer programs is still NP-hard. They key trick behind a vast number of approximation algorithms is to relax the integrality requirements of
the integer program. This results in a linear program that we call the {\em LP relaxation} of the integer program.

\[
\begin{array}{rrcll}
{\rm minimize}: & \displaystyle \sum_{v \in V} c(v) \cdot x_v & & & \\~\\
{\rm subject~to}: & x_u + x_v & \geq & 1 & \text{for each edge } (u,v) \in E \\~\\
& 0 \leq x_v & \leq & 1 & \text{for each vertex } v \in V
\end{array}
\]

We can solve this linear program in polynomial time. More on this later, but let us see what happens in this \textsc{Minimum Weight Vertex Cover Problem}.

Suppose the vector $\bx^*$ is an optimal solution to the linear programming relaxation. We have $\sum_{v \in V} c(v) \cdot x^*_v \leq OPT$ because the minimum weight
vertex cover corresponds naturally to a feasible solution to the linear program.

Let us form an integer solution $\bx'$ by rounding $\bx^*$. That is, set $x'_v = 1$ if $x^*_v \geq 1/2$, otherwise set $x'_v = 0$ if $x^*_v < 1/2$. Two things to observe:
\begin{itemize}
\item For every $v \in V$, $x'_v \leq 2 x^*_v$. Therefore, $\sum_{v \in V} c(v) \cdot x'_v \leq 2 \cdot \sum_{v \in V} c(v) \cdot x^*_v \leq 2 \cdot OPT$.
\item For every $(u,v) \in E$, since $x^*_u + x^*_v \geq 1$ (from the LP constraints), then one of $x^*_u$ or $x^*_v$ is $\geq 1/2$. Therefore, at least one of $x'_u$ or $x'_v$ is 1.
\end{itemize}
That is, $U = \{v \in V : x'_v = 1\}$ is a vertex cover of cost at most $2 \cdot OPT$. Simple!

This is more the exception than the norm. While LPs are used quite frequently in approximation algorithms it is hardly ever the case that rounding each variable off to the nearest
integer gives a feasible solution with cost close to the LP optimum.
Thus, the art is now in crafting a good rounding algorithm. In fact, in coming lectures we will see that coming up with a good LP relaxation can sometimes be
an interesting challenge by itself!

\section{General Theory}

Here we summarize the basic theory of the \textsc{Linear Programming} problem we need to get started. A few more things will be mentioned as the semester rolls on.
The claims about solving linear programs will not be prove here, but proofs of some of the structural results are in an appendix to these notes. These proofs are merely for the curious
as understanding them is not a requirement for future material.

A {\em linear program} (LP) in {\em standard form} is presented as
\[
\begin{array}{rrcl}
{\rm minimize}: & \bc^T \cdot \bx & & \\
{\rm subject~to}: & \bA \cdot \bx & \geq & \bb \\
& \bx & \geq & \bO
\end{array}
\]
A more succinct representation: $\min \{ \bc^T \cdot \bx : \bA \cdot \bx \geq \bb, \bx \geq \bO\}$.

Here, $\bx$ is an $n$-dimensional column vector of variables $(x_1, \ldots, x_n)^T$, $\bA = (a_{ji})$ is an $m \times n$ matrix, $\bb = (b_j)$
is an $m$-dimensional column vector, and $\bc = (c_i)$ is an $n$-dimensional column vector.
The matrices/vectors $\bA, \bb$ and $\bc$ are all known values (i.e. are not variables). The constraints $\bA \cdot \bx \geq \bb$ mean $(\bA \cdot \bx)_j \geq b_j$
for every $1 \leq j \leq m$. Similarly, $\bx \geq 0$ means all variables must take nonnegative values. We say that $n$ is the number of variables and $m$ is the number of constraints.

~

------------------------------------------------------------------------------------------------------------------------------------------------\\
{\bf Example 1}:\\
Consider the following linear program in standard form.

\[
\begin{array}{rrcl}
{\rm minimize}: & 3x_1 + x_2 & & \\~\\
{\rm subject~to}: & 8x_1 + x_2 & \geq & 8 \\
& 6x_1 + 4x_2 & \geq & 24 \\
& 4x_1 + 7x_2 & \geq & 28 \\~\\
& x_1, x_1 & \geq & 0
\end{array}
\]

In matrix/vector notation, the LP is $\max\{\bc^T\cdot\bx : \bA \cdot \bx \geq \bb, \bx \geq 0\}$ where
\[
\bx = \left(\begin{array}{r}x_1\\x_2\end{array}\right), ~~~
\bA = \left(\begin{array}{rr}8&1\\6&4\\4&7\end{array}\right), ~~~
\bb = \left(\begin{array}{r}8\\24\\28\end{array}\right), \text{ and }
\bc = \left(\begin{array}{r}3\\1\end{array}\right)
\]

Visually, the set of feasible solutions to this LP lie in the space above the lines in Figure \ref{fig:lp_1}.

\begin{figure}
 \centering
  \scalebox{0.8}{\psfig{figure=lp_1.pdf}}
  \caption{Sketch of the LP in the Example 1. The feasible solutions lie ``above'' all of the lines (by $\bA \cdot \bx \geq \bb$)
  and in upper-right quadrant (by $\bx \geq 0$). }\label{fig:lp_1}
\end{figure}

There is a unique optimum solution to this LP, namely $\bx^* = \left(\begin{array}{r}7/13\\48/13\end{array}\right)$
which has value $3 \cdot \frac{7}{13} + \frac{48}{13} = \frac{69}{13}$. Clearly it is feasible, to see it is optimum,
consider $\frac{17}{52}$ times the first constraint plus $\frac{5}{52}$ times the third constraint. This shows
$3x_1 + x_2 \geq \frac{69}{13}$ for any feasible solution, so the proposed solution must be optimal.

In general, there is always a way to prove optimality of an LP solution by considering some combination
of the constraints as in this example. We will discuss this later in the course when we talk about LP duality.

Finally, notice that in this example that the optimal solution is in one of the ``corners'' of the sketch
in Figure \ref{fig:lp_1}. More specifically, two of the constraints are satisfied by $\bx^*$ with equality. More on this below. \\
------------------------------------------------------------------------------------------------------------------------------------------------\\

~

The more general definition of a linear program is a bit more flexible.
We may want to maximize the objective function, we can use $=$ or $\geq$ instead of $\leq$ for some of the individual constraints $(\bA \cdot \bx)_j \geq b_j$,
and the nonnegativity constraints are not necessary.
It is a simple exercise to show that any LP according to this more general definition can
be transformed into an ``equivalent'' LP in standard form. For example, ${\rm maximize~} \bc^T \cdot \bx$ is equivalent to ${\rm minimize~} (-\bc)^T \cdot \bx$.

{\bf Assumption}: For the rest of this general LP introduction, we will assume the LPs are in standard form.
We will use many LPs that are not in this standard form in this course; everything discussed here generalizes in a simple way unless otherwise noted.

An assignment of values $\bx^*$ to $\bx$ is said to be {\em feasible} if $\bA \cdot \bx^* \geq \bb$ and $\bx^* \geq \bO$.
Linear programs can be classified in one of three ways.
\begin{itemize}
\item The linear program is {\em infeasible} if there is no feasible solution.
\item The linear program is {\em unbounded} if for all values $\gamma \in \mathbb R$ there is some feasible solution $\bx$ with $\bc^T \cdot x \leq \gamma$.
\item The linear program has a {\em optimum solution} if there is a feasible solution $\bx^*$ such that for all other feasible solutions $\bx'$,
$\bc^T \cdot \bx^* \leq \bc^T \cdot \bx'$. Call such a solution $\bx^*$ an {\em optimal solution}.
\end{itemize}

\begin{theorem}\label{thm:stronglp}
Every linear program is either infeasible, unbounded, or has an optimum solution.
\end{theorem}
Note that Theorem \ref{thm:stronglp} would not be true of we allowed strict inequalities, e.g. the one-dimensional problem $\min \{ x : x > 1\}$ cannot be classified in any of these three ways.

\subsection{Solving LPs}
While much of the theory behind linear programming works when the given values $\bA, \bb, \bc$ consist of arbitrary real numbers, we assume they are rational numbers as otherwise
the issue of even representing and performing arithmetic with the values becomes quite complicated.
One other parameter which we will rarely use beyond this section is the {\em bit complexity} of the linear program. This is the maximum number of bits used to represent any of the
$a_{ji},b_j,c_i$ values in the input (a single rational number $p/q$ is represented using $\sim \log_2 p + \log_2 q$ bits). Denote this quantity by $\Delta$.

To {\em solve} a \textsc{Linear Programming} instance means to correctly determine if it is infeasible, unbounded and has an optimal solution.
In the case there is an optimal solution, such an optimal solution should be produced.
\begin{theorem}\label{thm:solve}
An instance of \textsc{Linear Programming} can be solved in time ${\rm poly}(n, m, \Delta)$ with the additional guarantee that if an optimal solution exists,
then the optimal solution $\bx^*$ returned by the algorithm is rational and can be described in ${\rm poly}(n, \Delta)$ bits.
\end{theorem}
We will not discuss how to solve LPs, Theorem \ref{thm:solve} is really just
there to let you know that they can be solved in polynomial time and that an optimum solution $\bx^*$ can be stored using polynomially many bits.
Interior point methods such as those described in \cite{Y97} can be used to solve linear programs in polynomial time.

Note that the bit complexity of the optimum solution $\bx^*$ does not depend on the number of constraints; this will be important later
when we discuss how to solve LPs with exponentially many constraints that are described in an implicit manner.

\section{Integrality Gaps}

In this course, we often model an optimization problem as an integer program and then solve it's linear programming relaxation.
We hope to get good approximations by rounding an optimum LP solution to an integer value. If we find one with cost within an $\alpha$-factor
of the LP optimum then this is an $\alpha$-approximation.

One barrier to obtaining good approximations by comparing the cost of the solution versus the LP optimum is the following value.
\begin{definition}
Let $\mathcal P$ be a linear program. Let $OPT_{LP}$ be the optimum solution value to the $\mathcal P$ and let $OPT_{IP}$
be the optimum {\em integer} solution to $\mathcal P$. The {\em integrality gap} of $\mathcal P$ is the ratio $OPT_{IP} / OPT_{LP}$.
\end{definition}

Sometimes we say things like the integrality gap is at most $O(\log n)$ for an LP relaxation of a particular problem.
This means exactly what it sounds like. There are constants $c,n_0$ such that for any $n \geq n_0$ and any instance of the problem
with size $n$ (e.g. $n$ vertices, $n$ variables, $n$ items, etc.), the integrality gap of the LP relaxation for that instance is at most $c \cdot \log_2 n$.

\subsection{Vertex Cover Again}

Recall the LP relaxation for \textsc{Minimum Weight Vertex Cover} from Section \ref{sec:vc}.
We showed that any LP solution can be rounded to an integer solution with at most twice the cost of the LP solution.
This shows $OPT_{IP} \leq 2 \cdot OPT_{LP}$ for this LP meaning the integrality gap is at most 2.

Can we do better? Is the integrality gap less than two? Yes, but not by much. It is, in fact, possible to show the integrality gap of this LP relaxation
is at most $2 - \frac{2}{n}$ where $n = |V|$. This follows from an exercise question in Assignment 3.

This is essentially tight. Consider the complete graph $G = (V,E)$ with $c(v) = 1$ for each $v \in V$.
The solution $x_v = \frac{1}{2}$ is feasible for the LP and has cost $n/2$, but the optimum integer solution has cost $n-1$ as excluding any two vertices from the cover
would result in some edge not being covered. In this instance, the integrality gap is at least
\[ \frac{n-1}{n/2} = 2 - \frac{2}{n}. \]

So, if we want to find a better approximation for vertex cover we cannot rely solely on this LP for our lower bound.

\subsection{Maximum Independent Set}

What about \textsc{Maximum Independent Set}? In this problem, we have a graph $G = (V,E)$ and the goal is to find the largest $U \subseteq V$ such that no two $u,v \in U$ are adjacent.

A natural LP relaxation is the following
\[
\begin{array}{rrcll}
{\rm maximize}: & \displaystyle \sum_{v \in V}  x_v & & & \\~\\
{\rm subject~to}: & x_u + x_v & \leq & 1 & \text{for each edge } (u,v) \in E \\~\\
& x_v & \in [0,1] & \text{for each vertex } v \in V
\end{array}
\]
The idea is that in an integer solution, $x_v = 1$ means we include $v$ is included in the independent set and $x_v = 0$ means $v$ is excluded from the independent set. The objective function
is then the size of the independent set and the constraints ensure at most one vertex is in the independent set for each edge.

This seems quite close to the relaxation we used for \textsc{Minimum Weight Vertex Cover}. However, the integrality gap can be quite bad.

Consider the complete graph $G = (V,E)$ (again). Setting $x_v = \frac{1}{2}$ is a feasible solution with value $n/2$, yet the optimum integer solution is 1 (since any two vertices are adjacent).
Thus, the integrality gap is $\frac{1}{n/2} = \frac{2}{n}$ which is very bad!

\section{Geometry of LPs}
\begin{definition}
A subset $\mathcal P \subseteq \mathbb R$ is said to be {\em convex} if for any two $\bx, \bx' \in \mathcal P$ and any $0 \leq \lambda \leq $,
the point $\lambda \bx + (1-\lambda) \bx' \in \mathcal P$.
\end{definition}
That is, for any two points in $\mathcal P$ the line segment connecting these two points lies entirely in $\mathcal P$ as well.

Consider an LP $\min \{\bc^T \cdot \bx : \bA \cdot \bx \geq 0, \bx \geq 0 \}$.

\begin{lemma}
The set of feasible solutions to an LP is convex.
\end{lemma}
\begin{proof}
If $\bA \cdot \bx \geq \bb$ and $\bA \cdot \bx' \geq \bb$, then for any $0 \leq \lambda \leq 1$ we have
\[ \bA \cdot (\lambda \bx + (1-\lambda) \bx') = \lambda \bA \cdot \bx + (1-\lambda) \bA \cdot \bx' \geq \lambda \cdot \bb + (1-\lambda) \cdot \bb = \bb. \]
Similarly, $\lambda \bx + (1-\lambda) \bx' \geq \bO$.
\end{proof}

\begin{definition}
Let $\overline \bx$ be a feasible solution. Call $\overline \bx$ an {\em extreme point} if for any two feasible $\bx, \bx'$ and any $0 \leq \lambda \leq 1$ such that
$\overline \bx = \lambda \bx + (1-\lambda)\bx'$ it must be that either $\overline \bx = \bx$ or $\overline \bx = \bx'$. We say that $\overline \bx$ is a {\em convex combination}
of $\bx, \bx'$.
\end{definition}
That is, $\overline \bx$ is an extreme point if the only line segments that pass through $\overline \bx$ and are contained in the set of feasible LP solutions have $\overline \bx$ as an endpoint.

\begin{theorem} \label{thm:extreme_opt}
If an LP in standard form has an optimal solution, then there is an optimal solution that is an extreme point of that LP.
\end{theorem}
We emphasize that this is not necessarily true for LPs that are not in standard form. For example $\min \{x_1 + x_2 : x_1 + x_2 \geq 1\}$ has no extreme points yet
the optimum solution value is clearly 1. However, it is enough to assume that $x_i \geq 0$ is a constraint for every variable $x_i$ to ensure
that there is an extreme point optimum.

\begin{theorem} \label{thm:extreme_complex}
Any extreme point has bit complexity ${\rm poly}(n, \Delta)$.
\end{theorem}

\begin{definition}
Let $\overline \bx$ be a feasible LP solution. A constraint is said to be {\em tight} under $\overline \bx$ if it holds with equality (e.g. $(\bA \cdot \overline \bx)_j = b_j$ or $\overline x_i = 0$ for LPs in standard form).
\end{definition}

\begin{definition} \label{def:matrix}
Let $\overline \bx$ be a feasible LP solution. Let $\bA[\overline \bx]$ be the matrix consisting of rows from $\bA$ corresponding to constraints that are tight under $\bx$ as well
as rows of ${\bf I}_n$ corresponding to tight constraints under $\bx \geq \bO \equiv {\bf I_n} \cdot \bx \geq \bO$ (i.e. if $x_i = 0$ then include the $i$'th row of the identity matrix in $\bA[\overline \bx]$).
\end{definition}

An example illustrating this definition follows soon. First we present an alternative characterization of extreme points.

\begin{theorem} \label{thm:rank}
Let $\overline \bx$ be a feasible LP solution. Then $\overline \bx$ is an extreme point if and only if ${\rm rank}({\bA[\overline \bx]}) = n$ where $n$ is the number of variables.
\end{theorem}


------------------------------------------------------------------------------------------------------------------------------------------------\\
{\bf Example 2}:

Consider the following LP:
\[
\begin{array}{rrcl}
{\rm minimize}: & 3x_1 + 10x_2 + 5x_3 & & \\~\\
{\rm subject~to}
 & x_1 + 2x_2 & \geq & 8 \\
 & 5x_1 + 3x_2 + x_3 & \geq & 15 \\
 & 4x_2 + 5x_3 & \geq & 20 \\
 & 2x_1 + 8x_2 + 5x_3 & \geq & 36 \\~\\
 & x_1, x_2, x_3 & \geq & 0
\end{array}
\]

Below are various feasible solution $\bx'$ and their corresponding matrix of tight constraints $\bA[\bx']$.

{\bf Solution 1}\\
\[
\bx' = \left(\begin{array}{r}
22/39\\
145/39\\
40/3
\end{array}\right), ~~~
\bA[\bx'] = \left(\begin{array}{rrr}
1 & 2 & 0 \\
5 & 3 & 1 \\
0 & 4 & 5 \\
2 & 8 & 5
\end{array} \right)
\]
Every constraint apart from $\bx \geq 0$ is tight, so each of these rows appears in the matrix.
In fact, this is the optimal solution of the LP and has value 44. One can verify that ${\rm rank}(\bA[\bx']) = 3$
so this is also an extreme point.

~

{\bf Solution 2}\\
\[
\bx' = \left(\begin{array}{r}
8\\
0\\
4
\end{array}\right), ~~~
\bA[\bx'] = \left(\begin{array}{rrr}
1 & 2 & 0 \\
0 & 4 & 5 \\
2 & 8 & 5 \\
0 & 1 & 0
\end{array} \right)
\]
The first three rows of $\bA[\bx']$ are from the three tight constraints among the $\bA \cdot \bx \geq \bb$ constraints (namely, the 1st, 3rd, and 4th constraint).
Also, $x_2 \geq 0$ is a tight constraint under $\bx'$ so we include row 2 of ${\bf I}_n$ in $\bA[\bx']$.

~

{\bf Solution 3} \\
\[
\bx' = \left(\begin{array}{r}
0 \\
6 \\
0
\end{array}\right), ~~~
\bA[\bx'] = \left(\begin{array}{rrr}
1 & 0 & 0 \\
0 & 0 & 1
\end{array} \right)
\]
None of the $\bA \cdot \bx \geq \bb$ constraints are tight, only the two nonnegativity constraints $x_1 \geq 0, x_3 \geq 0$. Obviously ${\bf rank}(\bA[\bx']) = 2$.
By Theorem \ref{thm:rank}, we should be able to express $\bx'$ as the convex combination of two other feasible solutions.
We can express $\bx'$ as half of $(0,5,0)^T$ and half of $(0,7,0)^T$, both of which are feasible (in fact, $(0,5,0)^T$ is an extreme point).

~

{\bf Solution 4} \\
\[
\bx' = \left(\begin{array}{r}
167/39\\
145/78\\
98/39
\end{array}\right), ~~~
\bA[\bx'] = \left(\begin{array}{rrr}
1 & 2 & 0 \\
0 & 4 & 5 \\
2 & 8 & 5
\end{array} \right)
\]
Now, $\bA[\bx']$ has 3 rows, but ${\bf rank}(\bA[\bx']) = 2$. Letting $\bA' := \bA[\bx']$ (for notational simplicity), we see that ${\bf rank}(\bA[\bx']) < 3$ because
$2 \cdot \bA'_1 + \bA'_2 = \bA'_3$. By Theorem \ref{thm:rank}, this means we should be able to expression $\bx'$ as a convex combination of two other feasible solutions.
In fact, we can see that this $\bx'$ is exactly half of the $\bx$ values in {\bf Solution 1} and half of the $\bx$ values in {\bf Solution 2}.

------------------------------------------------------------------------------------------------------------------------------------------------\\

The following corollary is useful when the number of constraints is small with respect to the number of variables
(and also in more general settings we will see later).

\begin{corollary}
In any extreme point $\overline \bx$, the {\em support} $\{ i : \overline x_i > 0\}$ has size at most $m$.
\end{corollary}
\begin{proof}
Since ${\rm rank}(\bA[\overline \bx]) = n$ then there are at least $n$ tight constraints.  At least $n-m$ of these are of the form $x_i = 0$ for some $1 \leq i \leq n$.
\end{proof}

\section*{References}

\begin{itemize}
\item[Y97]
Y.~Ye,
{\em Interior Point Algorithms. Theory and Analysis.}
John Wiley \& Sons, 1997.

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{appendix}
\section{Some Proofs}

These can be skipped. They are only for the curious.

\begin{pproof}{Theorem \ref{thm:extreme_opt}}\\
Let $\bx^*$ be an optimal solution that has the most tight constraints among all optimal solutions. Let $\bA^* := \bA[\bx^*]$ be the matrix of these tight constraints
(c.f. Definition \ref{def:matrix}) and let $\bb^*$ denote the vector of $b_i$ (or 0) values corresponding to these tight constraints. Thus, $\bA^* \cdot \bx^* = \bb^*$.

We prove ${\rm rank}(\bA^*) = n$ by way of contradiction. Intuitively, this follows because if $\bA^*$ does not have full rank
then we can slide $\bx^*$ along a line until it ``hits'' another constraint (i.e. a new constraint becomes tight) while ensuring the cost $\bc^T \cdot \bx^*$
does not increase, contradicting our choice of $\bx^*$.

Suppose ${\rm rank}(\bA^*) < n$. Then there is some nonzero vector ${\bf z}$ such that $\bA^* \cdot {\bf z} = \bO$. Let $i'$ be such that $z_{i'} \neq 0$
and, by negating ${\bf z}$ if necessary, assume $z_{i'} > 0$.

Note that for every $\xi > 0$ that $\bA^* \cdot (\bx^* \pm \xi \cdot {\bf z}) = \bb^*$. Furthermore, for every constraint that is not tight under $\bf x^*$
we can choose $\xi$ to be small enough, yet still positive, such that both $\bx^* \pm \xi \cdot {\bf z}$ still satisfy this constraint (because there is some
slack in the constraint under $\bx^*$). So, there is in fact some $\xi > 0$ such that both $\bx^* \pm \xi \cdot {\bf z}$ are feasible.

Choose $\xi$ to be the {\em largest} value such that both $\bx^* \pm \xi \cdot {\bf z}$ are feasible. This is well-defined because $z_{i'} > 0$
and the constraint $x_{i'} \geq 0$ is a constraint in the LP (by assumption) so $x^*_{i'} - \xi \cdot z_{i'} \geq 0$ means $\xi$ cannot exceed $x^*_{i'}/z_{i'}$.
By choosing the largest possible $\xi$, we have that one of $\bx^* \pm \xi \cdot {\bf z}$ has more tight constraints than $\bx^*$.

Next, if $\bc^T \cdot {\bf z} \neq 0$ then one of $\bc^T \cdot (\bx^* \pm \xi \cdot {\bf z})$ is strictly less than $\bc^T \cdot \bx^*$ which contradicts optimality of $\bx^*$.
Thus, since $\bc^T \cdot {\bf z} = 0$ then both $\bc^T \cdot (\bx ^* \pm \xi \cdot {\bf z}) = \bc^T \cdot \bx^*$ meaning both are optimal solutions.

But then one of $\bx^* \pm \xi \cdot {\bf z}$ is an optimal solution with more tight constraints than $\bx^*$, a contradiction. Therefore,
${\rm rank}(\bA^*) = n$.
\end{pproof}

~

\begin{pproof}{Theorem \ref{thm:rank}}\\
The proof of Theorem \ref{thm:extreme_opt} shows that if ${\rm rank}(\bA^*[\overline \bx]) < n$ then $\overline \bx$ is not an extreme point (in the notation of the proof,
$\bx^*$ takes the place of $\overline \bx$ and $\bx^* \pm \xi {\bf z}$ are the endpoints of a line segment contained in the set of feasible solutions).

Now suppose $\overline \bx$ is not an extreme point and that $\overline \bx= \lambda \bx + (1-\lambda) \bx'$ where $\bx, \bx'$ are both feasible with $\overline \bx \neq \bx$ and
$\overline \bx \neq \bx'$. Since $\bA[\overline \bx] \cdot \overline \bx = \bb^*$ (where $\bb^*$ comes from the right hand side of the tight constraints) and since
$\bx, \bx'$ are both feasible, then it must be that $\bA[\overline \bx] \cdot \bx = \bA[\overline \bx] \cdot \bx' = \bb^*$ as well (i.e. if one of the constraints that was tight under $\overline \bx$
was slack under, say, $\bx$ then it would be violated under $\bx'$).

Thus, $\bA[\overline \bx] \cdot (\overline \bx - \bx) = \bO$. But $\overline \bx - \bx \neq \bO$ so ${\rm rank}(\bA[\overline \bx]) < n$.
\end{pproof}

~

\begin{pproof}{Theorem \ref{thm:extreme_complex}}\\
Let $\overline \bx$ be an extreme point, let $\bA^* := \bA[\overline \bx]$ and let $\bb^*$ denote the vector of $b_i$ (or 0) values corresponding to the tight
constraints of $\overline \bx$. Now, ${\bf rank}(\bA^*) = n$ by Theorem \ref{thm:rank}. By dropping some rows of $\bA^*$ and $\bb^*$ if necessary, assume
${\bf A^*}$ has exactly $n$ rows. That is, we are assuming that $\bA^*$ is an $n \times n$ matrix with full rank such that $\bA^* \cdot \overline \bx = \bb^*$.

Let $D$ be the product of all denominators of entries in $\bA^*$ and $\bb^*$. Replace $\bA^*$ and $\bb^*$ by $D \bA^*$ and $D \bb^*$ so that
the resulting matrices contain only integers. Since $D$ is the product of $n^2 + n$ integers, each having bit complexity at most $\Delta$, then
$D$ has bit complexity at most $O(n^2 \cdot \Delta)$. So, the bit complexity of the new matrix and vector has only increased by a ${\rm poly}(n, \Delta)$-factor.
We are now assuming $\bA^*$ and $\bb^*$ consist only of integer entries $z$ satisfying $|z| \leq 2^{\Delta'}$ for some $\Delta' = {\rm poly}(n, \Delta)$.

Let $\bA^*(i:\bb^*)$ denote the $n \times n$ matrix obtained by replacing column $i$ in $\bA^*$ by $\bb^*$. Cramer's rule states
\[ \overline x_i = \frac{\det(\bA^*(i:\bb^*))}{\det(\bA^*)}. \]
Since the determinant of an integer matrix is an integer (this immediately follows from the Leibniz formula for determinants),
we are left with proving that the determinant of an $n \times n$ integer matrix with entries having magnitude most $2^{\Delta'}$
is at most $2^{{\rm poly}(n, \Delta')}$ (the logarithm of this is the bit complexity).

Let $M$ be such a matrix. By Hadamard's determinant bound (to justify the first inequality below),
\[ \det(M) \leq \prod_{i=1}^n || M_i ||_2 = \prod_{i=1}^n \sqrt{\sum_{i=1}^n M_{ij}^2 } \leq \prod_{i=1}^n (n 2^{2\Delta'}) = (n 2^{2\Delta'})^n. \]
This last quantity can be crudely bounded by $2^{n^2 2 \Delta'}$. Thus, each numerator and denominator of each $\overline x_i$ value can be
described using $O(n^2 \Delta') = {\rm poly}(n, \Delta)$ bits which is exactly what we wanted to show.
\end{pproof}

~

{\bf Theorem 2}\\
We will not prove Theorem \ref{thm:solve}, but we simply note that the polynomial-time algorithms described in, say, \cite{Y97} will return an extreme point optimal solution if an optimal solution
exists (for LP in standard form).
By Theorem \ref{thm:extreme_complex}, the bit complexity of this solution is at most ${\rm poly}(n, \Delta)$.
The reduction from LPs in general form to LPs in standard form shows the bit complexity statement is true in general once we know it for LPs in standard form.

~

\begin{pproof}{Theorem \ref{thm:stronglp} (sketched)}\\
It suffices to prove the statement for LPs in standard form.
Suppose the LP is not infeasible and is not unbounded.
That is, there is a feasible LP solution but that there is also a lower bound $\ell$ such that every feasible solution $\overline \bx$ has $\bc^T \cdot \overline \bx \geq \ell$.


While Theorem \ref{thm:extreme_opt} assumed that the LP had an optimal solution, it is easy to see that the proof technique shows something stronger:
for every feasible solution $\overline \bx$ there is some extreme point solution $\overline \bx'$ such that $\bc^T \cdot \overline \bx' \leq \bc^T \cdot \overline \bx$
At a high level, if $\overline \bx$ is not extreme then we can slide it along some direction that does not increase the objective function value. If, in fact, the objective function
value strictly decreases then another constraint must eventually become tight as we slide because the LP is not unbounded. Otherwise, one of the two directions we can slide
along strictly decreases some $x_i$ value so we are, again, guaranteed that another constraint will become tight as we slide. In either case, if we still do not have an extreme point
after sliding until another constraint becomes tight, simply repeat this procedure.

Let $\bf x^*$ be an extreme point with the least objective function value among all extreme points. This is well defined because a) the feasibility of the LP, plus the above argument, shows that there is
in fact some extreme point and b) an extreme point is characterized by a collection of $n$ linearly independent tight constraints so there can only
be finitely many extreme points (at most ${m+n \choose n}$).

The claim is that $\bf x^*$ is in fact an optimal solution to the LP. Otherwise, we could employ the above
sketched argument to an even cheaper LP solution $\bf x'$ to find an even cheaper extreme point solution, contradicting the fact that $\bf x^*$ is the cheapest extreme point.

\end{pproof}
\end{appendix}
\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We will not cover the algorithm, but the proof of the rationality and bit complexity
is not so bad. Note that the bit complexity of $\bx^*_i$ in the statement does not depend on the number of constraints.

Before proving this first statement, there is one more required definition.

\begin{definition}
Let $\bx^*$ be a feasible solution to an LP $\min \{\bc^T \cdot \bx : \bA \cdot \bx \geq \bb, \bx \geq \bO\}$. A constraint $(\bA \cdot \bx)_j \geq b_j$
is said to be {\em tight} under $\bx^*$ if $(\bA \cdot \bx^*)_j = b_j$. Similarly, say that constraint $x_i \geq 0$ is {\em tight} if $x^*_i = 0$.
We construct the {\em matrix of tight constraints} $\bA^*$ for $\bx^*$ by first forming the matrix $\left(\begin{array}{r}\bA\\{\bf I}_n\end{array}\right)$
where the last $n$ rows correspond to the constraints $\bx \geq 0$, and then deleting every row of this matrix that does not
correspond to a tight constraint. The column vector $\bb^*$ corresponding to these tight constraints has the $b_j$ entries corresponding to the
rows of $\bA^*$ (using 0s for rows of $\bA^*$ coming from the identity matrix ${\bf I}_n$ for the tight constraints).
\end{definition}

EXAMPLE

\begin{lemma}\label{lem:tight}
If an LP has an optimal solution, then there is an optimal solution $\bx^*$ such that the matrix of tight constraints $\bA^*$ for $\bx^*$ has rank $n$
(where $n$ is the number of variables).
\end{lemma}

CORNER PICTURE REFERENCE

Finally, we can prove the bit complexity bound in Theorem \ref{thm:solve}.

\begin{pproof}{the first statement in Theorem \ref{thm:solve}}
Let $x^*$ be an optimal solution with associated matrix and vector of tight constraints $\bA^*, \bb^*$ where
${\rm rank}(\bA^*) = n$. By dropping some rows of $\bA^*$ and their associated entries in $\bb^*$, we may assume
$\bA^*$ is an $n \times n$ matrix of full rank.

For each $1 \leq i \leq n$, let $\bA^*(i:\bb^*)$ denote the matrix obtained by replacing column $i$ of $\bA^*$ by $\bb^*$.
Since $\bA^* \cdot \bx^* = \bb^*$, then Cramer's rule states 
\[ x^*_i = \frac{\det(\bA^*(i:\bb^*))}{\det(\bA^*)}. \]
Since $\bA^*$ and $\bA^*(i:\bb^*)$ are all matrices with rational entries, then their determinants are rational (recall
their determinant can be expressed as a sum of products of their entries) so $\bx^*$ has all rational entries.

To bound the bit complexity of an entry $x^*_i$, let $\kappa$ be the product of all denominators of all entries of $\bA^*$ and $\bb^*$.
So, $\kappa \cdot \bA^*$ and $\kappa \cdot \bb^*$ have only integer entries. If $D$ denotes the largest denominator, then $\kappa \leq D^{n^2 + n}$
so $\kappa$ has bit complexity $O(n^2 \log D)$ which is bounded by $O(n^2 \Delta)$.

We have $x^*_i = \frac{\det(\kappa \cdot \bA^*(i:\bb^*))}{\det(\kappa \cdot \bA^*)}$ (as $\kappa \cdot \bA^* \cdot \bx^* = \kappa \cdot \bb^*)$,
but now the expressed determinants are integers (since the matrices are integer matrices). So, to bound the bit complexity of $\bx^*$
it suffices to bound the size of the integer $\det(\kappa \cdot \bA^*)$ (and the values $\det(\kappa \cdot \bA(i:\bb^*))$, but this is similar).

Let $\Gamma$ be the largest entry of $k \cdot \bA^*$ and note that $\log_2 \Gamma$ is bounded by $O(\log \kappa \cdot \Delta) \leq {\rm poly}(n, \Delta)$.
The following inequalities bound $\det(\kappa \cdot \bA^*)$ where the first inequality is by Hadamard's determinant bound:
\[ \det(\kappa \cdot \bA^*) \leq \prod_{i=1}^n \sqrt{\sum_{j=1}^n (\kappa \cdot A^*_{i,j})^2} \leq \prod_{i=1}^n \sum_{j=1}^n \Gamma^2 \leq (n \cdot \Gamma^2)^n. \]
Therefore, the bit complexity of the denominator of $x^*_i$ is $O(\log_2 \det(\kappa \cdot \bA^*))$ which is bounded by $O(n \log n + n \log \Gamma) = {\rm poly}(n, \Delta)$.
As noted earlier, a similar bound holds for the numerator of $x^*_i$, so the bit complexity of $\bx^*$ is polynomial in $n$ and $\Delta$.

\end{pproof}
