% \lecture{date}{scribe}{course number}{course name}{instructor name}
\lecture{Jan.~20, 2009}{John Gubner}{ECE 729}{Information Theory}{John A. Gubner}

\section*{Lecture Outline}
\begin{itemize}
\item Course Organization
\item Course Introduction
\end{itemize}


\chapter{Introduction}


\section{Communication Systems}

Communication is the conveying of
a message (or its approximation) from one place,
called the \bdx{source location},
to another place, called the \bdx{destination} \cite{Shannon48}.
In order to accomplish this task, we
have available a \bdx{channel} that accepts inputs
$\bx$ at the source location,
and produces a related output $\by$ at the
destination;
\begin{figure}[htb]
\centering\includegraphics{figchannelBasic}
\end{figure}
the output $\by$ depends on
the input $\bx$, usually in some random way.
The first difficulty we have is that the message,
call it $U$, is usually not the same type of object
as the acceptable inputs to the channel.  For example,
the message may be a baseband voice waveform, but the channel
may accept only waveforms in a certain passband.
Or the channel may be a fiber-optic cable that accepts
only pulses of light.
So we usually need some sort of converter to transform
a message object into a channel-input object.  Depending
on the context, this converter may be called an \bdx{encoder},
a \bdx{modulator}, or a \bdx{transmitter}.
The second difficulty we have is that the channel output
$\by$ may not be the same type of object as the message
$U$ that is desired at the destination.  For this reason,
we need a deconverter that transforms the channel output $\by$
into an approximation of $U$, which we denote by $\widehat U$.
Depending on the context, the deconverter may be called a
\bdx{decoder}, a \bdx{demodulator}, or a \bdx{receiver}.
The concatenation of the converter, the channel, and the deconverter
constitutes a \bdx{communication system} as shown in Figure~\nr{figComSysBasic}.
\begin{figure}[htb]
\centering\includegraphics{figComSysBasic}
\caption{A communication system.}
\lb{figComSysBasic}
\end{figure}




\section{Digital Communication Systems}


In a {\bfseries {\itshape digital\/} communication system}\index{digital
communication system}\index{communication system!digital},
although there are many admissible values of the channel input
$\bx$, we intentionally
use only a finite number of them, say $N$.
This means that the converter in Figure~\nr{figComSysBasic}
can be broken down into two subsystems.\footnote{
Just because we can break a system into two subsystems
does not necessarily mean that it is a good idea to
do so.  We will say more about this shortly when we discuss
the {\bf Source-Channel Separation Theorem}\index{Source-Channel
Separation theorem}.
}
The first one is a
\bdx{source encoder} that takes a message and maps
it into one of the integers $1,\ldots,N$.  The second subsystem
is a \bdx{channel encoder}
that maps each of these integers into one of our
$N$ selected channel inputs, say $\bx_1,\ldots,\bx_N$.
The deconverter can be similarly broken down into
a \bdx{channel decoder} that takes channel outputs into the integers
$1,\ldots,N$ and a \bdx{source decoder} that takes these integers
into message values or their approximations.
\begin{figure}[htb]
\centering\includegraphics{figComSys}
\caption{A digital communication system.}
\lb{figComSys}
\end{figure}


Another consequence of using only $N$ channel inputs is that
if the message $U$ takes more than $N$ distinct values, we cannot recover $U$
in all cases.
This situation can occur whether message is discrete
or analog.
Source encoders for binary data are discussed in the next subsection.
As for analog data, for now we just mention that to
approximate real-valued messages, we use
\bdx{quantization}.
If the message is a continuous-time waveform or a spatially
continuous image, we sample it and then quantize the samples.
Sampling is justified by the \bdx{sampling
theorem} in the case of bandlimited data \cite{Shannon48,Shannon49}.
As a practical matter, all sources are approximately bandlimited ---
a nice discussion of this can be found in the first three pages
of Slepian~\cite{Slepian76} (easy reading --- I recommend it).




\subsection{Source Encoders for Binary Data}

Consider an information source that generates
an infinite stream of zeros and ones, which we denote by
$U_1, U_2, \ldots\,$.
Suppose we group the information symbols into blocks
of size $k$, e.g., $U \de (U_1,\ldots,U_k)$.
Then there are {\it at most\/} $2^k$ possible
values of $U$.  A \bdx{source encoder} is a function
that maps each possible $k$-tuple of bits into an
integer $i\in\{1,\ldots,N\}$, where $N \le 2^k$.
The {\bf rate}\index{rate!of a source encoder}
of a such a source encoder is
\bdm
R_s \de \frac{\log_2 N}{k},
\edm
measured in {\it bits per source symbol}.



\subsubsection*{Why Might We Use {\itshape N}
$\bm{<}$ {\upshape 2}$^{\mbox{\scriptsize\itshape k}}$?}


Suppose that $U_{2 i}=0$ for $i=1,2,\ldots\,$.
Then $(U_1,U_2,\ldots,U_k)$ can take only
$2^{k/2}$ possible values.  More generally,
not all length-$k$ blocks have equal probability.
Some may be so unlikely that we can ignore them
and still have a very low probability of error.
{\bf Shannon's Source Coding Theorem}\index{Shannon's
source coding theorem}
says that
this will be the case if $R_s$ greater than or equal
to the \bdx{entropy} $H$ of the source.


\subsection{Channel Codes}

A \bdx{channel encoder} is a function that maps each
$i \in \{1,\ldots,N\}$ into a vector $\bx_i$ in
a vector space $\sX$ of dimension $n$.  For example,
we might have $\sX=\{0,1\}^n$; i.e., the set
of $n$-tuples of bits.
In any case, if a source block $U=(U_1,\ldots,U_k)$
is mapped into the integer $i$, then the signal $\bx_i$
is transmitted over a channel, whose output $\by$
is determined by the transmitted signal $\bx_i$
as well as noise.  The \bdx{channel decoder} is a mapping that
assigns to each possible channel output $\by$, an element
in $\{1,\ldots,N\}$.
The {\bf rate}\index{rate!of a channel code}
of the channel code is
\bdm
R_c \de \frac{\log_2 N}{n},
\edm
measured in {\it bits per dimension} or {\it bits per channel use},
depending on the space $\sX$.  {\bf Shannon's Channel Coding
Theorem}\index{Shannon's channel coding theorem}
says that under suitable conditions, if $R_c$ is
less than or equal to the \bdx{capacity} $C$ of the channel,
then $\bP(i \ne \widehat i\ \,)$ can be made as small as we
like by increasing $n$.  In other words, the dashed box
in Figure~\nr{figComSys} can be made almost noiseless!

It is important to stress here that $C$ does not depend on $n$.
Since all we need is $(\log N)/n \le C$, as $n$ increases, we
can let $N$ increase as well!



\subsection{Source Decoder}


A \bdx{source decoder} is a mapping that assigns to
each element in $\{1,\ldots,N\}$, a unique binary $k$-tuple.



\subsection{Rate-Distortion Theory}


When the dashed box in Figure~\nr{figComSys} is almost
noiseless, what remains is the \bdx{rate-distortion} problem.
If $N$ is not large enough to code each message separately,
and this is always the case for analog data, how can we
make $\widehat U$ the best approximation of $U$?



\subsection{Source-Channel Separation}


The conditions $R_s \ge H$ and $R_c \le C$
together imply that
\bdm
k H \le \log_2 N \le n C.
\edm
If source symbols arrive at rate $\rho_s$ per second
and channel symbols are sent at rate $\rho_c$ symbols (or
dimensions) per second,
then during a time window of duration $T$,
$k=\rho_s T$ and $n=\rho_c T$.  Hence, the above
condition implies $\rho_s T H \le \rho_c T C$,
or $\rho_s H \le \rho_c C$.  If source symbols arrive
at the same rate as channel symbols (or dimensions) leave,
then we need $H \le C$.

Under suitable assumptions, there is {\it no\/} loss
in designing the source encoder and the channel encoder
separately.  This is known as the {\bf Source-Channel Separation
Theorem}\index{Source-Channel Separation theorem} \cite{CT06}.



\section{Course Outline}

As suggested by the foregoing discussion, key topics in the course
include the sampling theorem, quantization,
fixed-length source codes and Shannon's Source Coding Theorem along with
Shannon's Channel Coding Theorem.  In addition,
we will study the sampling theorem, quantization,
variable-length source codes, and
rate-distortion theory.  Various other
topics may be included as time permits.



\begin{thebibliography}{0}

\bibitem{CT06}
T. M. Cover and J. A. Thomas,
{\it Elements of Information Theory}, 2nd ed.
Hoboken, NJ: Wiley, 2006.
\\
\href{http://poincare.matf.bg.ac.rs/nastavno/vobul.html}{Link to chapters from 1st ed.}

\bibitem{Shannon48}
\href{http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html}%
{C. E. Shannon,
``A mathematical theory of communication,''
{\it Bell Syst.\ Tech.\ J.}, vol.~27,
pp.~379--423, 623--656, July, Oct.~1948.}

\bibitem{Shannon49}
\href{http://ieeexplore.ieee.org.ezproxy.library.wisc.edu/xpls/abs_all.jsp?isnumber=14381&arnumber=659497&count=12&index=9&tag=1}{C. E. Shannon,
``Communication in the presence of noise,''
{\it Proc.\ IRE}, vol.~37, no.~1, pp,~10-21,
Jan.~1949.  Reprinted in {\it Proc.\ IEEE},
vol.~86, no.~2, pp.~447--457, Feb.~1998.}

\bibitem{Slepian76}
\href{http://ieeexplore.ieee.org.ezproxy.library.wisc.edu/search/srchabstract.jsp?arnumber=1454379&isnumber=31239&punumber=5&k2dockey=1454379@ieeejrns&query=%28+%28%28on+bandwidth%29%3Cin%3Eti+%29+%3Cand%3E+%28%28slepian%29%3Cin%3Eau+%29+%29&pos=1&access=no}{D. Slepian,
``On bandwidth,''
{\it Proc.\ IEEE},
vol.~64, no.~3, pp.~292--300, Mar.~1976.}

\end{thebibliography}