MATH 978 - Advanced Topics in Communication Theory:
Information Theory for Systems with Memory
Instructor
-
Fady Alajaji
E-mail: fady@mast.queensu.ca
-
Textbook
-
Class Notes (there is no required textbook).
-
References
-
- R. B. Ash, Information Theory, Interscience, 1965.
- T. Berger, Rate Distortion Theory: A Mathematical Basis
for Data Compression, Prentice Hall, 1971.
- R. Blahut, Principles and Practice of Information Theory,
Addison-Wesley, 1987.
- T. M. Cover and J. A. Thomas, Elements of Information Theory,
Wiley, 1991.
- I. Csiszár and J. Körner, Information Theory:
Coding Theorems for Discrete Memoryless Systems, Academic Press, 1981.
- R. Gallager, Information Theory and Reliable Communication,
Wiley, 1968.
- R. M. Gray, Source Coding Theory,
Kluwer Academic, 1990.
- R. M. Gray, Entropy and Information Theory,
Springer-Verlag, 1990.
- T.-S. Han, Information Spectrum Methods in Information Theory,
Springer, 2003.
- R. J. McEliece, The Theory of Information and Coding,
Addison Wesley, 1977.
- A. J. Viterbi and J. K. Omura, Principles of Digital Communication
and Coding, Mc-Graw Hill, 1979.
-
Prerequisite
-
MATH 474/874 - Information Theory
-
Outline
-
This course is a follow up to the introductory course
on information theory (MATH 474/874). It will cover
advanced topics relating to
Shannon's
fundamental limits
of compression and transmission for communication
systems with memory. Specifically, the following
topics will be examined.
- Rate-Distortion Theory:
lossy fixed-length data
compression for memoryless sources;
scalar quantization and the Lloyd-Max algorithm;
rate-distortion function and its properties;
rate-distortion theorem for memoryless sources;
the lossy joint source-channel coding theorem
for memoryless source and channels, computation
of the capacity limit or the optimal performance
theoretically achievable (OPTA); rate-distortion
theorem for stationary ergodic sources.
- Channel Reliability Function:
discrete memoryless
channels, channel coding and error probability,
reliability function for discrete memoryless channels
and channels with memory.
- Source and Channel Coding for Arbitrary
Systems with Memory:
Spectral information measures;
arbitrary sources with memory;
fixed-length lossless source coding theorem for
arbitrary sources; information stable sources;
arbitrary channels with memory;
channel coding theorem for arbitrary channels;
information stable channels; classical lossless
joint source-channel coding theorem for information
stable systems.
- Feedback Capacity:
channels with output feedback,
capacity with feedback, discrete memoryless channels
and channels with memory.
- Models for Discrete Communication Channels with Memory:
burst noise channel models, finite-state channels, Gilbert-Elliott
channel, Polya contagion channel,
statistical properties and channel capacity.
-
Research Projects
-
The final grade will be determined solely on the basis of
the student's performance in a research project related
to the course material.
A detailed list of the projects is provided
here.
Students may work individually
or in pairs.
