Information Theory Methods in Physics

Spring 2021: PHYS 601 Methods of Theoretical Physics II

Tue-Thu 12:00-1:15 pm; Room PHYS 223 (First lecture is in person and the following lectures will be online using the Zoom application). The videos of the lectures will be recorded and submitted on YouYube. If you have signed up for the course, or planning to sign up please send me an email so that I can invite you to the SLACK channel.

Prerequisites: Basic familiarity with quantum mechanics and statistical physics. No previous knowledge of renormalization group is assumed.

Course description: In this course, we discuss two main applications of information theory (classical and quantum) in physics: Inference and Coarse-graining.
The course is divided into two parts:
Part I discusses some basic notions of information theory, probability and inference.
Part II discusses coarse-graining, scaling and renormalization group in physics.

The goal of the course is to highlight the information-theoretic origin of some tools and techniques used in statistical physics such as the Gibbs state, Monte Carlo simulations, coarse-graining, scaling and the renormalization group.

Part I: Classical Information, Probability and Inference

References: For part I of the course we will mostly follow lectures from David McKay’s textbook “Information theory, probability and learning algorithms” that can be found here for free.

Other recommended resources for the first part of the course are these video lectures by McKay.

Part II: Coarse-graining, Scaling, and Renormalization

References: For part II of the course we will mostly follow these lectures by John McGreevy.

Other useful resources:

Simon Dedeo’s lectures on renormalization with focus on the applications outside of physics.

Nigel Goldenfeld’s book “Lectures on Phase Transition and Renormalization Group”

John Cardy’s book “Scaling and renormalization in statistical physics”

David Tong’s lectures on “Statistical field theory”

We will use the application Slack to post course announcements, reports, discuss ideas and share interesting papers we have come across. If you are planning to attend the class please send me an email and I will add you to the course channel on Slack. All lectures are going to be recorded and posted on YouTube.


The students should read the relevant section of the lecture notes before coming to class and submit two questions on slack. Every lecture will have homework problems that will be assigned in class and have to be submitted on Slack up to a week after the lecture.


Click on the lecture to download the lecture note. The syllabus is subject to change as the course progresses.

  • Lecture 1: Introduction to Information Theory (reading: Section 1 of McKay’s book)
  • Lecture 2: Probability, Entropy and Inference (reading: Section 2 of McKay’s book)
  • Lecture 3: More about Inference (reading: Section 3 of McKay’s book)
  • Lecture 4: An Example Inference Task: Clustering (reading: Section 20 of McKay’s book)
  • Lecture 5: Exact Inference by Complete Enumeration (reading: Section 21 of McKay’s book)
  • Lecture 6: Maximum Likelihood and Clustering (reading: Section 22 of McKay’s book)
  • Lecture 7: Exact Marginalization (reading: Section 24 of McKay’s book)
  • Lecture 8: Laplace’s Method, Model Comparison and Occam’s Razor (reading: Sections 27 and 28 of McKay’s book)
  • Lecture 9: Monte Carlo Methods (reading: Section 29 of McKay’s book)
  • Lecture 10: Efficient Monte Carlo Methods (reading: Section 30 of McKay’s book)
  • Lecture 11: Ising Models (reading: Section 31 of McKay’s book)
  • Lecture 12: Exact Monte Carlo Sampling (reading: Section 32 of McKay’s book)
  • Lecture 13: Variational Methods (reading: Section 33 of McKay’s book)
  • Lecture 14: Independent Component Analysis (reading: Section 34 of McKay’s book)
  • Lecture 15: Coarse-graining

If you have any questions/comments or suggestions please do not hesitate to send me a message.

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close