Dirichlet Processes

I taught myself Dirichlet processes and Hierarchical DPs in the spring of 2015 in order to understand nonparametric Bayesian models and related inference algorithms. I wrote a bunch of code and took a bunch of notes that I published here for the benefit of others trying to learn this material. I also contributed an implementation of Nonparametric Latent Dirichlet Allocation to the data microscopes project.

The posts below are best read in this order:

1. Dirichlet Distribution and Dirichlet Processes: A quick review of the Dirichlet Distribution and an introduction to the Dirichlet Process by analogy with the Dirichlet Distribution.
2. Sampling from a Hierarchical Dirichlet Process: Code demonstrating how you can sample from a Hierarchical Dirichlet Process without generating an infinite number of parameters first.
3. Nonparametric Latent Dirichlet Allocation: An alternative view of latent Dirichlet allocation using a Dirichlet process, and a demonstration of how it can be easily extended to a nonparametric model using a hierarchical Dirichlet process.
4. Fitting a Mixture Model with Gibbs Sampling: Derivation of a full Gibbs sampler for a finite mixture model with a uniform Dirichlet prior.
5. Collapsed Gibbs Sampling for Bayesian Mixture Models (with a Nonparametric Extension): Derivation of a collapsed Gibbs sampler for a finite mixture model, extended to the Dirichlet Process Mixture Model.
6. Notes on Gibbs Sampling in Hierarchical Dirichlet Process Models: Notes on applying the equations from the Hierarchical Dirichlet Process paper to nonparametric Latent Dirichlet Allocation.
7. Econtalk Topics: Nonparametric LDA in Practice: Applying nonparametric LDA to discover topics in Econtalk podcast transcripts.
8. Nomenclature of Dirichlet Processes: A discussion of the confusing use of the term “Dirichlet process” in the literature.
9. Sample from Antoniak Distribution with Python: Code for drawing samples from the distribution of tables created by a Chinese restaurant process after n patrons are seated.

HDP-LDA Implementations

• Gregor Heinrich’s ILDA: A Java-based implementation of the “Posterior Assignment by Direct Sampling” MCMC algorithm from Teh et al (2005). Includes hyperparameter sampling.
• Shuyo’s Implementation: Pure Python implementation of “Posterior sampling in the Chinese restaurant franchise” MCMC algorithm. Doesn’t include hyperparameter sampling.
• Teh’s Original Implementations: Matlab and C code for MCMC accompanying original paper.
• HCA: C implementation
• HDP-Faster: C++ implementation by Chong Wang using split-merge algorithm.
• Gensim: Python-based variational inference (following Chong Wang et al (2011)).
• bnpy: Python implementation of variational inference.