Transcript pptx
STICK-BREAKING CONSTRUCTIONS Patrick Dallaire June 10th, 2011 Outline Introduction of the Stick-Breaking process Outline Introduction of the Stick-Breaking process Presentation of fundamental representation Outline Introduction of the Stick-Breaking process Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process Outline Introduction of the Stick-Breaking process Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process Definition of the Beta process Outline Introduction of the Stick-Breaking process Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process Definition of the Beta process A Stick-Breaking construction of Beta process Outline Introduction of the Stick-Breaking process Presentation of fundamental representation The Dirichlet process The Pitman-Yor process The Indian buffet process Definition of the Beta process A Stick-Breaking construction of Beta process Conclusion and current work The Stick-Breaking process The Stick-Breaking process Assume a stick of unit length The Stick-Breaking process Assume a stick of unit length The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut The Stick-Breaking process Assume a stick of unit length At each iteration, a part of the remaining stick is broken by sampling the proportion to cut How should we sample these proportions? Beta random proportions Let be the proportion to cut at iteration Beta random proportions Let be the proportion to cut at iteration The remaining length can be expressed as Beta random proportions Let be the proportion to cut at iteration The remaining length can be expressed as Thus, the broken part is defined by Beta random proportions Let be the proportion to cut at iteration The remaining length can be expressed as Thus, the broken part is defined by We first consider the case where Beta distribution The Beta distribution is a density function on Parameters and control its shape The Dirichlet process The Dirichlet process Dirichlet processes are often used to produce infinite mixture models The Dirichlet process Dirichlet processes are often used to produce infinite mixture models Each observation belongs to one of the infinitely many components The Dirichlet process Dirichlet processes are often used to produce infinite mixture models Each observation belongs to one of the infinitely many components The model ensures that only a finite number of components have appreciable weight The Dirichlet process A Dirichlet process, , can be constructed according to a Stick-Breaking process Where mass at is the base distribution and is a unit Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo Construction demo The Pitman-Yor process The Pitman-Yor process A Pitman-Yor process, , can be constructed according to a Stick-Breaking process Where and Evolution of the Beta cuts The parameter controls the speed at which the Beta distribution changes Evolution of the Beta cuts The parameter controls the speed at which the Beta distribution changes The parameter determines initial shapes of the Beta distribution Evolution of the Beta cuts The parameter controls the speed at which the Beta distribution changes The parameter determines initial shapes of the Beta distribution When , there is no changes over time and its called a Dirichlet process Evolution of the Beta cuts The parameter controls the speed at which the Beta distribution changes The parameter determines initial shapes of the Beta distribution When , there is no changes over time and its called a Dirichlet process MATLAB DEMO The Indian Buffet process The Indian Buffet process The Indian Buffet process was initially used to represent latent features The Indian Buffet process The Indian Buffet process was initially used to represent latent features Observations are generated according to a set of unknown hidden features The Indian Buffet process The Indian Buffet process was initially used to represent latent features Observations are generated according to a set of unknown hidden features The model ensure that only a finite number of features have appreciable probability The Indian Buffet process Recall the basic Stick-Breaking process The Indian Buffet process Recall the basic Stick-Breaking process The Indian Buffet process Recall the basic Stick-Breaking process Here, we only consider the remaining parts The Indian Buffet process Recall the basic Stick-Breaking process Here, we only consider the remaining parts The Indian Buffet process Recall the basic Stick-Breaking process Here, we only consider the remaining parts Each value corresponds to a feature probability of appearance Summary Summary The Dirichlet process induces a probability over infinitely many classes Summary The Dirichlet process induces a probability over infinitely many classes This is the underlying de Finetti mixing distribution of the Chinese restaurant process De Finetti theorem It states that the distribution of any infinitely exchangeable sequence can be written where is the de Finetti mixing distribution Summary The Dirichlet process induces a probability over infinitely many classes This is the underlying de Finetti mixing distribution of the Chinese restaurant process The Indian Buffet process induces a probability over infinitely many features Summary The Dirichlet process induces a probability over infinitely many classes This is the underlying de Finetti mixing distribution of the Chinese restaurant process The Indian Buffet process induces a probability over infinitely many features Its underlying de Finetti mixing distribution is the Beta process The Beta process The Beta process This process Beta with Stick-Breaking The Beta distribution has a Stick-Breaking representation which allows to sample from Beta with Stick-Breaking The Beta distribution has a Stick-Breaking representation which allows to sample from The construction is Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking Beta with Stick-Breaking The Beta distribution has a Stick-Breaking representation which allows to sample from The construction is The Beta process A Beta process is defined as as , and is a Beta process Stick-Breaking the Beta process The Stick-Breaking construction of the Beta process is such that Stick-Breaking the Beta process Expending the first terms Conclusion We briefly described various Stick-Breaking constructions for Bayesian nonparametric priors These constructions help to understand the properties of each process It also unveils connections among existing priors The Stick-Breaking process might help to construct new priors Current work Applying a Stick-Breaking process to select the number of support points in a Gaussian process Defining a stochastic process for unbounded random directed acyclic graph Finding its underlying Stick-Breaking representation