Hubbry Logo
Minimum message lengthMinimum message lengthMain
Open search
Minimum message length
Community hub
Minimum message length
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Minimum message length
Minimum message length
Minimum message length
View on Wikipedia
from Wikipedia

Minimum message length (MML) is a Bayesian information-theoretic method for statistical model comparison and selection.[1] It provides a formal information theory restatement of Occam's Razor: even when models are equal in their measure of fit-accuracy to the observed data, the one generating the most concise explanation of data is more likely to be correct (where the explanation consists of the statement of the model, followed by the lossless encoding of the data using the stated model). MML was invented by Chris Wallace, first appearing in the seminal paper "An information measure for classification".[2] MML is intended not just as a theoretical construct, but as a technique that may be deployed in practice.[3] It differs from the related concept of Kolmogorov complexity in that it does not require use of a Turing-complete language to model data.[4]

Definition

[edit]

Shannon's A Mathematical Theory of Communication (1948) states that in an optimal code, the message length (in binary) of an event , , where has probability , is given by .

Bayes's theorem states that the probability of a (variable) hypothesis given fixed evidence is proportional to , which, by the definition of conditional probability, is equal to . We want the model (hypothesis) with the highest such posterior probability. Suppose we encode a message which represents (describes) both model and data jointly. Since , the most probable model will have the shortest such message. The message breaks into two parts: . The first part encodes the model itself. The second part contains information (e.g., values of parameters, or initial conditions, etc.) that, when processed by the model, outputs the observed data.

MML naturally and precisely trades model complexity for goodness of fit. A more complicated model takes longer to state (longer first part) but probably fits the data better (shorter second part). So, an MML metric won't choose a complicated model unless that model pays for itself.

Continuous-valued parameters

[edit]

One reason why a model might be longer would be simply because its various parameters are stated to greater precision, thus requiring transmission of more digits. Much of the power of MML derives from its handling of how accurately to state parameters in a model, and a variety of approximations that make this feasible in practice. This makes it possible to usefully compare, say, a model with many parameters imprecisely stated against a model with fewer parameters more accurately stated.

Key features of MML

[edit]
  • MML can be used to compare models of different structure. For example, its earliest application was in finding mixture models with the optimal number of classes. Adding extra classes to a mixture model will always allow the data to be fitted to greater accuracy, but according to MML this must be weighed against the extra bits required to encode the parameters defining those classes.
  • MML is a method of Bayesian model comparison. It gives every model a score.
  • MML is scale-invariant and statistically invariant. Unlike many Bayesian selection methods, MML doesn't care if you change from measuring length to volume or from Cartesian co-ordinates to polar co-ordinates.
  • MML is statistically consistent. For problems like the Neyman-Scott (1948) problem or factor analysis where the amount of data per parameter is bounded above, MML can estimate all parameters with statistical consistency.
  • MML accounts for the precision of measurement. It uses the Fisher information (in the Wallace-Freeman 1987 approximation, or other hyper-volumes in other approximations) to optimally discretize continuous parameters. Therefore the posterior is always a probability, not a probability density.
  • MML has been in use since 1968. MML coding schemes have been developed for several distributions, and many kinds of machine learners including unsupervised classification, decision trees and graphs, DNA sequences, Bayesian networks, neural networks (one-layer only so far), image compression, image and function segmentation, etc.

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Minimum message length

View on Grokipedia
from Grokipedia
The Minimum Message Length (MML) principle is a Bayesian information-theoretic approach to inductive , , and statistical estimation that formalizes by selecting the model which minimizes the total length of a two-part encoded describing both the model itself and the observed given that model. Developed by Australian computer scientists and David Boulton in the late 1960s, MML treats as a problem of compression, where the "message" is measured in bits (or nats) using an optimal , ensuring that simpler models are preferred unless they inadequately explain the data. The principle was first introduced in Wallace and Boulton's 1968 paper on , marking it as a foundational method in and statistics that predates similar ideas like minimum description length (MDL). At its core, MML divides the message into an assertion part, which encodes the model's structure and parameters (e.g., via a prior distribution π(θ) over parameters θ), and a detail part, which encodes the data y using the likelihood p(y|θ) under that model, with the total length approximated as I(θ) + I(y|θ) = -log π(θ) - log p(y|θ). This formulation is strictly Bayesian, requiring explicit priors to compute the and enabling automatic parameter estimation alongside model choice, unlike non-Bayesian alternatives. For continuous parameters, MML addresses discretization challenges by approximating the matrix to bound parameter precision, ensuring statistical consistency and invariance under reparameterization. MML differs from related criteria like MDL—developed by Jorma Rissanen in 1978—and the (AIC) by its full Bayesian integration of priors and focus on joint model-data encoding, making it asymptotically equivalent to the (BIC) for large samples but more precise for small datasets or complex structures. It has been applied across diverse fields, including clustering (e.g., via the program), decision tree induction, mixture modeling, and structure learning, often yielding superior performance in balancing model complexity and fit. Wallace's comprehensive 2005 book, Statistical and Inductive Inference by Minimum Message Length, solidified MML as a robust framework for computational Bayesianism, influencing modern despite its computational intensity.

Introduction

Definition

The Minimum Message Length (MML) principle formalizes Occam's Razor within information theory by selecting the statistical model or hypothesis that enables the most concise encoding of both the model itself and the observed data, thereby favoring simpler models unless added complexity substantially enhances the data's fit. This approach treats inference as a communication problem, where the goal is to transmit the hypothesis and data using the fewest bits possible, balancing model complexity against explanatory power. The core formula for the message length arises from Claude Shannon's foundational work on , which establishes that the optimal code length for an event with probability PP is −log⁡2P-\log_2 P bits, representing the uncertainty or surprisal of the event. Applying this to inductive inference, the total message length for a HH (or model) and EE (or data) is given by Length(H∧E)=−log⁡2P(H)−log⁡2P(E∣H),\text{Length}(H \wedge E) = -\log_2 P(H) - \log_2 P(E \mid H), which is the negative base-2 logarithm of the joint probability P(H,E)P(H, E). The first term, −log⁡2P(H)-\log_2 P(H), quantifies the prior length needed to specify the from a distribution over possible models, penalizing overly complex or improbable hypotheses. The second term, −log⁡2P(E∣H)-\log_2 P(E \mid H), measures the additional length required to encode the data using the hypothesis as a compression scheme, rewarding models that assign high likelihood to the observed . To illustrate, consider a of 20 coin tosses yielding 10 heads and 10 tails. Under a simple model assuming a (p=0.5p = 0.5), the prior length is short if the prior favors basic hypotheses (e.g., near 1 bit under a uniform prior over discrete probabilities), and the encoding length is approximately 20 bits, reflecting the binomial entropy 20×h(0.5)≈2020 \times h(0.5) \approx 20 bits, for a total of about 21 bits. In contrast, a more complex model estimating p=0.5p = 0.5 with high precision (e.g., to several places) might reduce the length slightly but increases the prior length substantially (e.g., by 10+ bits to specify the precise value), leading to a longer total message and thus being disfavored unless the strongly demands such detail. This demonstrates how MML prefers parsimonious models that adequately explain the without unnecessary elaboration.

Historical Development

The Minimum Message Length (MML) principle was invented by in collaboration with David Boulton, with its foundational ideas emerging around 1968 during Wallace's tenure as Foundation Chair of (later ) at in . The initial motivation stemmed from addressing problems in amid the burgeoning field of , where traditional methods struggled with and inductive inference. This work built on earlier information-theoretic concepts, such as Ray Solomonoff's 1964 theory of inductive inference, providing a practical, Bayesian implementation for statistical modeling. The first formal publication appeared in 1968 as "An Information Measure for Classification" by Wallace and Boulton in The Computer Journal, deriving a measure of classification goodness based on the of an encoded describing the and model. Subsequent early developments included refinements in the , such as Boulton's 1975 PhD thesis on mixture modeling applications. A key milestone came in 1987 with the Wallace-Freeman , introduced in "Estimation and by Compact Coding" in the Journal of the Royal Statistical Society: Series B, which addressed coding for continuous parameters using to enable more tractable computations. Wallace's comprehensive synthesis culminated in his 2005 book, Statistical and Inductive by Minimum (Springer, ISBN 978-0-387-23795-4), which posthumously consolidated decades of theoretical and applied advancements. Following Wallace's death in 2004, MML continued to evolve through extensions by collaborators, including integrations into Bayesian modeling frameworks and open-source tools up to 2025. Notable post-2005 contributions include the 2011 exploration of MML in hybrid Bayesian networks by Dowe et al., and software implementations like , an MML-based system for clustering and mixture modeling developed at . Recent works, such as the 2021 MML inference for censored exponential data, the 2022 introductory manuscript by Dowe, and 2025 applications to learning logical rules from noisy data, have further refined applications in statistical estimation and , emphasizing computational efficiency.

Theoretical Foundations

Information-Theoretic Basis

The Minimum Message Length (MML) principle draws its foundational roots from Claude Shannon's , established in 1948, which posits that the minimal length of a encoding a source's output approximates the in that output, as quantified by its . In this framework, the H(X)=−∑p(x)log⁡2p(x)H(X) = -\sum p(x) \log_2 p(x) represents the average number of bits required for of data from a probabilistic source, providing a lower bound on achievable lengths for efficient encoding. MML extends this concept to a universal coding scheme for , where both the ( HH) and the data ( EE) are jointly encoded using prefix-free codes to guarantee unique decodability. These codes adhere to the Kraft inequality, ∑2−li≤1\sum 2^{-l_i} \leq 1, where lil_i are the codeword lengths, ensuring that no codeword is a prefix of another and allowing instantaneous decoding without . This structure enables MML to treat as an over compressible descriptions of HH and EE, minimizing the total encoded length while approximating the source coding limits for arbitrary data distributions. A key theoretical link exists between MML and , where the latter defines the shortest program length needed to describe an object on a . MML approximates this algorithmic ideal by employing probabilistic priors over models, rendering the approach computationally feasible for practical , unlike the uncomputable nature of exact . The core derivation of message length in MML stems from the joint probability of the hypothesis and , expressed as the negative log-probability −log⁡2P(H∧E)-\log_2 P(H \wedge E), which directly quantifies the bits required for their lossless transmission under an optimal . In the limit of large data volumes, MML achieves asymptotic equivalence to , converging to the as the probabilistic approximations tighten and the data dominates the total description length.

Relation to Bayesian Inference

The Minimum Message Length (MML) principle serves as a criterion for and hypothesis testing, where the total message length decomposes into two components: the prior length, which is −log⁡2P(H)-\log_2 P(H) and encodes the of the HH through a prior distribution, and the likelihood term −log⁡2P(E∣H)-\log_2 P(E|H), which quantifies how well the explains the EE. This decomposition directly mirrors the Bayesian formulation, balancing model against fit in a probabilistic encoding framework. Minimizing the message length is mathematically equivalent to maximizing the posterior probability P(H∣E)P(H|E) when using uniform reference priors, such as Jeffreys priors, which provide an objective basis for non-informative priors in Bayesian inference. In this interpretation, the MML approach approximates the negative log-posterior up to a constant term independent of the hypothesis, ensuring that shorter messages correspond to higher posterior probabilities. Unlike plug-in likelihood methods, such as maximum likelihood estimation, MML explicitly accounts for parameter uncertainty by incorporating the variability in parameter estimates into the encoding length, leading to more robust inferences that penalize overly precise but uncertain specifications. From the perspective of inductive inference, C.S. Wallace viewed MML as a practical of Bayesian induction, particularly effective for comparing non-nested models where traditional Bayesian methods may struggle due to incommensurable parameter spaces. This alignment enables MML to approximate posterior odds ratios directly through differences in message lengths, as given by the relation: log⁡(P(H1∣E)P(H2∣E))≈Length(H2∧E)−Length(H1∧E)\log \left( \frac{P(H_1|E)}{P(H_2|E)} \right) \approx \text{Length}(H_2 \wedge E) - \text{Length}(H_1 \wedge E) where the logarithm is base-2, reflecting the bit-based encoding, and the approximation holds under the coding theorem linking probabilities to code lengths. This formulation facilitates hypothesis testing by quantifying the evidential support for one model over another in terms of .

Parameter Estimation in MML

Discrete Parameters

In the Minimum Message Length (MML) framework, discrete parameters are handled through exact encoding schemes that leverage the finite nature of the parameter , allowing for precise without the approximations required for continuous parameters. Hypotheses with discrete parameters, such as the selection of a from a or the assignment of categories in a task, are encoded using combinatorial codes that reflect the structure of the possible configurations. For instance, when selecting kk items from nn possibilities, the code length corresponds to log⁡2(nk)\log_2 \binom{n}{k} bits under a uniform prior, capturing the needed to specify the among the available options. The message length component attributable to the discrete hypothesis HH is given by −log⁡2P(H)-\log_2 P(H), where P(H)P(H) is the assigned to the , often assuming a uniform distribution over the finite space or a multinomial prior for multi-state scenarios. This term quantifies the bits required to transmit the values, ensuring the encoding is optimal in the information-theoretic sense by matching the of the prior distribution. For models involving multiple discrete choices, such as category assignments, the total length sums these contributions, enabling direct minimization over the discrete possibilities. A representative example arises in mixture models where the number of components kk is discrete and finite; here, the encoding includes the length to specify the partition of points into these components, typically using a multinomial code over the knk^n possible assignments for nn points, adjusted by a prior that favors balanced partitions to reduce . Computation proceeds exactly by evaluating and summing −log⁡2P(Hi)-\log_2 P(H_i) over all relevant discrete hypotheses HiH_i, or via in small spaces, yielding the minimum message length without approximations. This approach is computationally simpler than for continuous parameters, as it avoids approximations and relies solely on discrete summations. Early applications of MML to discrete parameters, such as taxonomic , demonstrated this simplicity by minimizing message length over discrete class assignments and numbers, as detailed in Wallace and Boulton's seminal work on measures for .

Continuous-Valued Parameters

Encoding continuous-valued parameters in the Minimum Message Length (MML) framework presents a fundamental challenge due to the infinite possibilities in continuous spaces, which precludes exact combinatorial encoding. Instead, code lengths are approximated by integrating over probability densities to capture the uncertainty in parameter estimates, ensuring the total message remains finite and decodable. The seminal Wallace-Freeman (1987) approximation resolves this by tying parameter precision to the inverse square root of the Fisher information matrix, yielding an efficient encoding strategy. This leads to a message length for the parameters of approximately d2log⁡2n+12log⁡2det⁡(I(θ))\frac{d}{2} \log_2 n + \frac{1}{2} \log_2 \det(I(\theta)), where dd is the dimensionality of the parameter space, nn is the number of observations, and I(θ)I(\theta) denotes the per-observation Fisher information matrix evaluated at the parameter value θ\theta. The derivation stems from optimal quantization of parameters centered on the maximum likelihood estimate θ^\hat{\theta}, exploiting the asymptotic local normality of the likelihood function to define the uncertainty volume in the parameter space. Incorporating the hypothesis prior and data likelihood, the overall continuous message length is approximated as −log⁡2π(θ^)−log⁡2p(E∣θ^)+k2log⁡2n+12log⁡2det⁡(I(θ^))+C,-\log_2 \pi(\hat{\theta}) - \log_2 p(E \mid \hat{\theta}) + \frac{k}{2} \log_2 n + \frac{1}{2} \log_2 \det(I(\hat{\theta})) + C, where kk is the number of free parameters, π(θ^)\pi(\hat{\theta}) is the prior density on the parameters, p(E∣θ^)p(E \mid \hat{\theta}) is the maximized likelihood of the EE, I(θ^)I(\hat{\theta}) is the per-observation matrix, and CC is a constant that may include terms such as k2log⁡2(2π)\frac{k}{2} \log_2 (2\pi) depending on the specific prior and coding scheme. This formulation naturally extends to multi-dimensional parameters by leveraging the determinant of the information matrix to account for correlations across dimensions. It has been practically implemented in Wallace's software for tasks involving continuous data.

Properties and Features

Key Advantages

The Minimum Message Length (MML) principle promotes parsimony by naturally penalizing model complexity through the length required to encode the prior distribution of parameters, which favors simpler models without arbitrary tuning parameters. This built-in penalty enables flexible comparisons across non-nested models, such as versus , where traditional likelihood-based methods may struggle due to incommensurable parameter spaces. A key strength of MML is its , as the total message length remains unchanged under monotonic transformations of the data or parameters, ensuring consistent regardless of units or scaling—unlike some likelihood-based criteria that can be sensitive to such changes. This property arises from the information-theoretic encoding that treats models equivalently under reparameterizations. MML effectively mitigates by incorporating uncertainty in parameter estimates through a Bayesian prior that quantifies the information needed to specify parameters precisely, often resulting in sparser models, particularly in high-dimensional settings where maximum likelihood tends to overfit. This leads to more robust generalizations by balancing fit and without explicit regularization terms. In small-sample scenarios, MML has demonstrated superior performance to , as evidenced in Wallace and Boulton's 1968 work on , where the MML-derived measure produced more accurate groupings with limited data by accounting for encoding efficiency. Empirically, MML exhibits better predictive performance in simulations involving models compared to methods assuming equal priors, as it optimally allocates prior probabilities to components, leading to improved cluster recovery and out-of-sample accuracy in Gaussian mixtures.

Statistical Consistency

The minimum message length (MML) principle exhibits statistical consistency in , meaning that under mild conditions—such as identifiable models, a growing sample size nn, and appropriate prior distributions—the probability that MML selects the true approaches 1 as n→∞n \to \infty. This property ensures that MML reliably identifies the correct model asymptotically, distinguishing it from inconsistent criteria that may persistently favor overparameterized alternatives. A sketch of the proof relies on the decomposition of the MML score into the data compression term and the prior term. For the true HtrueH_{\text{true}}, the negative log-likelihood term −log⁡2P(E∣Htrue)-\log_2 P(E \mid H_{\text{true}}) converges to its by the , providing an efficient encoding of the evidence EE. In contrast, false hypotheses incur an excess message length due to model-data mismatch, which grows linearly with nn because the likelihood under a misspecified model deviates systematically from the true distribution. The prior term, encoding the complexity, becomes negligible relative to the data term as nn increases, ensuring that the total MML length for HtrueH_{\text{true}} is asymptotically minimal. Unlike non-consistent criteria like the (AIC), which tend to overfit and do not select the true model asymptotically, MML achieves consistent under its conditions, with a penalty informed by the matrix that better accounts for parameter uncertainty and model dimensionality. Wallace (2005) provides a detailed demonstration of this consistency for both nested and non-nested models, leveraging Cramér-Rao bounds to quantify the information-theoretic penalties for misspecification and establish the required identifiability conditions. However, MML's consistency does not hold in all cases; for example, in the Neyman-Scott problem involving incidental parameters, MML estimators have been shown to be inconsistent even with natural prior choices.

Applications

In Model Selection

In , the Minimum Message Length (MML) principle is applied to choose among competing statistical models by identifying the one that allows the shortest encoding of the observed data plus the model itself. This involves computing the total length for each candidate model and selecting the minimizer, where the message length quantifies the bits required to transmit the model parameters and the data under that model. For two competing models M1M_1 and M2M_2, the decision rule compares ΔLength=Length(M1)−Length(M2)\Delta \text{Length} = \text{Length}(M_1) - \text{Length}(M_2); if ΔLength>0\Delta \text{Length} > 0, then M2M_2 is preferred as it yields a shorter overall . A prominent application of MML in is in , where it aids in determining the optimal degree for fitting data. For instance, in univariate , MML compares message lengths across models of varying orders, such as linear (degree 1) versus quadratic (degree 2), to balance goodness-of-fit against model complexity and avoid noisy data. Empirical evaluations demonstrate that MML outperforms classical criteria like AIC and BIC in selecting the true degree, particularly with small sample sizes or high levels, by leveraging approximations for continuous parameters in the encoding process. This approach has been extended to for variable selection, where MML promotes sparsity by favoring models with fewer parameters that still adequately explain the data, showing superior performance in simulations from the 1990s. In clustering tasks, MML determines the optimal number of clusters in models like Gaussian mixtures by encoding cluster assignments and parameters to minimize the total message length. Wallace and Boulton (1968) introduced this in their seminal work on , applying MML to partition data into multinomial or Gaussian components, where the shortest message corresponds to the most plausible clustering structure. For example, in a Gaussian mixture, the message length includes the cost of specifying means, covariances, and mixing proportions alongside the data likelihood, enabling automatic selection of the number of components without predefined hyperparameters.

In Machine Learning and Data Mining

In , the minimum message length (MML) principle has been applied to decision tree induction and rule learning by encoding the tree structure and leaf predictions to guide and . This approach favors compact trees that minimize the total message length required to describe both the model and the , thereby avoiding while maintaining predictive accuracy. For instance, extensions to algorithms like C5.0 incorporate MML-based , where the cost of encoding splits and predictions is balanced against improvements in fit, leading to smaller, more generalizable trees compared to traditional methods. MML also plays a key role in structure learning for Bayesian networks, particularly in hybrid models that combine discrete and continuous variables. By minimizing the joint message length for nodes, edges, and conditional dependencies, MML enables the inference of network topologies from data using techniques like Markov chain Monte Carlo sampling. This results in networks that efficiently capture local structures, such as decision trees within conditional probability distributions, outperforming alternatives like minimum description length (MDL) or Bayesian Dirichlet equivalent (BDe) metrics in scenarios with limited data. In , MML supports unsupervised tasks such as and through clustering algorithms that identify outliers as data points poorly encoded by the best-fitting . Post-2005 applications have integrated MML into tools for multivariate finite s, where anomalies are flagged based on deviations from cluster densities, enhancing detection in high-dimensional datasets like intrusion detection systems. Recent extensions include its use in species distribution modeling for ecological data analysis (as of 2024) and for learning rules from noisy data (as of 2025). MML implementations appear in software packages, including Wallace's program from the 1980s for ing and clustering, as well as modern adaptations such as the PyMML package in Python and the GMKMcharlie package in for scalable Gaussian ing. A notable example is in high-dimensional , where MML identifies relevant genes by selecting subsets that yield the shortest encoding of sequencing data under spatial autoregressive models. This approach, combined with regularization techniques like adaptive , prioritizes informative genetic markers for risk prediction, demonstrating superior performance in simulations with thousands of features.

Comparisons with Other Criteria

Versus Minimum Description Length (MDL)

The Minimum Message Length (MML) principle and the Minimum Description Length (MDL) principle share fundamental similarities as information-theoretic approaches to and . Both seek to minimize the total length required to encode a model and the observed data, thereby approximating the of the data in a practical manner. This shared goal promotes parsimonious models that compress the data effectively while avoiding . Despite these commonalities, MML and MDL diverge in their encoding strategies and theoretical foundations. MDL, introduced by Rissanen in , typically employs a two-part : first encoding the model parameters, followed by encoding the conditional on those parameters, often relying on plug-in maximum likelihood estimates. In contrast, MML incorporates Bayesian priors to facilitate a joint encoding of the model and , treating as a communication problem where the receiver infers both from a single message. A key distinction lies in their likelihood treatments: MDL centers on the normalized maximum likelihood for model coding, whereas MML emphasizes expected message lengths under posterior distributions. According to Wallace (2005), this makes MML generally more accurate for small sample sizes, as it better accounts for prior uncertainty in parameter estimation. Empirically, the principles exhibit complementary strengths in application contexts. MDL performs particularly well in sequential prediction tasks, leveraging its complexity formulation for cumulative coding efficiency over . MML, however, is better suited to batch inductive , where the full is available upfront, enabling more precise joint optimization of model and data encodings. Overall, MML is often regarded as a Bayesian refinement of MDL, extending its non-Bayesian framework with principled prior integration for enhanced robustness in complex scenarios.

Versus Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC)

The (AIC), introduced by Akaike in 1973, is a tool that balances model fit and complexity through the formula −2log⁡L+2k-2 \log L + 2k, where LL is the maximized likelihood and kk is the number of parameters. This criterion penalizes model complexity linearly but is inconsistent, meaning it may select overparameterized models with positive probability even as sample size increases, particularly when the true model is among the candidates. The (BIC), proposed by Schwarz in , extends this approach with the formula −2log⁡L+klog⁡n-2 \log L + k \log n, where nn is the sample size, imposing a stronger penalty that grows with nn. BIC achieves statistical consistency under large-sample asymptotics and fixed alternative models, converging to the true model with probability approaching 1, but it relies on assumptions like sufficiently large nn and a fixed number of alternative models. In contrast, Minimum Message Length (MML) employs a Bayesian framework with a logarithmic prior on parameters, often a derived from the square root of the determinant of the matrix, and specifies parameter precision based on the to minimize the total message length. This information-theoretic approach avoids the arbitrary constants in AIC's fixed penalty or BIC's logarithmic scaling, providing a more principled penalty derived from . MML also handles non-nested models more effectively by evaluating total descriptive complexity in bits, enabling direct comparisons across disparate model structures without reliance on asymptotic approximations. Simulations from the late 1990s demonstrate that MML outperforms BIC in finite samples for segmentation, accurately identifying the number of segments (e.g., 3 segments with n=60n=60) more reliably than BIC or AIC. Additionally, MML is scale-invariant, preserving performance under data rescaling, unlike BIC which can bias toward simpler models in such cases. However, MML is computationally more intensive than the closed-form expressions of AIC and BIC, requiring optimization of the full posterior message length.

References

  1. https://arxiv.org/pdf/2209.14571.pdf
  2. https://users.monash.edu/~dld/MML.html
  3. https://doi.org/10.1007/0-387-27656-4
  4. https://scholar.google.com/citations?user=d7k7_7wAAAAJ&hl=en
  5. https://doi.org/10.1093/comjnl/11.2.185
  6. https://users.monash.edu/~dld/CSWallacePublications/
  7. https://doi.org/10.1016/B978-0-444-51862-0.50030-7
  8. https://researchmgt.monash.edu/ws/portalfiles/portal/390015615/376081031_oa.pdf
  9. https://arxiv.org/abs/2209.14571
  10. https://arxiv.org/abs/2508.06230
  11. https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
  12. https://users.monash.edu/~dld/Publications/1999/WallaceDowe1999aMinimumMessageLengthAndKolmogorovComplexity.pdf
  13. https://academic.oup.com/comjnl/article/42/4/270/558949
  14. https://academic.oup.com/comjnl/article-pdf/11/2/185/1075925/11-2-185.pdf
  15. https://link.springer.com/book/10.1007/0-387-27656-4
  16. https://homepages.cwi.nl/~paulv/papers/mdlindbayeskolmcompl.pdf
  17. https://doi.org/10.1111/j.2517-6161.1987.tb01453.x
  18. https://ieeexplore.ieee.org/document/8847629
  19. https://academic.oup.com/comjnl/article-abstract/11/3/236/379208
  20. https://users.monash.edu/~dld/Publications/2004/FitzgibbonDoweVahid_ICISIP2004.pdf
  21. https://onlinelibrary.wiley.com/doi/abs/10.4218/etrij.05.0104.0180
  22. https://users.monash.edu/~dld/Publications/2002/AgustaDowe2002ClusteringGaussian%2BtMML.pdf
  23. https://doi.org/10.1007/11941439_23
  24. https://pmc.ncbi.nlm.nih.gov/articles/PMC11764307/
  25. https://logarithmic.net/pfh/pymml
  26. https://cran.r-project.org/package=GMKMcharlie
  27. https://www.semanticscholar.org/paper/MDL-and-MML%253A-Similarities-and-differences-Baxter-Oliver/1e1b094a71796c18496cb4593127422997bb0bbf
  28. https://www.sciencedirect.com/science/article/pii/0005109878900055
  29. https://www.scirp.org/reference/referencespapers?referenceid=1741007
  30. https://pmc.ncbi.nlm.nih.gov/articles/PMC7299313/
  31. https://sites.stat.washington.edu/courses/stat527/s13/readings/ann_stat1978.pdf
  32. https://cs.fit.edu/~pkc/ml/related/oliver-pakdd99.pdf
Add your contribution
Related Hubs
User Avatar
No comments yet.