Hubbry Logo
Bayesian probabilityBayesian probabilityMain
Open search
Bayesian probability
Community hub
Bayesian probability
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Bayesian probability
Bayesian probability
Bayesian probability
View on Wikipedia
from Wikipedia
Part of a series on
Bayesian statistics
Posterior = Likelihood × Prior ÷ Evidence
Background
Model building
Posterior approximation
Estimators
Evidence approximation
Model evaluation

Bayesian probability (/ˈbeɪziən/ BAY-zee-ən or /ˈbeɪʒən/ BAY-zhən)[1] is an interpretation of the concept of probability, in which, instead of frequency or propensity of some phenomenon, probability is interpreted as reasonable expectation[2] representing a state of knowledge[3] or as quantification of a personal belief.[4]

The Bayesian interpretation of probability can be seen as an extension of propositional logic that enables reasoning with hypotheses;[5][6] that is, with propositions whose truth or falsity is unknown. In the Bayesian view, a probability is assigned to a hypothesis, whereas under frequentist inference, a hypothesis is typically tested without being assigned a probability.

Bayesian probability belongs to the category of evidential probabilities; to evaluate the probability of a hypothesis, the Bayesian probabilist specifies a prior probability. This, in turn, is then updated to a posterior probability in the light of new, relevant data (evidence).[7] The Bayesian interpretation provides a standard set of procedures and formulae to perform this calculation.

The term Bayesian derives from the 18th-century English mathematician and theologian Thomas Bayes, who provided the first mathematical treatment of a non-trivial problem of statistical data analysis using what is now known as Bayesian inference.[8]: 131  Mathematician Pierre-Simon Laplace pioneered and popularized what is now called Bayesian probability.[8]: 97–98 

Bayesian methodology

[edit]

Bayesian methods are characterized by concepts and procedures as follows:

  • The use of random variables, or more generally unknown quantities,[9] to model all sources of uncertainty in statistical models including uncertainty resulting from lack of information (see also aleatoric and epistemic uncertainty).
  • The need to determine the prior probability distribution taking into account the available (prior) information.
  • The sequential use of Bayes' theorem: as more data become available, calculate the posterior distribution using Bayes' theorem; subsequently, the posterior distribution becomes the next prior.
  • While for the frequentist, a hypothesis is a proposition (which must be either true or false) so that the frequentist probability of a hypothesis is either 0 or 1, in Bayesian statistics, the probability that can be assigned to a hypothesis can also be in a range from 0 to 1 if the truth value is uncertain.

Objective and subjective Bayesian probabilities

[edit]

Broadly speaking, there are two interpretations of Bayesian probability. For objectivists, who interpret probability as an extension of logic, probability quantifies the reasonable expectation that everyone (even a "robot") who shares the same knowledge should share in accordance with the rules of Bayesian statistics, which can be justified by Cox's theorem.[3][10] For subjectivists, probability corresponds to a personal belief.[4] Rationality and coherence allow for substantial variation within the constraints they pose; the constraints are justified by the Dutch book argument or by decision theory and de Finetti's theorem.[4] The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.

History

[edit]
Main article: History of statistics § Bayesian statistics

The term Bayesian derives from Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes' theorem in a paper titled "An Essay Towards Solving a Problem in the Doctrine of Chances".[11] In that special case, the prior and posterior distributions were beta distributions and the data came from Bernoulli trials. It was Pierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence.[12] Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes).[13] After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.[13]

In the 20th century, the ideas of Laplace developed in two directions, giving rise to objective and subjective currents in Bayesian practice. Harold Jeffreys' Theory of Probability (first published in 1939) played an important role in the revival of the Bayesian view of probability, followed by works by Abraham Wald (1950) and Leonard J. Savage (1954). The adjective Bayesian itself dates to the 1950s; the derived Bayesianism, neo-Bayesianism is of 1960s coinage.[14][15][16] In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed.[17] No subjective decisions need to be involved. In contrast, "subjectivist" statisticians deny the possibility of fully objective analysis for the general case.

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods and the consequent removal of many of the computational problems, and to an increasing interest in nonstandard, complex applications.[18] While frequentist statistics remains strong (as demonstrated by the fact that much of undergraduate teaching is based on it [19]), Bayesian methods are widely accepted and used, e.g., in the field of machine learning.[20]

Justification

[edit]

The use of Bayesian probabilities as the basis of Bayesian inference has been supported by several arguments, such as Cox axioms, the Dutch book argument, arguments based on decision theory and de Finetti's theorem.

Axiomatic approach

[edit]

Richard T. Cox showed that Bayesian updating follows from several axioms, including two functional equations and a hypothesis of differentiability.[10][21] The assumption of differentiability or even continuity is controversial; Halpern found a counterexample based on his observation that the Boolean algebra of statements may be finite.[22] Other axiomatizations have been suggested by various authors with the purpose of making the theory more rigorous.[9]

Dutch book approach

[edit]
Main article: Dutch book

Bruno de Finetti proposed the Dutch book argument based on betting. A clever bookmaker makes a Dutch book by setting the odds and bets to ensure that the bookmaker profits—at the expense of the gamblers—regardless of the outcome of the event (a horse race, for example) on which the gamblers bet. It is associated with probabilities implied by the odds not being coherent.

However, Ian Hacking noted that traditional Dutch book arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. For example, Hacking writes[23][24] "And neither the Dutch book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."

In fact, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics"[25] following the publication of Richard C. Jeffrey's rule, which is itself regarded as Bayesian[26]). The additional hypotheses sufficient to (uniquely) specify Bayesian updating are substantial[27] and not universally seen as satisfactory.[28]

Decision theory approach

[edit]

A decision-theoretic justification of the use of Bayesian inference (and hence of Bayesian probabilities) was given by Abraham Wald, who proved that every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.[29] Conversely, every Bayesian procedure is admissible.[30]

Personal probabilities and objective methods for constructing priors

[edit]

Following the work on expected utility theory of Ramsey and von Neumann, decision-theorists have accounted for rational behavior using a probability distribution for the agent. Johann Pfanzagl completed the Theory of Games and Economic Behavior by providing an axiomatization of subjective probability and utility, a task left uncompleted by von Neumann and Oskar Morgenstern: their original theory supposed that all the agents had the same probability distribution, as a convenience.[31] Pfanzagl's axiomatization was endorsed by Oskar Morgenstern: "Von Neumann and I have anticipated ... [the question whether probabilities] might, perhaps more typically, be subjective and have stated specifically that in the latter case axioms could be found from which could derive the desired numerical utility together with a number for the probabilities (cf. p. 19 of The Theory of Games and Economic Behavior). We did not carry this out; it was demonstrated by Pfanzagl ... with all the necessary rigor".[32]

Ramsey and Savage noted that the individual agent's probability distribution could be objectively studied in experiments. Procedures for testing hypotheses about probabilities (using finite samples) are due to Ramsey (1931) and de Finetti (1931, 1937, 1964, 1970). Both Bruno de Finetti[33][34] and Frank P. Ramsey[34][35] acknowledge their debts to pragmatic philosophy, particularly (for Ramsey) to Charles S. Peirce.[34][35]

The "Ramsey test" for evaluating probability distributions is implementable in theory, and has kept experimental psychologists occupied for a half century.[36] This work demonstrates that Bayesian-probability propositions can be falsified, and so meet an empirical criterion of Charles S. Peirce, whose work inspired Ramsey. (This falsifiability-criterion was popularized by Karl Popper.[37][38])

Modern work on the experimental evaluation of personal probabilities uses the randomization, blinding, and Boolean-decision procedures of the Peirce-Jastrow experiment.[39] Since individuals act according to different probability judgments, these agents' probabilities are "personal" (but amenable to objective study).

Personal probabilities are problematic for science and for some applications where decision-makers lack the knowledge or time to specify an informed probability-distribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.

Indeed, some Bayesians have argued the prior state of knowledge defines the (unique) prior probability-distribution for "regular" statistical problems; cf. well-posed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace to John Maynard Keynes, Harold Jeffreys, and Edwin Thompson Jaynes. These theorists and their successors have suggested several methods for constructing "objective" priors (Unfortunately, it is not always clear how to assess the relative "objectivity" of the priors proposed under these methods):

Each of these methods contributes useful priors for "regular" one-parameter problems, and each prior can handle some challenging statistical models (with "irregularity" or several parameters). Each of these methods has been useful in Bayesian practice. Indeed, methods for constructing "objective" (alternatively, "default" or "ignorance") priors have been developed by avowed subjective (or "personal") Bayesians like James Berger (Duke University) and José-Miguel Bernardo (Universitat de València), simply because such priors are needed for Bayesian practice, particularly in science.[40] The quest for "the universal method for constructing priors" continues to attract statistical theorists.[40]

Thus, the Bayesian statistician needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.

See also

[edit]

References

[edit]

Bibliography

[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Bayesian probability

View on Grokipedia
from Grokipedia
Bayesian probability is an interpretation of the of probability in which probabilities represent degrees of or subjective in the occurrence of an event or the truth of a hypothesis, rather than objective long-run frequencies, and these beliefs are rationally updated using in response to new evidence. The foundational principle, , provides a mathematical framework for this updating process, expressed as
P(H∣E)=P(E∣H)⋅P(H)P(E),P(H|E) = \frac{P(E|H) \cdot P(H)}{P(E)},
where P(H∣E)P(H|E) is the posterior probability of hypothesis HH given evidence EE, P(E∣H)P(E|H) is the likelihood of observing EE if HH is true, P(H)P(H) is the prior probability of HH, and P(E)P(E) is the marginal probability of EE. This theorem, derived from the definition of , allows for the incorporation of prior knowledge or beliefs into , treating unknown parameters as random variables described by probability distributions. Historically, the ideas trace back to the , when English mathematician and Presbyterian minister (c. 1701–1761) developed the theorem as part of an effort to quantify , possibly motivated by the philosophical arguments of on causation and evidence. ' work remained unpublished during his lifetime and was edited and presented to the Royal Society by his colleague in 1763, under the title "An Essay towards solving a Problem in the Doctrine of Chances." The approach gained prominence in the 20th century through advocates like and , who formalized subjective probability interpretations, though it faced criticism for perceived subjectivity until computational advances revived its use. In contrast to frequentist statistics, which views probabilities as limits of relative frequencies in repeated experiments and estimates parameters as fixed unknowns, Bayesian methods enable direct probabilistic statements about , such as the probability that a parameter exceeds a certain value, by integrating over the posterior distribution. This framework is particularly powerful in handling , small sample sizes, and hierarchical models, where priors can encode expert knowledge or regularization. Bayesian probability has broad applications across fields, including for parameter estimation and hypothesis testing, algorithms like Bayesian networks and Gaussian processes for prediction and classification, medical diagnostics to update disease probabilities based on test results, and decision-making under uncertainty in and . Notable modern uses include spam detection in filters, adaptive clinical trials that adjust sample sizes dynamically, and probabilistic modeling in to manage complex, high-dimensional data.

Foundations of Bayesian Probability

Definition and Interpretation

Bayesian probability interprets probability as a measure of the degree of belief in a or , rather than as a long-run relative of in repeated trials. This subjective view allows probabilities to represent personal or epistemic about unknown quantities, such as parameters in a , and enables the incorporation of prior or beliefs before observing data. In contrast, the frequentist interpretation treats probability as an objective defined by the limiting of an event occurring in an infinite sequence of identical trials under fixed conditions. For instance, in estimating the of a from a small number of flips—say, observing 3 heads in 5 flips—a frequentist approach would compute a point estimate of the heads probability (e.g., 0.6) along with a based on hypothetical repeated sampling, without assigning probability to the parameter itself. A Bayesian approach, however, would update an initial belief about the using the observed data, yielding a full over possible values that quantifies directly. Central to this framework are several key concepts: the prior distribution, which encodes initial beliefs about an unknown parameter before seeing data; the likelihood, which measures how well the observed data support different parameter values; the posterior distribution, representing updated beliefs after incorporating the data; and the evidence (or marginal likelihood), which is the probability of the data averaged over all possible parameter values and serves as a normalizing factor. These elements facilitate belief updating, where provides the mathematical mechanism for combining the prior and likelihood to obtain the posterior (detailed in subsequent sections). The term "Bayesian" derives from the 18th-century work of , whose essay laid foundational ideas for , though the modern approach encompasses broader developments in . A simple illustration of updating occurs when assessing the likelihood of : an individual might start with a 30% prior based on seasonal patterns, then observe dark clouds and a , adjusting their to 80% as the new strengthens the case for without requiring repeated observations. This highlights how Bayesian probability accommodates incomplete or finite , providing a coherent way to revise uncertainties in real-world scenarios.

Bayes' Theorem

Bayes' theorem provides the mathematical foundation for updating probabilities based on new evidence in Bayesian inference. It states that the posterior probability of an event AA given evidence BB, denoted P(A∣B)P(A|B), is equal to the likelihood of the evidence given AA, P(B∣A)P(B|A), times the prior probability of AA, P(A)P(A), divided by the marginal probability of the evidence, P(B)P(B): P(A∣B)=P(B∣A)P(A)P(B)P(A|B) = \frac{P(B|A) P(A)}{P(B)} Here, P(A)P(A) represents the prior belief about AA before observing BB, P(B∣A)P(B|A) is the likelihood measuring how well BB supports AA, and P(B)P(B) normalizes the result to ensure probabilities sum to 1. The derives directly from the axioms of . The joint probability of AA and BB can be expressed as P(A∩B)=P(A∣B)P(B)P(A \cap B) = P(A|B) P(B) or equivalently P(A∩B)=P(B∣A)P(A)P(A \cap B) = P(B|A) P(A). Equating these forms yields P(A∣B)P(B)=P(B∣A)P(A)P(A|B) P(B) = P(B|A) P(A), and solving for P(A∣B)P(A|B) gives the theorem. An equivalent formulation uses ratios, which express relative probabilities. The posterior of AA versus its complement ¬A\neg A given BB equal the prior times the likelihood : P(A∣B)P(¬A∣B)=P(B∣A)P(B∣¬A)×P(A)P(¬A)\frac{P(A|B)}{P(\neg A|B)} = \frac{P(B|A)}{P(B|\neg A)} \times \frac{P(A)}{P(\neg A)}. This form highlights how multiplies the initial by a factor quantifying the evidence's evidential value. For continuous parameters, P(B)P(B) is the marginal likelihood obtained by integrating over all possible values of AA: P(B)=∫P(B∣A)P(A) dAP(B) = \int P(B|A) P(A) \, dA. This integral accounts for the total probability of the evidence across the prior distribution. A common application is in diagnostic testing, where computes the probability of given a positive test result. Suppose a has a prior prevalence of 1% (P(D)=0.01P(D) = 0.01), and a test has 99% sensitivity (P(+∣D)=0.99P(+|D) = 0.99) and 99% specificity (P(−∣¬D)=0.99P(-|\neg D) = 0.99, so P(+∣¬D)=0.01P(+|\neg D) = 0.01). The posterior probability of given a positive test is P(D∣+)=0.99×0.010.99×0.01+0.01×0.99≈0.50P(D|+) = \frac{0.99 \times 0.01}{0.99 \times 0.01 + 0.01 \times 0.99} \approx 0.50, showing that even with high test accuracy, the low prevalence halves the odds of true .

Philosophical Perspectives

Subjective Bayesianism

Subjective Bayesianism views probabilities as personal degrees of belief, or credences, that reflect an individual's subjective assessment of uncertainty rather than objective frequencies or long-run tendencies. These credences are coherent if they satisfy the axioms of and are updated rationally using when new evidence becomes available. This approach, pioneered by , emphasizes that probability is inherently subjective, with each person's priors representing their unique state of knowledge or opinion prior to observing data. Coherence in subjective Bayesianism requires adherence to key probability axioms to ensure consistency in one's beliefs and avoid opportunities for sure loss in betting scenarios. Specifically, credences must be non-negative (no belief can have negative probability), normalized (certainty in a tautology is 1, and in a contradiction is 0), and additive (the credence in a disjunction of mutually exclusive events equals the sum of their individual credences). These axioms, as articulated by de Finetti, form the foundation for rational belief structures, where violations lead to incoherence and potential Dutch book arguments against the agent. By maintaining coherence, subjective Bayesians ensure their degrees of belief are logically consistent and amenable to probabilistic reasoning. An illustrative example of subjective Bayesian updating occurs in everyday decision-making, such as predicting . Suppose an individual initially holds a credence of 0.4 that tomorrow, based on seasonal patterns and personal experience (their prior). Upon observing a detailed forecast indicating high and wind patterns favorable for , they incorporate this evidence via to revise their credence upward to 0.8 (the posterior). This process demonstrates how subjective beliefs evolve dynamically with incoming information, allowing for personalized yet rational adjustments without relying on objective frequencies. The implications of subjective Bayesianism for position Bayesian as the normative ideal for , prescribing that individuals should proportion their credences to the to achieve coherent and evidence-responsive opinions. This framework argues that any , regardless of their starting priors, will converge toward truth over time through repeated , provided the is reliable. However, critics argue that over-reliance on personal priors can foster dogmatism, as strongly held initial beliefs may require overwhelming contrary to shift significantly, potentially trapping individuals in entrenchment even when faced with compelling . For instance, a dogmatic prior close to 1 or 0 can render posterior beliefs nearly unchanged, undermining the method's responsiveness to reality.

Objective Bayesianism

Objective Bayesianism seeks to establish priors through formal principles that promote and minimize personal , deriving probabilities from logical rules or informational constraints rather than individual beliefs. This approach contrasts with subjective Bayesianism by emphasizing methods that different rational agents would agree upon, such as invariance under transformations or maximization of uncertainty. It positions itself as a framework for objective inference within the Bayesian , often justified by requirements like consistency across parameterizations. A core method in objective Bayesianism is the principle of indifference, formulated by as the principle of insufficient reason. This principle dictates that, in the absence of distinguishing evidence, equal probabilities should be assigned to all mutually exclusive and exhaustive hypotheses. For discrete parameters, it results in a prior distribution. applied this to sequential predictions via the : after observing ss successes in nn trials of a , the predictive probability of success on the next trial is s+1n+2\frac{s+1}{n+2}, reflecting an initial prior over the success probability updated by data. This approach aims for neutrality but has been critiqued for ambiguity in continuous cases. The maximum principle, advanced by Edwin T. Jaynes, provides a more general tool for constructing objective priors by selecting the distribution that maximizes Shannon subject to constraints encoding available information. , defined as H(p)=−∑pilog⁡piH(p) = -\sum p_i \log p_i for discrete cases or the integral analog for continuous, measures ; maximizing it yields the least informative prior consistent with the constraints. For example, with no constraints beyond normalization and a fixed , the maximum prior is exponential; with a fixed variance, it is Gaussian. Jaynes argued this aligns with scientific by avoiding unfounded assumptions. Jeffreys priors exemplify objective methods through invariance considerations. Proposed by , these priors are proportional to the square root of the determinant of the matrix, ensuring the posterior is invariant under reparameterization. For scale parameters θ>0\theta > 0, such as variance in location-scale models, the Jeffreys prior simplifies to p(θ)∝1/θp(\theta) \propto 1/\theta: p(θ)∝1θ,θ>0.p(\theta) \propto \frac{1}{\theta}, \quad \theta > 0. This form arises because the for scale parameters scales with 1/θ21/\theta^2, leading to a prior that treats logarithmic scales uniformly. In inference for a normal distribution's standard deviation, this prior yields posteriors that are scale-invariant, facilitating consistent conclusions across units. Objective Bayesianism serves as a middle ground between pure and frequentist objectivity, retaining the subjective interpretation of probability while imposing invariance and minimality requirements on priors to achieve consensus. Proponents like James O. Berger argue that such rules, including reference priors (an extension of Jeffreys), balance flexibility with rigor, allowing Bayesian methods to approximate frequentist properties in large samples. This hybrid nature enables applications in complex statistical modeling where subjective elicitation is impractical. Despite these strengths, objective Bayesian methods can produce counterintuitive results, particularly in complex models. The principle of indifference may yield paradoxes, such as differing probabilities from alternative event partitions in geometric problems. Maximum entropy priors can be improper or lead to posteriors that overweight tails in high dimensions, while Jeffreys priors sometimes fail to integrate to finite values in multiparameter settings, complicating normalization. These issues highlight challenges in ensuring priors remain noninformative across intricate structures.

Historical Development

Precursors and Early Formulations

The foundations of probabilistic reasoning that would later underpin Bayesian probability emerged in the through efforts to quantify in games of chance. and developed early concepts of and in interrupted games, such as the "," where Pascal's correspondence with in 1654 laid groundwork for calculating probabilities based on combinatorial analysis. Huygens extended this in his 1657 treatise De ratiociniis in ludo aleae, formalizing the concept of mathematical expectation as the average outcome over possible events, providing a rigorous framework for under that influenced subsequent . These works shifted probability from qualitative judgment to quantitative computation, setting the stage for inverse inference. Jacob Bernoulli's (1713) advanced this foundation with the first proof of the , demonstrating that the relative frequency of an event converges to its probability as trials increase, thereby linking empirical observation to theoretical probability in a way that resonated with later Bayesian updating of beliefs based on evidence. Bernoulli viewed probability as a degree of certainty, incorporating subjective elements into his analysis of binomial trials, which prefigured Bayesian approaches to by emphasizing how repeated observations refine estimates of underlying chances. The explicit formulation of inverse probability appeared posthumously in Thomas Bayes's 1763 essay, "An Essay towards Solving a Problem in the Doctrine of Chances," edited and submitted to Society by . Bayes addressed the challenge of inferring the probability of a cause from observed effects, framing it as a method to update prior assessments of an event's likelihood based on new data, which Price recognized as a novel tool for in . Price's editorial role was pivotal, as he not only published the work but also highlighted its potential for applications beyond chance, ensuring its dissemination among contemporary mathematicians. Pierre-Simon Laplace built directly on Bayes's ideas in his 1774 Mémoire sur la probabilité des causes par les événements, where he generalized inverse probability to determine the likelihood of competing hypotheses given observed , applying it to problems in physics and astronomy such as predicting planetary perturbations. Over the following decades, Laplace refined these concepts in works like Théorie analytique des probabilités (1812), introducing the —a formula for estimating the probability of future successes after a of observed ones, assuming priors—which he used to assess astronomical stability, such as the probability of the solar system's endurance. These contributions transformed Bayes's tentative essay into a systematic methodology for scientific inference, emphasizing the role of prior probabilities in updating beliefs with evidence.

Revival and Modern Advancements

The revival of Bayesian probability in the mid-20th century began with the development of subjective probability frameworks by Frank Ramsey and . In his 1926 essay "Truth and Probability," Ramsey laid foundational ideas for interpreting probabilities as degrees of belief, measurable through betting behavior, which gained renewed attention in and amid debates on statistical . Independently, de Finetti advanced subjective probability in , notably through his 1937 work La prévision: ses lois logiques, ses sources subjectives, arguing that all probabilities are inherently personal and coherence requires adherence to Dutch book avoidance, influencing Bayesian thought through the 1950s. Leonard J. Savage's 1954 book The Foundations of Statistics further solidified this resurgence by axiomatizing subjective probability within a decision-theoretic framework, linking Bayesian updating to expected utility maximization and providing a normative basis for personal probabilities in . This work bridged probability and utility theory, encouraging the application of Bayesian methods to practical problems in and during the post-war era. From the 1960s onward, computational advancements enabled the widespread adoption of Bayesian techniques, particularly through (MCMC) methods. The Metropolis-Hastings algorithm, introduced in 1953 but largely popularized in the 1990s for Bayesian , allowed sampling from complex posterior distributions, revolutionizing inference in high-dimensional spaces. Key figures like Dennis V. Lindley promoted through his advocacy for decision-theoretic approaches and editorial roles, such as on the Journal of the Royal Statistical Society Series B, which emphasized Bayesian perspectives. George E. P. Box contributed seminal work on Bayesian robustness and , including transformations and hierarchical structures in time series analysis during the 1960s and 1970s. advanced modern Bayesian practice in the late 20th and early 21st centuries, co-authoring influential texts like Bayesian Data Analysis (1995, updated 2013) that integrated with hierarchical modeling. Post-2000 developments have integrated Bayesian methods into , hierarchical models, and analytics, addressing scalability and . Bayesian hierarchical models, which pool information across levels to improve estimates in varied datasets, have become standard for applications like and social sciences. In , Bayesian approaches enhance neural networks and by incorporating priors for regularization and uncertainty, as seen in scalable inference techniques for large-scale data. The 2020s have witnessed accelerated growth in Bayesian applications to , driven by needs for reliable probabilistic predictions in areas like autonomous systems and , amid challenges like computational efficiency and prior elicitation.

Justifications for Bayesian Inference

Axiomatic Foundations

Bayesian probability aligns with the foundational axioms of , providing a rigorous mathematical justification for its use in . The standard axioms, formulated by in , define probability as a measure on a Ω\Omega: non-negativity requires 0≤P(E)≤10 \leq P(E) \leq 1 for any event E⊆ΩE \subseteq \Omega, normalization states P(Ω)=1P(\Omega) = 1, and countable additivity holds that for a countable collection of pairwise disjoint events EiE_i, P(⋃iEi)=∑iP(Ei)P\left(\bigcup_i E_i\right) = \sum_i P(E_i). These axioms ensure that probability functions are consistent and behave like measures, forming the basis for all probabilistic reasoning. In the Bayesian framework, probabilities represent degrees of belief that satisfy these axioms, interpreted as coherent previsions—fair prices for gambles over uncertain outcomes that avoid opportunities. emphasized this coherence, showing that subjective probabilities must conform to Kolmogorov's axioms to maintain logical consistency in prevision assessments. Thus, Bayesian updating preserves additivity and other properties, ensuring that posterior beliefs remain valid probability measures. The extension to conditional probabilities, central to , follows from , which derives the rules of probability—including —from qualitative desiderata such as transitivity of reasoning (if AA implies BB and BB implies CC, then AA implies CC) and dominance (a conclusion supported by more evidence cannot be less probable than one supported by less). Richard T. Cox demonstrated that any of satisfying these conditions is isomorphic to the standard probability calculus. To illustrate, consider a non-Bayesian rule where an agent overweights new without fully adjusting for prior structure; for instance, in a setting with multiple hypotheses, such a can lead to posterior beliefs that violate additivity over disjoint events, as the updated probabilities fail to sum correctly for unions. This incoherence highlights why adherence to Bayesian rules is necessary for maintaining the axioms. However, the axioms permit non-uniqueness in infinite sample spaces, where multiple probability measures can satisfy the conditions on the same σ\sigma-algebra, complicating the representation of beliefs without additional structure like regularity assumptions.

Dutch Book Arguments

A Dutch book refers to a collection of bets structured such that the bettor incurs a guaranteed loss irrespective of the actual outcome of the underlying events. This concept, originating in the work of , serves as a pragmatic tool to demonstrate the necessity of coherence in subjective probabilities, where degrees of belief are equated with fair betting quotients. In essence, if an agent's stated probabilities permit such a set of wagers, their beliefs are deemed incoherent, as they expose the agent to sure financial detriment without any compensating gain. In his seminal 1937 paper, de Finetti established a foundational asserting that any assignment of probabilities failing finite additivity—meaning the probability of a does not equal the sum of individual probabilities—admits a Dutch book. Specifically, de Finetti demonstrated that non-additive previsions (betting quotients) over a finite partition of events allow a to construct a sequence of acceptable bets that yields a positive net gain for the regardless of which event occurs. This underpins the subjective Bayesian view by linking probabilistic coherence directly to avoidance of sure loss in betting scenarios. The Dutch book argument extends naturally to conditional probabilities and betting, reinforcing the requirement for Bayesian updating. de Finetti showed that coherence under conditional wagers—bets resolved only if a conditioning event occurs—necessitates that conditional probabilities satisfy the P(A|B) = P(A ∩ B)/P(B), thereby ensuring that revisions of beliefs upon new do not introduce vulnerabilities to Dutch books. Violations of this conditional coherence, such as inconsistent updating rules, permit a to exploit the agent through a series of conditional bets that guarantee loss after the conditioning event transpires. A illustrative example arises in a horse race with mutually exclusive outcomes. Suppose a bettor assigns probabilities such that the sum over all horses exceeds 1, say P(Horse A wins) = 0.6 and P(Horse B wins) = 0.6 for a two-horse race. A bookmaker can then accept bets from the bettor on both horses at these odds (equivalent to paying out $1 for a $0.6 stake if the horse wins) while simultaneously offering a bet against the race occurring (or exploiting the overestimation). By wagering appropriately on each, the bookmaker secures a net profit: if A wins, the bookmaker pays $1 on A's bet but collects from the excess; the imbalance ensures overall gain exceeding stakes across outcomes. This arbitrage-like loss for the bettor highlights how additive violations enable exploitation. Extensions of de Finetti's argument to continuous probability spaces involve approximating infinite partitions with finite ones, where coherence still demands avoidance of Dutch books through constraints akin to additivity. However, such extensions often rely on limits of finite cases and face challenges in rigorously constructing sure-loss bets without additional regularity conditions. Critiques of the Dutch book framework commonly point to its implicit neutrality, as the argument presumes agents accept small bets at fair odds without curvature, potentially failing for risk-averse or risk-seeking individuals who might rationally decline such wagers to avoid variance. Despite these limitations, the argument remains a cornerstone for justifying probabilistic coherence in .

Decision-Theoretic Justifications

Decision-theoretic justifications for Bayesian probability emphasize its role in rational under , where choices are evaluated based on expected maximization. In this framework, subjective probabilities serve as inputs to functions, enabling agents to select actions that optimize outcomes according to their preferences. A foundational contribution comes from Leonard J. Savage's axiomatic system in The Foundations of Statistics (1954), which derives subjective expected from a set of postulates including completeness (every pair of acts can be compared), transitivity (preferences are consistent across comparisons), and continuity (preferences allow for probabilistic mixtures). These axioms imply that rational agents represent beliefs via subjective probabilities and evaluate decisions by maximizing expected , providing a normative basis for Bayesian methods in uncertain environments. Bayesian updating aligns with this framework by offering an optimal strategy for minimizing in sequential decisions. Upon receiving new evidence, the posterior distribution minimizes the posterior for actions, ensuring that decisions incorporate all available to achieve the lowest anticipated . For instance, in decision-making, a physician might use Bayesian updating to assess the posterior probability of a given test results and prior prevalence, then select a treatment that minimizes —such as weighing the risks of false positives against treatment side effects to avoid unnecessary interventions. This approach connects to Abraham Wald's statistical , outlined in Statistical Decision Functions (1950), where Bayes rules are shown to be admissible, meaning no other rule can perform better in all states without performing worse in some, thus justifying Bayesian procedures as minimally suboptimal in . Critiques of these justifications highlight the sensitivity of Bayesian decisions to prior specifications, particularly in high-stakes contexts where differing priors can lead to substantially varied expected utilities and potentially suboptimal choices if priors are misspecified.

Prior Distributions

Eliciting Personal Priors

Eliciting personal priors involves structured processes to translate an individual's subjective beliefs into formal probability distributions for Bayesian analysis, rooted in the subjective Bayesianism paradigm where priors reflect personal degrees of belief. Practical methods for direct elicitation include questionnaires that prompt experts to specify quantiles or percentiles of their beliefs about , such as estimating the 25th, 50th, and 75th percentiles for a distribution's shape.00175-9/fulltext) Imagining scenarios, known as predictive elicitation, asks individuals to forecast outcomes under hypothetical conditions to infer prior distributions indirectly, reducing direct focus on parameter values. Betting analogies, like the method, simulate wagering on outcomes to reveal implicit probabilities, helping to quantify beliefs through relative . In assigning priors, individuals must recognize encoding biases such as or , where overly positive or negative expectations can skew distributions toward extreme values, and anchoring effects, where initial suggestions unduly influence subsequent judgments.00175-9/fulltext) To mitigate these, elicitation protocols often incorporate clear instructions, randomized question orders, and feedback to encourage balanced assessments.00175-9/fulltext) A representative example occurs in clinical trials, where the elicits priors for treatment efficacy parameters by iteratively surveying experts anonymously, providing aggregated feedback after each round to converge on a consensus distribution, such as for a drug's response rate. Personal priors are updated iteratively with incoming data through , where the posterior from one stage becomes the prior for the next, allowing beliefs to evolve sequentially as evidence accumulates. Challenges in elicitation include interpersonal variability, where experts in the same domain may produce substantially different prior distributions due to diverse experiences, leading to divergent posterior inferences. Anchoring effects exacerbate this by causing reliance on initial elicited values across individuals, complicating aggregation into group priors.

Objective Methods for Prior Construction

Objective methods for prior construction in aim to select prior distributions that are free from subjective personal beliefs, instead relying on formal principles to achieve desirable inferential properties such as invariance, optimality in information gain, or frequentist coverage guarantees. These methods emerged as a response to the challenges of eliciting informative priors, particularly in complex models where expert opinion may be unreliable or unavailable. By focusing on the model's structure and sampling properties, objective priors facilitate reproducible and objective Bayesian analyses. One foundational approach is the , which derives a non-informative prior proportional to the of the of the Fisher information matrix. Formally, for a parameter θ\theta, the prior is given by π(θ)∝det⁡I(θ),\pi(\theta) \propto \sqrt{\det \mathcal{I}(\theta)},
Add your contribution
Related Hubs
User Avatar
No comments yet.