Hubbry Logo
search
logo
MuZero avatar
MuZero
MuZero
current hub
17040

MuZero

logo
Community Hub0 Subscribers
Read side by side
MuZero
View on Wikipedia
from Wikipedia
This article is part of the series on
Chess programming
Board representations
Evaluation functions
Tuning and training algorithms
Graph and tree search algorithms
Chess computers
Chess engines

MuZero is a computer program developed by artificial intelligence research company DeepMind to master games without knowing their rules.[1][2][3] Its release in 2019 included benchmarks of its performance in go, chess, shogi, and a standard suite of Atari games. The algorithm uses an approach similar to AlphaZero. It matched AlphaZero's performance in chess and shogi, improved on its performance in Go, and improved on the state of the art in mastering a suite of 57 Atari games (the Arcade Learning Environment), a visually-complex domain.

MuZero was trained via self-play, with no access to rules, opening books, or endgame tablebases. The trained algorithm used the same convolutional and residual architecture as AlphaZero, but with 20 percent fewer computation steps per node in the search tree.[4]

History

[edit]

MuZero really is discovering for itself how to build a model and understand it just from first principles.

— David Silver, DeepMind, Wired[5]

On November 19, 2019, the DeepMind team released a preprint introducing MuZero.

Derivation from AlphaZero

[edit]
Further information: AlphaZero

MuZero (MZ) is a combination of the high-performance planning of the AlphaZero (AZ) algorithm with approaches to model-free reinforcement learning. The combination allows for more efficient training in classical planning regimes, such as Go, while also handling domains with much more complex inputs at each stage, such as visual video games.

MuZero was derived directly from AZ code, sharing its rules for setting hyperparameters. Differences between the approaches include:[6]

  • AZ's planning process uses a simulator. The simulator knows the rules of the game. It has to be explicitly programmed. A neural network then predicts the policy and value of a future position. Perfect knowledge of game rules is used in modeling state transitions in the search tree, actions available at each node, and termination of a branch of the tree. MZ does not have access to the rules, and instead learns one with neural networks.
  • AZ has a single model for the game (from board state to predictions); MZ has separate models for representation of the current state (from board state into its internal embedding), dynamics of states (how actions change representations of board states), and prediction of policy and value of a future position (given a state's representation).
  • MZ's hidden model may be complex, and it may turn out it can host computation; exploring the details of the hidden model in a trained instance of MZ is a topic for future exploration.
  • MZ does not expect a two-player game where winners take all. It works with standard reinforcement-learning scenarios, including single-agent environments with continuous intermediate rewards, possibly of arbitrary magnitude and with time discounting. AZ was designed for two-player games that could be won, drawn, or lost.

Comparison with R2D2

[edit]

The previous state of the art technique for learning to play the suite of Atari games was R2D2, the Recurrent Replay Distributed DQN.[7]

MuZero surpassed both R2D2's mean and median performance across the suite of games, though it did not do better in every game.

Training and results

[edit]

MuZero used 16 third-generation tensor processing units (TPUs) for training, and 1000 TPUs for selfplay for board games, with 800 simulations per step and 8 TPUs for training and 32 TPUs for selfplay for Atari games, with 50 simulations per step.

AlphaZero used 64 second-generation TPUs for training, and 5000 first-generation TPUs for selfplay. As TPU design has improved (third-generation chips are 2x as powerful individually as second-generation chips, with further advances in bandwidth and networking across chips in a pod), these are comparable training setups.

R2D2 was trained for 5 days through 2M training steps.

Initial results

[edit]

MuZero matched AlphaZero's performance in chess and Shogi after roughly 1 million training steps. It matched AZ's performance in Go after 500,000 training steps and surpassed it by 1 million steps. It matched R2D2's mean and median performance across the Atari game suite after 500 thousand training steps and surpassed it by 1 million steps, though it never performed well on 6 games in the suite.

[edit]

MuZero was viewed as a significant advancement over AlphaZero, and a generalizable step forward in unsupervised learning techniques.[8][9] The work was seen as advancing understanding of how to compose systems from smaller components, a systems-level development more than a pure machine-learning development.[10]

While only pseudocode was released by the development team, Werner Duvaud produced an open source implementation based on that.[11]

MuZero has been used as a reference implementation in other work, for instance as a way to generate model-based behavior.[12]

In late 2021, a more efficient variant of MuZero was proposed, named EfficientZero. It "achieves 194.3 percent mean human performance and 109.0 percent median performance on the Atari 100k benchmark with only two hours of real-time game experience".[13]

In early 2022, a variant of MuZero was proposed to play stochastic games (for example 2048, backgammon), called Stochastic MuZero, which uses afterstate dynamics and chance codes to account for the stochastic nature of the environment when training the dynamics network.[14]

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

MuZero

View on Grokipedia
from Grokipedia
MuZero is a model-based reinforcement learning algorithm developed by researchers at DeepMind that achieves superhuman performance in a variety of challenging games, including the perfect-information board games Go, chess, and shogi, as well as the imperfect-information video games from the Atari 2600 suite, without requiring explicit knowledge of the game rules or environment dynamics.[1] Introduced in a preliminary form in 2019 and detailed in a 2020 Nature paper, MuZero learns an internal representation of the environment through self-play, using a neural network to predict key elements such as rewards, action policies, and state values, which enable effective planning via integration with Monte Carlo tree search (MCTS).[2][1] Unlike its predecessor AlphaZero, which relies on a known model of the environment's transition function to simulate future states, MuZero's innovation lies in its ability to implicitly learn these dynamics alongside the policy and value functions, making it applicable to domains where rules are unknown, partially observable, or computationally expensive to simulate.[3][1] The algorithm consists of three core components: a representation network that encodes observations into hidden states, a dynamics network that predicts subsequent hidden states and rewards from action choices, and a prediction network that outputs policy and value estimates from hidden states, all trained end-to-end using temporal difference learning and self-play trajectories.[1] In terms of achievements, MuZero matches or exceeds AlphaZero's superhuman performance in Go (with Elo ratings over 3,000), chess, and shogi after equivalent training compute, while setting new state-of-the-art scores on the Atari benchmark across 57 games, achieving an average human-normalized score of 99.7% when trained on 200 million frames per game and 100.7% on a subset with 20 billion frames.[1] These results demonstrate MuZero's sample efficiency and scalability, as it improves performance dramatically with additional planning steps during inference—for instance, gaining over 1,000 Elo points in Go when search time increases from 0.1 to 50 seconds per move.[3][1] Beyond gaming, MuZero's principles of learning latent models for planning have been extended to real-world applications, such as optimizing video compression in YouTube's infrastructure, where it outperformed human-engineered heuristics by reducing bandwidth usage while maintaining quality.[4] This adaptability highlights MuZero's potential in broader artificial general intelligence pursuits, emphasizing model-based planning in unknown environments.[3]

Background

Reinforcement Learning Foundations

Reinforcement learning (RL) is a paradigm in machine learning where an intelligent agent learns to select actions in an environment through trial-and-error interactions, with the goal of maximizing the expected cumulative reward over time. Unlike supervised learning, which relies on labeled data, or unsupervised learning, which seeks patterns without explicit feedback, RL emphasizes sequential decision-making under uncertainty, where actions influence future states and rewards. This framework draws from optimal control and behavioral psychology, enabling agents to discover optimal behaviors autonomously. At the core of RL is the Markov Decision Process (MDP), a mathematical model that formalizes the agent's interaction with the environment. An MDP is defined by a tuple ⟨S,A,P,R,γ⟩\langle S, A, P, R, \gamma \rangle, where SS is the set of states representing the agent's situation, AA is the set of possible actions, P(s′∣s,a)P(s'|s,a) denotes the transition probabilities to next states s′s' given state ss and action aa, R(s,a)R(s,a) is the reward function providing immediate feedback, and γ∈[0,1)\gamma \in [0,1) is the discount factor prioritizing near-term rewards. Central to solving MDPs are policies π(a∣s)\pi(a|s), which map states to action probabilities, and value functions: the state-value function Vπ(s)V^\pi(s) estimating expected discounted returns from state ss under policy π\pi, and the action-value function Qπ(s,a)Q^\pi(s,a) for state-action pairs. These elements enable the agent to evaluate and improve decision-making strategies. RL algorithms are broadly categorized into model-free and model-based approaches. Model-free methods, exemplified by Q-learning for value estimation and policy gradient techniques for direct policy optimization, learn policies or value functions solely from sampled experiences (state-action-reward-next state tuples) without explicitly modeling the environment's dynamics. This simplicity allows them to operate in unknown environments but often at the cost of requiring extensive data. Model-based RL, conversely, involves learning approximations of the transition function PP and reward function RR, which can then support planning algorithms to simulate trajectories and derive better policies, potentially enhancing efficiency in data-scarce settings.[5] A foundational principle in model-based RL and dynamic programming for MDPs is the Bellman equation, which expresses the optimal value function recursively:
V∗(s)=max⁡a[R(s,a)+γ∑s′P(s′∣s,a)V∗(s′)] V^*(s) = \max_a \left[ R(s,a) + \gamma \sum_{s'} P(s'|s,a) V^*(s') \right]
This equation decomposes the value of a state into the immediate reward plus the discounted value of the best subsequent state, satisfying the Bellman optimality principle. Value iteration, a dynamic programming algorithm, solves it by initializing V0(s)=0V^0(s) = 0 and iteratively applying the Bellman update operator until convergence to V∗V^*, providing the basis for optimal policy derivation via π∗(s)=arg⁡max⁡aQ∗(s,a)\pi^*(s) = \arg\max_a Q^*(s,a). Despite its strengths, RL encounters significant challenges, including sample inefficiency—where agents must generate vast amounts of interaction data to achieve reliable performance, limiting applicability to real-world systems with costly or risky trials—and handling partial observability, where the agent observes only incomplete state information, necessitating extensions like partially observable MDPs (POMDPs) to maintain the Markov property through belief states. These issues underscore the need for hybrid approaches that balance exploration, generalization, and robustness. Advanced RL applications, such as AlphaZero in board games, highlight how overcoming these hurdles can yield superhuman performance in structured domains.[6][7]

Predecessors: AlphaZero and Related Algorithms

AlphaZero, developed by DeepMind in 2017, represents a landmark in reinforcement learning by combining deep neural networks with Monte Carlo Tree Search (MCTS) to achieve superhuman performance in complex board games including chess, shogi, and Go through self-play.[7] The algorithm employs a single neural network that outputs both a policy function—approximating the probability distribution over actions—and a value function—estimating the expected outcome from a given state—trained end-to-end using data generated from games played against versions of itself.[7] During gameplay, MCTS uses the neural network to guide search, simulating thousands of possible futures based on the known game rules to select high-value actions, enabling tabula rasa learning without human knowledge or domain-specific heuristics.[7] This approach demonstrated dramatic efficiency, mastering chess in under 24 hours of training on a single machine cluster, far surpassing traditional engines reliant on handcrafted evaluations.[7] Despite its successes, AlphaZero assumes access to a perfect model of the environment's transition dynamics, as it requires explicit knowledge of the rules to perform MCTS simulations, limiting its applicability to domains with fully specified mechanics.[7] In environments like Atari games, where dynamics are unknown or observations are raw pixels without predefined rules, AlphaZero's reliance on simulation becomes inefficient or infeasible, as constructing an accurate world model from scratch is computationally prohibitive.[7] These constraints highlight the need for algorithms that can learn effective representations of dynamics implicitly while retaining planning capabilities. To address challenges in model-free settings, DeepMind introduced R2D2 in 2019, a distributed reinforcement learning agent designed for partially observable environments like Atari-57, leveraging recurrent neural networks (RNNs) to maintain hidden states across frames and handle temporal dependencies.[8] R2D2 extends distributional Q-learning with prioritized experience replay across multiple actors, enabling off-policy training on diverse trajectories while using LSTMs to process sequential pixel inputs, achieving state-of-the-art scores on 52 of 57 Atari games through scalable distributed training.[8] However, as a purely model-free method, R2D2 lacks explicit planning mechanisms like MCTS, relying instead on value estimation for action selection, which can hinder performance in tasks requiring long-term strategic foresight or sparse rewards.[8] The table below compares key aspects of AlphaZero and R2D2, illustrating their complementary strengths and the motivation for hybrid approaches like MuZero that integrate learned models with planning in unknown environments.
AspectAlphaZeroR2D2
Learning ParadigmModel-based (uses known transition rules for MCTS simulations)Model-free (direct policy/value learning from experience replay)
Core ComponentsNeural policy/value network + MCTS for planningRecurrent distributional Q-network + distributed prioritized replay
DomainsBoard games (e.g., chess, Go) with discrete, rule-based statesAtari games with pixel observations and partial observability
StrengthsSuperior long-term planning via search; superhuman in strategic gamesScalable to high-dimensional inputs; handles temporal abstraction
LimitationsRequires explicit environment model; inefficient for unknown dynamicsNo built-in planning; struggles with sparse rewards and strategy
Path to MuZeroMuZero learns implicit dynamics to enable planning without rulesMuZero adds model learning and MCTS to enhance strategic capabilities

Development

Origins at DeepMind

MuZero was developed at DeepMind, an artificial intelligence research laboratory and subsidiary of Google, by a team led by researchers Julian Schrittwieser and David Silver.[1] The project emerged as a natural extension of DeepMind's prior breakthroughs in reinforcement learning, particularly AlphaZero, which had demonstrated superhuman performance in board games through self-play and Monte Carlo tree search.[1] The algorithm's initial public announcement came on November 19, 2019, via a preprint uploaded to arXiv titled "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model," authored by Schrittwieser and collaborators including Ioannis Antonoglou, Thomas Hubert, and others.[2] This marked the first detailed disclosure of MuZero, with no significant prior public leaks or announcements from DeepMind before late 2019.[3] The core motivations for MuZero's creation stemmed from the limitations of existing planning algorithms in handling environments with unknown or complex dynamics, such as real-world scenarios where rules cannot be hardcoded.[1] DeepMind aimed to create a more general reinforcement learning system capable of learning predictive models directly from observations, mirroring human-like adaptation without explicit domain knowledge.[1] Internal development involved experiments that integrated learned models with tree-based search techniques, building toward the system's ability to master diverse domains.[9] Following the preprint, MuZero was presented at the NeurIPS 2019 conference and elaborated in a full peer-reviewed paper published in Nature on December 23, 2020, solidifying its place in the progression of model-based reinforcement learning.[1][3]

Key Innovations Over Prior Work

MuZero represents a significant advancement in reinforcement learning by introducing a learned model that predicts future outcomes implicitly, without relying on explicit rules or domain knowledge about the environment's dynamics. Unlike prior algorithms such as AlphaZero, which required predefined transition and reward functions for perfect-information games, MuZero learns a model through three core components: a representation function that encodes observations into latent states, a dynamics function that simulates transitions in this latent space, and a prediction function that estimates rewards and values. This implicit modeling allows the agent to anticipate the consequences of actions solely from interaction data, enabling rule-agnostic performance across diverse domains.[2] A key innovation lies in MuZero's hybrid architecture, which merges model-based planning—exemplified by AlphaZero's Monte Carlo Tree Search (MCTS)—with model-free learning strategies akin to those in R2D2, particularly for handling partial observability in environments like Atari games. By integrating a learned model into the planning process, MuZero performs lookahead simulations in the latent space during decision-making, while the model itself is trained end-to-end using model-free techniques on self-play trajectories. This combination achieves superhuman performance in both board games (matching AlphaZero's results in Go, chess, and shogi) and Atari, without needing human-provided rules, thus broadening applicability to real-world scenarios with imperfect information.[2] MuZero addresses partial observability by deriving latent state representations directly from raw image or video inputs, transforming sequential observations into a compact hidden state that captures the underlying environment dynamics. This approach generalizes seamlessly from fully observable board games to partially observable video games, where history-dependent states are inferred without explicit belief-state maintenance. The conceptual flow proceeds from raw observations to a latent model that simulates future states and rewards, culminating in informed planning that guides action selection—reducing reliance on human expertise and enabling efficient mastery with fewer assumptions about the environment. As outlined in the 2019 DeepMind paper introducing the algorithm, this framework yields efficiency gains, such as outperforming prior model-free methods on 57 Atari games while requiring no game-specific engineering.[2] To illustrate the high-level process:
  1. Observation Encoding: Raw inputs (e.g., board positions or pixel frames) are mapped to a latent state via the representation function.
  2. Latent Simulation: The dynamics function predicts subsequent latent states and immediate rewards based on actions.
  3. Outcome Prediction: The prediction function evaluates values and policies from simulated states, informing tree-based planning.
This streamlined pipeline underscores MuZero's departure from explicit modeling, fostering greater flexibility and scalability in reinforcement learning applications.[2]

Technical Architecture

Learned Model Components

MuZero's learned model consists of three primary neural network functions—representation, dynamics, and prediction—that collectively approximate the environment's dynamics without explicit rules, enabling planning in latent space. These components form a single parameterized network θ\theta, allowing the agent to predict future states, rewards, policies, and values based on observations and actions. By learning these representations end-to-end, MuZero achieves superhuman performance across diverse domains like board games and Atari, surpassing model-free methods while avoiding the need for a full environment simulator.[2] The representation function hθh_\theta encodes raw observations into an initial hidden state s0s_0, capturing the current environment configuration in a compact latent form. For Atari games, it processes a stack of the last 4 grayscale frames (resized to 84x84 pixels) using a convolutional residual network with 16 blocks and 256 filters per plane, with downsampling in the residual blocks before the prediction head. In board games such as Go, it stacks the last 8 planes (19x19 for Go), while chess uses 100 planes to represent the longer history. This function initializes the latent trajectory for subsequent predictions.[2] The dynamics function gθg_\theta models state transitions by predicting the next hidden state sks_k and immediate reward rkr_k given the current state sk−1s_{k-1} and action aka_k: rk,sk=gθ(sk−1,ak)r_k, s_k = g_\theta(s_{k-1}, a_k). It employs the same residual architecture as the representation function (16 blocks, 256 planes), with the action encoded as a one-hot vector concatenated to the input state. This deterministic update allows MuZero to simulate multi-step trajectories in the latent space, learning implicit dynamics without reconstructing observable states.[2] The prediction function fθf_\theta estimates the policy π\pi (action probabilities) and value vv (expected future rewards) from any latent state sks_k: pk,vk=fθ(sk)p_k, v_k = f_\theta(s_k). It uses a lighter convolutional head similar to AlphaZero, followed by fully connected layers to output the policy as a softmax over actions and the value as a scalar (or distribution for Atari). For Atari, rewards and values undergo a scaling transformation h(x)=sign⁡(x)(∣x∣+ϵx)h(x) = \operatorname{sign}(x) \left( \sqrt{|x|} + \epsilon x \right) where ϵ=0.001\epsilon = 0.001 to represent distributions over a bounded support. These outputs guide action selection and evaluate positions during planning.[2] Training optimizes the combined loss Lt(θ)=∑k=0K[lr(ut+k,rkt)+lv(zt+k,vkt)+lp(πt+k,pkt)]+c∥θ∥2L_t(\theta) = \sum_{k=0}^K [l_r(u_{t+k}, r_k^t) + l_v(z_{t+k}, v_k^t) + l_p(\pi_{t+k}, p_k^t)] + c \|\theta\|^2, where KK is the unroll length (typically 5). The reward loss lrl_r is mean squared error on observed rewards ut+ku_{t+k} (sparse, typically zero except at terminal states for board games); the value loss lvl_v is squared error against n-step returns zt+kz_{t+k} (or cross-entropy for Atari distributions); and the policy loss lpl_p is cross-entropy with MCTS-improved targets πt+k\pi_{t+k}. This multi-component objective, weighted equally, enables efficient learning of dynamics through temporal-difference errors and behavioral targets.[2] Architecturally, the functions share parameters within the unified network θ\theta for computational efficiency, with residual blocks reducing gradient vanishing in deep convolutions. For sequential Atari environments, the representation relies on frame stacking rather than recurrence, processing fixed-history inputs to handle partial observability. In board games, plane stacking preserves history without additional recurrent units, ensuring scalability across domains.[2]

Tree-Based Search and Planning

MuZero's planning mechanism integrates a tree-based search algorithm, specifically an adaptation of the Monte Carlo Tree Search (MCTS) used in AlphaZero, with its learned model to enable decision-making without relying on explicit environment rules. The search begins at a root node constructed from the current observation, where the representation function processes the input to yield an initial hidden state $ s_0 $. From this root, the algorithm performs multiple simulations to explore the action space, expanding the search tree by applying the learned dynamics model to predict subsequent states and rewards, thereby simulating possible trajectories during planning.[2] In each simulation step, actions are selected using a variant of the Predictor Upper Confidence Bound for Trees (PUCT) formula, which balances exploitation of known high-value actions and exploration of promising untried ones. The selection policy is given by:
ak=arg⁡max⁡a[Q(s,a)+P(s,a)⋅∑bN(s,b)1+N(s,a)(c1+log⁡(∑bN(s,b)+c2+1c2))], a_k = \arg\max_a \left[ Q(s, a) + P(s, a) \cdot \frac{\sqrt{\sum_b N(s, b)}}{1 + N(s, a)} \left( c_1 + \log\left(\frac{\sum_b N(s, b) + c_2 + 1}{c_2}\right) \right) \right],
where $ Q(s, a) $ is the average value estimate for action $ a $ in state $ s $, $ P(s, a) $ is the prior policy probability from the learned model, $ N(s, a) $ is the visit count for that action, $ c_1 = 1.25 $ and $ c_2 = 19652 $ are constants tuning exploration, and the sums are over actions $ b $. This process repeats until a terminal state is reached or a maximum depth is attained, after which the simulation bootstraps its value using the model's value function estimate $ v $. Unlike traditional rollouts that interact with the environment, MuZero's simulations are fully model-based, generating trajectories internally by iteratively applying the dynamics function $ g_\theta $ to predict rewards $ r_k $ and next states $ s_k $ from prior states and actions, with the return computed as $ G_k = \sum_{\tau=0}^{l-1-k} \gamma^\tau r_{k+1+\tau} + \gamma^{l-k} v_l $, where $ \gamma $ is the discount factor and $ l $ is the rollout length.[2] After a fixed number of simulations—typically 800 per move for board games like Go, chess, and shogi, and 50 for Atari games—the root node's action values and visit counts inform the final policy. The action to execute is sampled from the improved policy $ \pi(a|s_0) = \frac{N(s_0, a)^{1/\tau}}{\sum_b N(s_0, b)^{1/\tau}} $, where $ \tau $ is a temperature parameter that starts at 1 during early training steps and decays to 0.25 to encourage more deterministic choices as performance improves. This setup allows MuZero to plan effectively even in environments without known rules, as the learned model substitutes for a perfect simulator, marking a key departure from AlphaZero, which relies on domain-specific transition rules for tree expansion and evaluation.[2]

Training Methodology

Self-Play and Data Generation

MuZero generates its training data through a self-play mechanism, where the agent plays games against versions of itself using the current neural network parameters to select actions. During self-play, actions are chosen via Monte Carlo Tree Search (MCTS) guided by the learned model, with the visit counts from the search root used to compute a probability distribution over actions. A temperature parameter is applied to this distribution to encourage exploration, starting at T=1 for the initial 500,000 training steps, decaying to T=0.5 for the next 250,000 steps, and then to T=0.25 for the remaining steps, promoting diverse gameplay early in training while focusing on promising moves later.[1] The trajectories generated during self-play—consisting of observations (such as board states for games like Go or pixel inputs for Atari), actions taken, and rewards received—are stored in a replay buffer for subsequent learning. For board games (Go, chess, and shogi), the buffer maintains the most recent 1 million complete games in an in-memory FIFO queue, ensuring a focus on fresh, high-quality data from recent self-play. In contrast, for Atari games, the buffer holds 125,000 sequences of 200 consecutive timesteps each, with prioritized sampling based on the absolute temporal-difference error to emphasize trajectories with higher learning potential; trajectories are batched and sent to the buffer every 200 moves to balance efficiency and completeness.[1] Action selection during self-play adapts to the domain for scalability: in board games, full MCTS with 800 simulations per move is employed to deeply explore the game tree, while in Atari, a lightweight version uses only 50 simulations every fourth timestep, with the selected action repeated for the next three timesteps to handle the continuous, high-dimensional environment without excessive computation. Episodes terminate according to the environment's rules—such as game completion in board games (yielding final rewards of +1 for win, 0 for draw, and -1 for loss) or natural endings in Atari—or reach a maximum length of 108,000 frames (equivalent to 30 minutes of gameplay) to prevent indefinite runs.[1] To achieve efficiency, self-play is distributed across specialized hardware: 1,000 TPUs generate games in parallel for board games, while 32 TPUs handle Atari self-play, enabling rapid data collection that matches the pace of model updates on 16 TPUs for board games and 8 TPUs for Atari. This setup allows MuZero to produce millions of training examples over the course of 1 million training steps per domain, facilitating superhuman performance without human knowledge of rules.[1]

Optimization and Scalability

MuZero employs an end-to-end learning algorithm that optimizes its neural network components—representation, dynamics, and prediction—jointly through backpropagation-through-time. This approach trains the model on trajectories generated during self-play, minimizing a combined loss function that incorporates terms for reward prediction, value estimation, and policy improvement. The reward loss uses cross-entropy for Atari games to predict scalar rewards at each step, while for board games it is omitted as rewards are sparse and determined at episode ends; the value loss employs squared error (for board games) or cross-entropy (for Atari) to align predictions with bootstrapped estimates; and the policy loss is a cross-entropy between the model's output and an improved target policy derived from Monte Carlo Tree Search (MCTS) visit counts, which refines action selection beyond the raw policy network. An L2 regularization term is added to prevent overfitting.[2] The targets for optimization are carefully designed to leverage planning insights. For rewards and values, n-step returns are used, where n=5 for the initial unroll and extended via reanalysis; these bootstrap from the value function to estimate future outcomes, with a discount factor of 0.997 applied across steps. The policy target is constructed from the MCTS search probabilities proportional to visit counts raised to the power of 1/τ with τ=1, providing a sharper distribution than the untrained policy and enabling the model to learn from simulated improvements. This end-to-end setup allows gradients to flow through the unrolled dynamics model for K=5 steps, updating all parameters simultaneously without separate supervised phases.[2] To facilitate efficient training, MuZero uses the Adam optimizer, processing batches of 2048 sequences for board games (1024 for Atari), and unrolling the dynamics for 5 steps per update. These hyperparameters balance convergence speed and stability, with the model updated every 800 simulations per position during training. Scalability is achieved through distributed computing on Google Cloud TPUs v3, utilizing 16 TPUs for training and up to 1000 for self-play in board games, or 8 and 32 respectively for Atari; this setup matches the total compute of AlphaZero while generalizing to model-free environments without environment-specific rules.[2] Addressing sample efficiency remains a key challenge in model-based reinforcement learning, which MuZero tackles via prioritized experience replay and reanalysis. Prioritized replay stores and samples trajectories nonuniformly based on temporal-difference errors (for Atari) or uniformly (for board games), focusing updates on high-uncertainty experiences from a buffer of recent self-play data. Additionally, reanalysis periodically reruns MCTS on stored positions using the latest model parameters to generate improved value and policy targets, effectively reusing old data without additional environment interactions and boosting learning from past games. These techniques enable MuZero to achieve superhuman performance with compute budgets comparable to prior specialized algorithms.[2]

Performance Evaluation

Results on Board Games

MuZero achieved superhuman performance across the board games of Go, chess, and shogi, matching or slightly surpassing the results of AlphaZero without access to the games' rules or dynamics.[1] In these domains, the algorithm was trained using self-play for 1 million steps, employing 800 simulations per move during both training and evaluation.[2] Performance was assessed via Elo ratings derived from tournaments of 100 games against prior versions of itself or baselines like AlphaZero, using Bayesian logistic regression for rating computation.[2] For Go, MuZero reached an Elo rating of approximately 4,900 after full training, matching AlphaZero's superhuman level.[1] The Elo rating exceeded 3,000 after approximately 500,000 training steps, demonstrating effective learning in a highly combinatorial environment solely from interactions. This performance highlights MuZero's ability to learn effective planning. In chess, MuZero attained superhuman proficiency after 1 million training steps, achieving an Elo rating of around 3,600.[1] It matched AlphaZero's strength, underscoring the algorithm's robustness in tactical and strategic depth without predefined movement rules. For shogi, MuZero similarly reached professional-level play after 1 million steps, equivalent to AlphaZero's superhuman benchmark.[1] It adapted to the game's larger state space and drop rules through learned representations.

Results on Atari Games

MuZero was evaluated on the Atari 2600 suite of 57 games, achieving state-of-the-art performance without knowledge of the environment dynamics. Using 200 million frames of experience per game, the MuZero Reanalyze variant obtained a median human-normalized score of 731% and a mean of 2,169%, outperforming prior methods such as Rainbow DQN (median ~125%) and setting new records in most games.[2] With extended training to 20 billion frames on a subset, performance further improved, demonstrating scalability and sample efficiency. Evaluations used no-op starts or human starts, with results averaged over multiple runs and 1,000 episodes per game. These outcomes highlight MuZero's generalization to imperfect-information settings with partial observability.[1]

Applications and Extensions

Gaming and Simulation Domains

Stochastic MuZero, introduced in 2022, extends the original MuZero algorithm to handle stochastic environments by learning and planning with probabilistic models that capture uncertainty in transitions and rewards.[10] This variant has been applied to games like 2048, where it achieves superhuman performance by sampling multiple possible futures during planning, and backgammon, matching or exceeding state-of-the-art results in this classic two-player zero-sum stochastic game.[10] These extensions demonstrate MuZero's adaptability to imperfect-information settings beyond deterministic board games and Atari benchmarks.[10] For video game AI prototyping, open-source implementations facilitate experimentation in custom environments.[11] The muzero-general repository by Werner Duvaud, released in 2019 and last updated in 2022, provides a flexible PyTorch-based implementation of MuZero that supports Gym environments like CartPole, allowing community-driven experiments in simple control simulations.[11] In physics simulations, the continuous extension of MuZero applies to MuJoCo-based control tasks, such as inverted pendulum and walker environments, where it learns dynamics models for continuous action spaces and achieves state-of-the-art sample efficiency by combining model-based planning with reinforcement learning.[12] These applications highlight MuZero's potential in simulated domains requiring precise control and long-term planning. Recent open-source frameworks, such as MiniZero (updated through 2025), extend MuZero for training in games like Go, Othello, and Atari, supporting variants like Gumbel MuZero.[13][14]

Real-World Adaptations and Variants

MuZero has been adapted for practical applications beyond gaming, demonstrating its potential in optimizing complex sequential decision-making processes in real-world systems. A prominent example is its use in video compression for YouTube, where DeepMind collaborated to enhance the open-source VP9 codec (libvpx). By framing rate control as a reinforcement learning problem with bitrate constraints, MuZero selects encoding decisions to minimize file size while preserving video quality, achieving an average 4% bitrate reduction across diverse videos without perceptible quality loss. This adaptation was tested on a portion of YouTube's live traffic, enabling more efficient delivery of content and reducing overall internet bandwidth demands.[4][15] In resource allocation for data centers, techniques inspired by MuZero and its predecessors, such as AlphaZero, have been applied to Google's Borg cluster management system. AlphaZero analyzed anonymized workload data to predict and optimize resource distribution across thousands of concurrent jobs, reducing underused hardware by up to 19% in simulations by learning scalable rules tailored to varying demands.[16] Key variants of MuZero have emerged to address efficiency and generalization challenges. EfficientZero (2021) builds on MuZero by incorporating self-supervised auxiliary tasks, value prefix prediction, and off-policy corrections, enabling superhuman performance on Atari games with just 100,000 frames—equivalent to two hours of gameplay—reaching 194.3% of mean human performance.[17] This makes it far more sample-efficient than the original MuZero, requiring 500 times less data to match prior benchmarks. From 2023 onward, MuZero's influence extends to algorithms like the Student of Games (updated implementations in 2023), which unifies search, self-play, and game-theoretic reasoning for imperfect-information games such as poker, adapting MuZero's model-free planning to handle hidden states and opponent modeling.[18] Other recent variants include Equivariant MuZero (2023), which enforces symmetry in the learned model for data efficiency in symmetric environments,[19] and Stochastic MuZero (2022), improving robustness in noisy, probabilistic settings.[20] Adapting MuZero to real-world domains faces significant challenges, including scaling to high-dimensional continuous state spaces, where learned models struggle with sim-to-real gaps and partial observability. Safety concerns arise from exploration in physical environments, potentially leading to unintended actions without constrained rewards or human oversight. Additionally, computational demands of Monte Carlo Tree Search limit deployment on resource-constrained hardware, necessitating further efficiency improvements for broad applicability.[21][22][23]

Impact and Future Directions

Academic and Industry Reactions

Upon its release in 2019, MuZero garnered significant praise from the reinforcement learning community for advancing model-based planning without explicit environmental rules. David Silver, who leads DeepMind's reinforcement learning research group, highlighted MuZero's ability to "start from nothing, and just through self-play, learn how to play games," positioning it as a key progression toward more general AI agents.[24] The seminal paper introducing MuZero has been cited over 5,000 times as of 2025, underscoring its foundational impact on subsequent research in planning and decision-making algorithms. Critics, however, pointed to MuZero's substantial computational demands as a barrier to accessibility, with training requiring clusters of Google's third-generation tensor processing units (TPUs)—8 for the learner and 800 for self-play generation—making replication challenging outside major labs.[1] Additionally, the original formulation was limited to environments with discrete action spaces, restricting its immediate applicability to continuous control tasks like robotics.[2] In industry, Google DeepMind applied MuZero to real-world optimization, notably integrating it into YouTube's VP9 video codec for rate control, yielding an average 6.28% bitrate reduction while maintaining video quality and saving significant bandwidth costs.[4] This marked an early transition from game mastery to practical deployment. Media outlets, including a feature in Nature, celebrated MuZero's rule-free learning as a milestone toward artificial general intelligence, with coverage emphasizing its superhuman performance across diverse domains.[1]

Related Algorithms and Ongoing Research

MuZero has inspired several extensions in model-based reinforcement learning (RL), notably DreamerV3, which advances world model techniques to handle over 150 diverse control tasks across continuous and discrete domains using a single hyperparameter configuration.[25] Published in Nature in 2025, DreamerV3 builds on MuZero's latent model learning by incorporating RSSM (Recurrent State-Space Model) architectures for improved prediction accuracy and sample efficiency in real-world robotics and simulation environments.[26] Another related development is AlphaEvolve, a 2025 DeepMind system that leverages self-improving mechanisms akin to MuZero's iterative planning to evolve code for algorithmic discovery, achieving superior performance in kernel optimization and mathematical problem-solving through an autonomous LLM-driven pipeline.[27][28] Ongoing research integrates MuZero-like model-based RL with large language models (LLMs) for hybrid planning, as seen in variants of GPT models that employ RL-enhanced chain-of-thought reasoning to simulate multi-step decision-making in complex tasks. For instance, systems like OpenAI's o1 model use planning modules inspired by RL world models to improve logical inference and long-horizon strategies.[29] Applications extend to climate modeling, where model-based RL principles are applied to optimize adaptive strategies under uncertainty.[30] Key research gaps persist in adapting MuZero to continuous action spaces and multi-agent settings, where extensions like Sampled MuZero address complex action sampling but still face scalability challenges in high-dimensional environments.[31] Recent arXiv preprints from 2023–2025 explore MuZero variants for quantum simulations, such as RL-based architecture search for quantum circuits, highlighting potential in optimizing variational quantum algorithms though limited by noise and entanglement modeling.[32] Future directions emphasize integrating MuZero's planning with diffusion models for generative trajectory synthesis, enabling more flexible exploration in stochastic environments via diffusion-based policy optimization.[33] Ethical considerations in RL, including bias mitigation in self-play data generation and fairness in multi-agent deployments, are increasingly prioritized to ensure responsible scaling of MuZero-inspired systems. As of November 2025, no direct sequel to MuZero has been released by DeepMind, yet its algorithms remain foundational in RL pipelines at DeepMind for tasks like game AI.

References

  1. Mastering Atari, Go, chess and shogi by planning with a learned model
  2. Mastering Atari, Go, Chess and Shogi by Planning with a Learned ...
  3. MuZero: Mastering Go, chess, shogi and Atari without rules
  4. MuZero's first step from research into the real world
  5. Part 2: Kinds of RL Algorithms — Spinning Up documentation
  6. Reinforcement Learning: Advancements, Limitations, and Real ...
  7. [1712.01815] Mastering Chess and Shogi by Self-Play with a ... - arXiv
  8. Recurrent Experience Replay in Distributed Reinforcement Learning
  9. AlphaZero and MuZero - Google DeepMind
  10. Planning in Stochastic Environments with a Learned Model
  11. L21- MuZero for Point Robotic Motion Planning - CRML
  12. [PDF] AI Programming (IT-3105) Spring 2025 Main Project: A MuZero ...
  13. werner-duvaud/muzero-general - GitHub
  14. Continuous Control for Searching and Planning with a Learned Model
  15. MuZero with Self-competition for Rate Control in VP9 Video ... - arXiv
  16. MuZero, AlphaZero, and AlphaDev: Optimizing computer systems
  17. [2111.00210] Mastering Atari Games with Limited Data - arXiv
  18. Student of Games: A unified learning algorithm for both perfect and ...
  19. Equivariant MuZero - Google DeepMind
  20. DeepMind & UCL's Stochastic MuZero Achieves SOTA Results in ...
  21. (PDF) An empirical investigation of the challenges of real-world ...
  22. Challenges of real-world reinforcement learning - ResearchGate
  23. Beyond AlphaZero: New AI capabilities in MuZero - Leisureguy
  24. Deep Reinforcement Learning from AlphaGo to AlphaStar - YouTube
  25. Mastering diverse control tasks through world models | Nature
  26. [2301.04104] Mastering Diverse Domains through World Models
  27. A Gemini-powered coding agent for designing advanced algorithms
  28. AlphaEvolve: A coding agent for scientific and algorithmic discovery
  29. Workshop on Reasoning and Planning for Large Language Models
  30. Reinforcement learning–based adaptive strategies for climate ...
  31. Reinforcement Learning | Climate Change AI
  32. [2104.06303] Learning and Planning in Complex Action Spaces
  33. [2510.12253] Diffusion Models for Reinforcement Learning - arXiv
User Avatar
No comments yet.