Hubbry Logo
Sierra (supercomputer)Sierra (supercomputer)Main
Open search
Sierra (supercomputer)
Community hub
Sierra (supercomputer)
logo
8 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Sierra (supercomputer)
Sierra (supercomputer)
Sierra (supercomputer)
View on Wikipedia
from Wikipedia
Sierra
ActiveSince 2018[1]
OperatorsNational Nuclear Security Administration
LocationLawrence Livermore National Laboratory
ArchitectureIBM POWER9 CPUs
Nvidia Tesla V100 GPUs
Mellanox EDR InfiniBand[2]
Power11 MW
Operating systemRed Hat Enterprise Linux[3]
Memory2–2.4 PiB[1]
Speed125 petaflops (peak)[2]
RankingTOP500: 20, June 2025
PurposeNuclear weapon simulations[4]
Websitehpc.llnl.gov/hardware/compute-platforms/sierra

Sierra or ATS-2 is a supercomputer built for the Lawrence Livermore National Laboratory for use by the National Nuclear Security Administration as the second Advanced Technology System. It is primarily used for predictive applications in nuclear weapon stockpile stewardship, helping to assure the safety, reliability, and effectiveness of the United States' nuclear weapons.

Sierra is very similar in architecture to the Summit supercomputer built for the Oak Ridge National Laboratory. The nodes in Sierra are Witherspoon IBM S922LC OpenPOWER servers with two GPUs per CPU and four GPUs per node. These nodes are connected with EDR InfiniBand. In 2019 Sierra was upgraded with IBM Power System AC922 nodes.[5][6]

Sierra is composed of 4,474 nodes, 4,284 of which are compute nodes. Each node has 256GB of RAM, 44 IBM POWER9 cores spread across two physical sockets, and Four Nvidia Tesla V100 GPUs, each providing 16GB of VRAM. This gives the complete system 8,948 CPUs, 17,896 GPUs, 1.14 PB of RAM, and 286 TB of VRAM.[7]

Sierra has consistently appeared on the Top500 list, peaking at second place in November 2018.[8] As of November 2023, it is in tenth place.[9] Only 4.6 petaflops of its performance come from its CPUs, with the large majority (120.9 petaflops) coming from the Tesla GPUs.[7]

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Sierra (supercomputer)

View on Grokipedia
from Grokipedia
Sierra is an IBM-built supercomputer deployed at the (LLNL) in 2018 for the U.S. (NNSA), primarily supporting the Advanced Simulation and Computing (ASC) program for nuclear simulations. Equipped with 4,320 compute nodes—each featuring two POWER9 CPUs and four V100 GPUs—Sierra spans 240 racks across 7,000 square feet, delivering a peak performance of 125 petaFLOPS while consuming 11 megawatts of power. Upon activation, it ranked third on the list of supercomputers, later ascending to second place with a Linpack benchmark score of 94.6 petaFLOPS, enabling six to ten times the computational throughput of its predecessor, Sequoia, for complex, large-scale scientific modeling without physical nuclear testing. As a pre-exascale system, Sierra facilitated breakthroughs in high-fidelity simulations of nuclear weapons physics, , and , underpinning the U.S. certification of the nuclear arsenal's reliability and safety through predictive modeling rather than explosive experiments. By 2025, Sierra had been decommissioned at LLNL, succeeded by more advanced systems like , reflecting the rapid evolution in architectures toward exascale capabilities.

Development and Procurement

Origins and Funding

The U.S. Department of Energy established the Collaboration for Oak Ridge, Argonne, and Livermore () program in early 2014 to coordinate supercomputing acquisitions across its national laboratories, with the goals of optimizing investments, streamlining procurement processes, and reducing development costs for advanced systems. On November 14, 2014, the DOE announced $325 million in funding for two pre-exascale supercomputers under , allocating Sierra to (LLNL) as a successor to the Sequoia system, expected to deliver at least seven times greater computational power. Primary funding for Sierra originated from the DOE's National Nuclear Security Administration (NNSA) budget, directed toward the Advanced Simulation and Computing (ASC) program to enable simulation-based of the U.S. nuclear stockpile in adherence to international test ban treaties prohibiting full-scale underground testing. This initiative addressed the need for high-fidelity modeling of nuclear weapons effects, thermonuclear processes, and materials behavior under extreme conditions, compensating for the absence of empirical data from live tests since 1992. LLNL engaged through 's competitive bidding framework, which emphasized innovative architectures capable of scaling toward while meeting NNSA's simulation requirements. The selection process prioritized vendors offering integrated solutions for defense simulations, culminating in 's contract to deliver Sierra by late 2017. This procurement model facilitated cost efficiencies and technology sharing across DOE labs, setting a precedent for subsequent acquisitions.

Design and Construction

Sierra's architecture employs a heterogeneous design integrating POWER9 central processing units with V100 graphics processing units to optimize performance for compute-intensive tasks requiring high parallelism and data throughput. Each compute node incorporates two 22-core processors operating at 3.45 GHz, delivering 44 cores total, alongside four V100 GPUs, each with 16 GB of high-bandwidth memory, and 256 GB of system RAM using DDR4 configuration. This node-level fusion, facilitated by IBM's interconnects, enables direct high-speed data exchange between CPUs and GPUs, addressing bottlenecks in memory-bound computations through elevated bandwidth exceeding traditional PCIe interfaces. IBM led the construction of Sierra at Lawrence Livermore National Laboratory, assembling 4,320 compute nodes across 240 racks within a 7,000-square-foot footprint. Engineering choices prioritized scalable integration of POWER9's multi-chip module design with Volta GPU tensor cores, tailored for workloads demanding simultaneous handling of vector and scalar operations in complex modeling scenarios. The build process, commencing shipments in 2017, incorporated redundant power and cooling systems to sustain operational integrity under sustained high loads. Fault-tolerant elements in the network fabric, utilizing a fat-tree topology with Mellanox EDR , provide resilience against node failures and link disruptions, ensuring minimal interruption in distributed processing across the cluster. These design decisions stem from requirements for robust error handling and checkpointing in extended runs, with POWER9's coherence protocols further enhancing data consistency in heterogeneous environments.

Deployment Timeline

The U.S. Department of Energy's National Nuclear Security Administration awarded a $325 million contract to IBM on November 14, 2014, to develop Sierra as part of the CORAL initiative, with delivery targeted for 2017 at Lawrence Livermore National Laboratory (LLNL). Installation of Sierra's components began in late 2017, comprising approximately 260 racks equipped with IBM Power9 processors and NVIDIA Tesla V100 GPUs, marking a phased rollout to integrate the heterogeneous system into LLNL's data center infrastructure. Sierra achieved initial operational status in early 2018, undergoing acceptance testing and calibration to validate its performance for nuclear security simulations. By June 2018, it demonstrated 94.6 petaFLOPS on the High-Performance Linpack benchmark, securing the third position on the list of the world's fastest supercomputers. Full operational capability was realized by mid-2018, enabling seamless integration alongside LLNL's predecessor system, Sequoia, to enhance high-fidelity modeling within the laboratory's ecosystem. In October 2018, LLNL formally unveiled Sierra, confirming its role in supporting the National Nuclear Security Administration's three laboratories for advanced simulations, with subsequent remeasurement in November 2018 elevating its ranking to second place at 94.6 petaFLOPS sustained performance. This timeline reflected rigorous validation phases to ensure reliability before production-scale use.

Technical Specifications

Hardware Architecture

The Sierra supercomputer comprises 4,320 compute nodes, each equipped with two POWER9 processors totaling 44 cores, four V100 GPUs, and 256 GB of DDR4 memory, resulting in over 190,000 CPU cores, more than 17,000 GPUs, and approximately 1.1 petabytes of total system memory. This configuration, based on AC922 servers, leverages the heterogeneous CPU-GPU architecture to deliver high peak theoretical performance of 125 petaFLOPS. Intra-node connectivity utilizes for direct, high-bandwidth communication between the CPUs and V100 GPUs, providing up to 900 GB/s bidirectional bandwidth per GPU to enhance transfer efficiency for compute-intensive workloads. Inter-node scaling is achieved through Mellanox EDR interconnects, forming a non-blocking fat-tree that supports low-latency, high-throughput messaging across the cluster. Power consumption for the full system is limited to 11 megawatts, with advanced liquid cooling systems employing high-performance cold plates to directly water-cool both CPUs and GPUs, enabling dense rack configurations of up to 40 nodes per rack while maintaining operational temperatures. This cooling approach prioritizes thermal management for sustained high-density computing without compromising component reliability or performance.

Software Stack and Programming

The Sierra supercomputer runs a Linux-based operating system, derived from and customized for , providing a stable foundation for distributed workloads across its Power9 CPU and GPU nodes. This environment integrates the Spectrum Computing suite, encompassing interfaces, compilers, and optimization tools tailored for scalable scientific simulations. Programming on Sierra emphasizes hybrid parallel models, combining (MPI) for distributed-memory communication between nodes with directives for shared-memory threading within nodes, enabling efficient utilization of the system's multi-core processors and accelerators. NVIDIA's programming model is central for GPU offloading, allowing developers to port compute kernels to the Volta architecture GPUs for accelerated floating-point operations in domains like and . Compilers such as XL Fortran/C and GCC variants support these paradigms, with optimizations for vectorization and prefetching to minimize latency in large-scale runs. The stack is optimized for Advanced Simulation and Computing (ASC) program codes, including HYDRA for multi-physics hydrodynamics and radiation transport simulations, and ALE3D for arbitrary Lagrangian-Eulerian modeling of material deformation under extreme conditions. Configurations enforce deterministic reproducibility through fixed random seeds, consistent floating-point semantics, and , ensuring bit-for-bit identical outputs essential for NNSA certification of nuclear simulations without physical testing. Subsequent enhancements incorporated frameworks leveraging GPU tensor cores, such as within the CogSim framework, to surrogate complex physics sub-models and reduce computational costs in iterative workflows like analysis. This integration augments deterministic codes by providing probabilistic , with training accelerated via CUDA-enabled libraries on Sierra's heterogeneous architecture.

Scalability and Interconnects

Sierra's interconnect infrastructure is based on a dual-rail Mellanox EDR network delivering 100 Gb/s bandwidth per rail, employing ConnectX-5 host channel adapters and Switch-IB2 directors in a fat-tree topology. This configuration ensures low-latency communication across the system's 4,320 compute nodes and integrates with parallel storage subsystems totaling over 100 PB, enabling seamless data movement for distributed workloads. The fabric incorporates In-Network Computing offload capabilities, such as Mellanox's SHARP technology, which reduces CPU overhead for collective operations and enhances overall system throughput during large-scale data exchanges. The design supports weak scaling to full-system utilization for petascale applications, as evidenced by benchmarks like HPCG and miniFE that maintain parallel efficiency through progressive node allocation without proportional increases in communication overhead. Strong scaling tests on Sierra further validate sustained performance across subsets of nodes, with the InfiniBand's non-blocking fabric minimizing contention in irregular communication patterns typical of scientific simulations. These features differentiate Sierra by prioritizing system-level integration over isolated node performance, allowing applications to exploit the entire cluster for problems requiring massive parallelism. Fault management is embedded in the interconnect and runtime environment, with hardware-level error detection via InfiniBand's reliable transport protocols and software mechanisms for process recovery that prevent full job abortion upon isolated node failures. Recovery abstractions tested on Sierra reduce restart times for transient faults, enabling resilience in extended runs—such as multi-week stewardship codes—by localizing impacts and resuming from checkpoints without global rescheduling. This approach, combined with proactive monitoring, sustains operational continuity at high utilization rates, where empirical tests show over 90% efficiency in scaled workloads despite occasional hardware events.

Performance Metrics

Benchmark Achievements

Sierra achieved a measured performance of 94.6 petaFLOPS (Rmax) on the High Performance Linpack (HPL) benchmark, as verified in the November 2018 TOP500 list. This result reflected optimizations in the benchmark execution following its initial deployment, elevating its standing from 71.6 petaFLOPS recorded in the June 2018 list. The HPL score represented approximately 75.7% efficiency relative to Sierra's theoretical peak performance of 125 petaFLOPS, demonstrating effective utilization of its IBM Power9 CPUs and NVIDIA Volta GPUs in parallel floating-point operations. Subsequent TOP500 evaluations through 2020 confirmed sustained HPL performance at 94.6 petaFLOPS, with no significant degradation reported in official measurements despite increasing competition from emerging systems. This stability was attributed to ongoing software tuning and system optimizations by engineers, as documented in Department of Energy-affiliated reports, enabling consistent benchmark reproducibility across multiple runs. In the June 2020 list, for instance, Sierra registered the same Rmax value, underscoring the reliability of its architecture for standardized testing.

Energy Efficiency and Sustainability

The Sierra supercomputer consumes approximately 11 megawatts at peak operation, supporting its 125 petaFLOPS theoretical peak performance and yielding an efficiency of about 11.4 gigaFLOPS per watt. This metric reflects the system's hybrid architecture, where GPU acceleration—primarily from over 17,000 V100 GPUs contributing 120.96 petaFLOPS—dominates computational throughput, vastly outperforming CPU-only predecessors like Sequoia in power-normalized output by a factor of roughly five. Such GPU-centric design enables empirically lower energy per floating-point operation compared to traditional CPU-based systems for Sierra's targeted workloads, as heterogeneous aligns compute with demands in nuclear stewardship tasks, reducing overall joules expended per result. Cooling relies on direct water-cooled cold plates for all CPUs and GPUs, integrated into the Power9 nodes, which sustains high densities without excessive air-handling overhead while maintaining thermal thresholds critical for continuous operation. Power management features emphasize reliability over incremental efficiency tweaks, such as interconnects for low-latency GPU data flow that minimize idle cycles, prioritizing uptime in mission-critical environments over speculative green optimizations that could compromise availability. The facility supports this with 7,200 tons of dedicated , ensuring scalability without proportional power escalation.

Comparative Rankings

Sierra attained its peak position of second on the TOP500 list in November 2018, delivering 94.6 petaFLOPS on the High Performance LINPACK benchmark, behind only the Summit supercomputer at Oak Ridge National Laboratory. This ranking reflected Sierra's robust heterogeneous architecture combining IBM POWER9 CPUs and NVIDIA V100 GPUs, which sustained its No. 2 spot through multiple list updates, including November 2019. In June 2020, Japan's Fugaku supercomputer claimed the top ranking with 415.5 petaFLOPS, displacing Summit to second and Sierra to third; Sierra's position underscored the enduring competitiveness of U.S. GPU-accelerated systems against CPU-centric international rivals like Fugaku, particularly in workloads demanding high memory bandwidth and parallel processing capabilities. Subsequent lists saw Sierra maintain a top-tier presence into 2021, but its ranking eroded post-2020 amid the emergence of exascale systems. The introduction of at Oak Ridge in June 2022, the first to exceed 1 exaFLOPS, accelerated Sierra's descent outside the top five, with further displacement by systems like Aurora in 2023. By mid-2024, Sierra ranked in the low twenties on the , a decline attributable to generational leaps in compute density and interconnect speeds rather than obsolescence in its core design, as it continued operational utility at through 2023 and 2024 for classified and scientific workloads.

Primary Applications

Nuclear Stockpile Stewardship

Sierra supercomputer supports the U.S. Program (SSP), launched in the 1990s after the 1992 nuclear testing moratorium, by enabling predictive simulations that certify the safety, security, and effectiveness of aging nuclear warheads without underground explosive tests. These computations model complex phenomena such as plutonium pit degradation over decades, transport, and hydrodynamic instabilities in weapon primaries, drawing on empirical data from prior tests and ongoing subcritical experiments at sites like the Nevada National Security Site. Sierra's architecture, with its peak performance exceeding 125 petaflops, processes these multi-physics integrations—combining radiation transport, , and equation-of-state models—at resolutions unattainable on prior systems like Sequoia. Key advancements include routine high-fidelity 3D simulations of boost processes, where tritium-deuterium fusion enhances fission yield, allowing certification of warhead performance margins within specified uncertainties. For instance, Sierra has executed full-weapon-system models for programs, such as the Alt 370, resolving aging effects on high-explosive lenses and tamper materials that could impact yield and reliability. These efforts have supported annual SSP assessments since 2018, providing quantitative confidence in stockpile viability—typically exceeding 95% for key metrics like minimum yield—validated against decades of archived hydrodynamic and radiographic data. By accelerating 3D multi-physics runs up to 10 times faster than predecessors, Sierra minimizes reliance on resource-intensive subcritical tests while enhancing predictive fidelity, as evidenced by convergence in simulated neutronics and thermonuclear burn rates matching historical benchmarks. This computational capability underpins deterrence sustainability, ensuring virtual verification of weapon integrity amid treaty constraints like the Comprehensive Test Ban Treaty preparatory regime, without which empirical degradation models would lack sufficient resolution for credible assessments.

Broader Scientific Simulations

Sierra's computational capabilities have extended to non-classified scientific domains, enabling high-fidelity simulations of complex phenomena such as turbulent flows and high-energy-density (HED) physics with applications in and . In , researchers leveraged Sierra for (DSMC) modeling of turbulent over riblet surfaces, utilizing 1,000 nodes to resolve molecular-scale effects in rarefied gases, which informs hypersonic and microscale flow behaviors beyond defense contexts. Similarly, large-scale simulations on Sierra examined reaction-induced deviations from continuum Navier-Stokes equations in turbulent reacting flows, employing 1,500 nodes with advanced GPU acceleration to capture subgrid-scale physics relevant to and processes. In and equation-of-state validations, Sierra supported predictive modeling of material responses under extreme conditions, including high-pressure fluid flows where tensor-decomposed reduced-order models enhanced efficiency and accuracy for property calculations, facilitating broader applications in and . These efforts draw from HED science frameworks that overlap with astrophysical phenomena, such as dynamics and planetary interiors, where Sierra's pre-exascale performance allowed validation of multi-physics models integrating and . Collaborations with initiatives have amplified spillover benefits, notably in (ICF) energy research, where Sierra's integration with experimental data from the (NIF) enabled predictive simulations of implosion dynamics and yield performance, contributing to the anticipation of ignition achieved on , 2022—yielding gain exceeding unity for the first time. From 2018 to 2023, such runs produced datasets and informed peer-reviewed outputs in and HED fields, demonstrating how defense-funded architectures advance civilian-oriented breakthroughs in predictive modeling. While direct modeling allocations remain limited due to prioritization of core missions, Sierra's and materials simulations provide foundational tools adaptable to atmospheric and geophysical .

National Security Simulations

Sierra's computational power has enabled detailed simulations of nuclear weapon effects across varied environments, supporting national security objectives by modeling phenomena such as blast dynamics, radiation propagation, and electromagnetic pulses that could arise in conflict scenarios. These capabilities, part of the NNSA's Advanced Simulation and Computing program, allow analysts to predict outcomes of potential nuclear events with greater fidelity than prior systems, informing defensive postures and deterrence strategies without reliance on underground testing, which has been prohibited since 1992. For instance, Sierra's heterogeneous architecture, combining IBM Power9 CPUs and NVIDIA V100 GPUs, processes multiphysics models at scales exceeding 100 petaFLOPS, enabling resolutions that capture turbulent mixing and material responses critical to assessing threat trajectories. In threat assessment applications, Sierra integrates physics-based codes to evaluate adversary weapon performance hypotheticals, drawing on validated models to simulate yields, delivery systems, and countermeasures in realistic geopolitical contexts. This approach yields verifiable predictions grounded in empirical data from historical tests and subcritical experiments, reducing uncertainties in strategic planning; simulations that once required weeks on older platforms complete in hours, accelerating decision cycles for policymakers. Such efficiency contrasts with experimental alternatives, which are logistically constrained and costly, thereby strengthening U.S. confidence in response options against evolving foreign capabilities documented in intelligence assessments. For scenarios, Sierra supports impact modeling relevant to interceptors, simulating transfers and debris fields at speeds exceeding 10 km/s to refine system architectures. These classified runs, leveraging Sierra's peak performance of 125 petaFLOPS achieved in 2018, provide causal insights into failure modes and optimizations, empirically demonstrating improved hit-to-kill probabilities over analytic approximations. Overall, Sierra's role in these simulations has empirically bolstered by enabling data-driven iterations that outpace adversarial development timelines, as evidenced by its contributions to annual stockpile assessments extended to broader deterrence contexts.

Achievements and Impacts

Key Scientific Breakthroughs

Sierra's advanced computational power resolved longstanding uncertainties in aging models by enabling multi-scale simulations that integrated microstructural evolution with macroscopic behavior, validated against declassified historical nuclear test data from the U.S. program. These simulations addressed specific issues in plutonium pit assessments, such as phase transformations and void formation over decades, enhancing predictive accuracy for long-term material degradation without reliance on new tests. In inertial confinement fusion research, Sierra supported high-fidelity multiphase flow simulations via the HYDRA radiation-hydrodynamics code, modeling turbulent mixing, implosion asymmetries, and material interfaces in 3D spherical geometries. This capability reduced simulation times for complex ICF implosions from weeks to hours, allowing exploration of high-dimensional design spaces through hundreds of thousands of runs and refinement of turbulence models with quantified reductions in predictive uncertainties via Bayesian inference and machine learning integration. These efforts directly contributed to the December 5, 2022, at the , where Sierra-driven pre-shot predictions achieved a 50.2% probability of success, yielding 3.15 MJ fusion output from 2.05 MJ input and demonstrating scientific with narrower in yield forecasts compared to prior campaigns. Resulting insights have been detailed in peer-reviewed publications, including analyses in Physics of Plasmas on simulation-validated ICF physics.

Contributions to U.S.

The Sierra supercomputer, operational from to its decommissioning in early 2023, played a pivotal role in the U.S. National Nuclear Security Administration's (NNSA) Program by enabling high-fidelity simulations essential for annually certifying the reliability and effectiveness of the nation's approximately 3,800 nuclear warheads without conducting physical tests. This capability upheld the U.S. commitment to the 1992 nuclear testing moratorium, allowing laboratory directors at Lawrence Livermore, Los Alamos, and to deliver formal annual assessments to the President and affirming stockpile safety and performance. Sierra's sustained performance exceeding 100 petaflops facilitated multi-physics simulations of weapon aging, material degradation, and yield under extreme conditions, providing empirical data that replaced empirical testing data lost since 1992. In a counterfactual absent Sierra-level computational power, stewardship of the would necessitate resuming underground nuclear tests to validate weapon performance, contravening the de facto moratorium and the (signed by the U.S. in 1996), potentially eroding international non-proliferation norms and inviting escalatory responses from adversaries like or . NNSA officials have emphasized that advanced systems like Sierra are indispensable for maintaining deterrence credibility, as they generate predictive models grounded in validated physics that ensure warhead functionality despite decades without live detonations. This computational approach has directly supported U.S. nuclear posture by quantifying uncertainties in stockpile viability, thereby sustaining a reliable second-strike capability critical to extended deterrence alliances. Sierra's simulations further enhanced by informing verifiability, such as modeling treaty-compliant inspections and forensic analysis of potential violations, which bolsters U.S. negotiating leverage in bilateral talks. For instance, its capacity for 3D, high-resolution renders of weapon subsystems has contributed to in dismantlement verification protocols, reducing risks of undetected by peer competitors. These outputs, integrated into NNSA's broader mission, have empirically preserved deterrence stability amid geopolitical tensions, averting the need for rebuilds that could signal weakness or provoke arms races.

Technological Innovations Enabled

Sierra's hybrid architecture, combining IBM POWER9 CPUs with over 17,000 NVIDIA Tesla V100 GPUs interconnected via , pioneered the acceleration of legacy scientific simulation codes through GPU offloading. The RAJA abstraction layer emerged as a key innovation, enabling portability for applications comprising millions of lines of code—such as ALE3D, ARES, and MFEM—by encapsulating platform-specific optimizations like kernels without necessitating full rewrites. This facilitated 5-20x speedups in complex 3D simulations, such as , reducing computation times from 30 days on predecessor systems to 60 hours. Supporting tools like and CHAI automated memory allocation and data movement between CPU and GPU domains, minimizing developer overhead while maximizing heterogeneous performance. Refinements in hybrid programming paradigms, favoring RAJA over for C++-based codes due to its support for incremental GPU adoption and cross-backend compatibility (e.g., , HIP), addressed profiling challenges in scalable GPU environments through enhanced tools like nvprof and HPCToolkit. These models, validated on Sierra, informed industry-wide practices in , extending to AI training pipelines that integrate similar CPU-GPU orchestration for large-scale model optimization. In data management, Sierra's IBM Spectrum Scale file system provided 154 petabytes of storage with 1.54 TB/s read/write bandwidth across 24 racks of Elastic Storage Servers, supporting efficient I/O for petabyte-scale datasets and up to 100 billion files per filesystem. This capability, bolstered by 100-gigabit networking, ensured high-throughput data movement critical for data-intensive workflows, yielding measurable efficiency gains over prior CPU-centric systems. As the inaugural NNSA production supercomputer with GPU-centric design, Sierra's code modernization strategies—emphasizing abstraction and memory efficiency—directly influenced exascale successors like , paving the way for sustained performance scaling in heterogeneous architectures targeting 1.5 exaFLOPS.

Criticisms and Debates

Resource Allocation and Opportunity Costs

The development of Sierra was funded by the U.S. Department of Energy's (NNSA) through the Collaboration for Oak Ridge, Argonne, and Livermore () program, with an allocation of $325 million to construct both Sierra at and the companion Summit system at . This investment supported Sierra's deployment in 2018, emphasizing mission-specific hardware from and to achieve sustained performance exceeding 100 petaflops for classified simulations. Opportunity cost considerations have centered on whether NNSA's prioritization of nuclear security computing foregoes equivalent gains in civilian domains, such as large-scale climate or epidemiological modeling that could address public health challenges. Proponents of redirection argue that comparable funding applied to open-access systems might accelerate non-defense simulations, given Sierra's restricted access under security classifications that limit broader academic utilization. However, NNSA assessments highlight empirical returns, including computational advances that have reduced nuclear stockpile maintenance costs by up to $2 billion through enhanced predictive modeling, demonstrating a targeted return on investment in core mandate areas. Sierra's design underscores government efficiency in constrained, high-stakes applications compared to deployments, where companies like Meta and operate AI clusters costing hundreds of millions annually but often without equivalent sustained scientific throughput due to diffuse commercial objectives. Dual-use spillovers from defense-funded HPC, including scalable architectures and algorithms refined on Sierra, have informed civilian technologies, such as advanced GPU utilization later adopted in exascale prototypes, yielding indirect economic benefits beyond initial security-focused expenditures. These factors, per DOE program evaluations, position such allocations as yielding compounded value through technology maturation not readily replicated in fragmented private investments.

Ethical and Policy Controversies

Disarmament advocates have criticized the Stockpile Stewardship Program (SSP), which utilizes Sierra for nuclear simulations, as perpetuating a weapons-oriented scientific culture that hinders global efforts. Organizations such as the Natural Resources Defense Council (NRDC) have argued that advanced computational capabilities like those provided by Sierra enable "virtual testing" that maintains or potentially expands design expertise, subverting commitments under treaties like the (CTBT) by preserving the infrastructure for future weapon innovations rather than facilitating stockpile reductions. These critiques, often from groups with a historical emphasis on non-proliferation, posit that such programs signal to proliferators that nuclear powers retain active stewardship capacities, complicating diplomatic pushes for multilateral . Proponents of the SSP, including Department of Energy officials and analysts, rebut these claims by emphasizing that Sierra's simulations ensure the safety, reliability, and effectiveness of the existing U.S. without resuming full-scale testing, thereby complying with U.S. moratoriums and obligations while upholding deterrence stability against peer adversaries. This approach, they argue, supports verifiable policy goals of "stockpile-only" maintenance under first-principles of , where empirical validation through subcritical experiments and historical data confirms aging warhead performance without new production. Conservative-leaning policy perspectives, such as those from , further contend that abandoning such would erode U.S. and invite instability, prioritizing realist security needs over idealistic repurposing of resources. Policy debates also center on the classified nature of Sierra's work, which restricts external and fosters concerns about unverified assumptions in simulations critical to decisions. Critics, including transparency advocates, highlight that in programs like the Advanced Simulation and Computing (ASC) initiative limits broader scientific scrutiny, potentially embedding biases or errors insulated from open debate, as noted in analyses calling for increased and academic engagement. Supporters counter that internal validations, periodic declassifications of non-sensitive results, and cross-lab collaborations provide rigorous checks, with empirical successes in predicting behavior demonstrated through aligned subcritical tests since the SSP's in 1995. Left-leaning calls for redirecting supercomputing toward civilian applications like climate modeling contrast with right-leaning assertions that nuclear prioritization reflects causal realities of geopolitical threats, where deterrence empirically prevents conflict more effectively than symbolic gestures.

Technical Limitations and Failures

Despite its advanced hybrid CPU-GPU architecture featuring POWER9 processors and V100 GPUs, Sierra encountered scaling bottlenecks in certain solver implementations, particularly within the Sierra/SD structural dynamics code. For problems exceeding 1 billion (DOFs), the coarse problem size grew disproportionately with processor count, dominating solution times beyond approximately 1,000 processors and leading to inefficient strong scaling. Communication-intensive orthogonalization steps further exacerbated these issues by exhibiting poor due to increased overhead at large scales. Memory constraints and 32-bit overflows limited for meshes larger than 2.1 billion DOFs, necessitating solver adjustments such as switching to smaller coarse spaces or employing multi-level parallel direct solvers like MKL to mitigate bottlenecks. These challenges were addressed through application-specific patches and consultations with development teams, enabling solutions up to 1.5 billion DOFs on 18,432 processors, though performance impacts persisted for weakly scaled workloads. GPU-related reliability issues, common in multi-GPU nodes akin to Sierra's configuration of four V100 GPUs per node, included frequent software and failures such as malfunctions, which accounted for a significant portion of in comparable systems. While hardware MTBF for individual GPUs reached around 226 hours in optimized setups, system-wide failures often involved multiple GPUs simultaneously, with mean time to recovery averaging 55 hours due to diagnostic and repair complexities. These factors contributed to operational interruptions, underscoring vulnerabilities in GPU-heavy architectures for sustained extreme-scale computing. Porting large multiphysics codes—often comprising millions of lines—to Sierra's GPU-accelerated environment required substantial refactoring, as many legacy applications were not inherently GPU-optimized, leading to inefficiencies in workload balancing between CPUs and GPUs. Tools like for portability and for were essential mitigations, but the hybrid design's high GPU-to-CPU ratio highlighted inherent limitations for CPU-bound simulation components, ultimately driving the transition to more advanced exascale systems capable of higher fidelity in complex, large-scale models.

Decommissioning and Legacy

Phase-Out Process

The phase-out of Sierra commenced following the initial ramp-up of in late 2023, with resource allocations gradually shifting to prioritize validation and scaling on the newer system. By February 2025, as reached full operational capability, Sierra's computational utilization declined sharply, culminating in its complete decommissioning by early 2025. Key procedures during this period emphasized data migration from Sierra's storage systems and porting of simulation codes to preserve continuity in high-fidelity modeling for nuclear security applications. LLNL application teams addressed architectural differences—transitioning from Sierra's Power9 CPUs paired with V100 GPUs to compatible frameworks—through targeted strategies, including use of abstraction layers like RAJA for performance equivalence testing on pre-production hardware. These efforts, informed by centers of excellence, mitigated disruptions by validating code scalability prior to Sierra's offline status. Empirical indicators of the phase-out included reduced job throughput on Sierra, as allocations favored systems offering over 20 times the performance, reflecting standard LLNL practices for end-of-life hardware after sustained petascale service. Hardware components were powered down systematically post-migration, with no reported interruptions to ongoing classified workloads.

Transition to Successors

The Sierra supercomputer directly influenced the design of its successor, , deployed at in 2024 as part of the U.S. Department of Energy's (DOE) CORAL-2 initiative to replace Sierra's and V100 GPU architecture. adopts a GPU-centric paradigm similar to Sierra but scales to exascale performance exceeding 2 exaFLOPS, achieving a peak of 2.79 exaFLOPS and ranking as the world's fastest supercomputer by 2024. This handoff emphasized continuity in accelerated computing, transitioning from discrete GPUs to integrated MI300A accelerated processing units (APUs) while maintaining workflows for simulations. Porting legacy codes from Sierra to El Capitan's AMD MI300A GPUs proved relatively straightforward, with many applications running effectively without modifications during early access system validations. Sierra served as a bridge for testing and optimizing these ports, enabling developers to validate performance on its GPU environment before full exascale deployment, thus minimizing disruptions in codebases developed over years for Sierra's architecture. The DOE's strategy for this transition relied on iterative upgrades through Centers of Excellence (COEs), fostering collaboration between national labs, vendors like HPE and , and application teams to evolve Sierra-era software stacks incrementally toward exascale compatibility. This approach avoided wholesale resets by leveraging Sierra's operational data and validation runs to inform El Capitan's , ensuring sustained productivity in compute-intensive domains without requiring complete application rewrites.

Long-Term Influence on Supercomputing

Sierra's integration of IBM POWER9 CPUs with NVIDIA V100 GPUs exemplified architectures, which combined general-purpose processing with specialized accelerators to optimize workloads such as scientific simulations. This design shift, validated through Sierra's deployment in 2018, influenced subsequent U.S. Department of Energy systems, including exascale platforms like and Aurora, by demonstrating scalable performance gains in memory-bound and compute-intensive tasks. Globally, Sierra's success contributed to the dominance of GPU-accelerated nodes in the list, where heterogeneous systems rose from niche to over 90% of entries by 2023, prompting commercial vendors to prioritize similar hybrid configurations for data centers and AI training clusters. The system's sustained throughput, exceeding predecessors like Sequoia by over sixfold in aggregate performance metrics, enabled researchers to iterate complex models—such as multiphysics simulations—at accelerated rates, reducing computation times from weeks to days for tasks in and . This efficiency gain facilitated policy-relevant predictions in areas like climate modeling and energy research, where higher fidelity outputs informed decision-making timelines previously constrained by serial processing limitations. For instance, Sierra's supported five times the scalable science throughput of prior systems, allowing for broader exploration of parameter spaces and that advanced algorithmic development for next-generation hardware. Sierra's operational phase fostered advancements in portable programming models, with ported applications leveraging frameworks like OpenACC and contributing to open-source libraries for heterogeneous code migration, such as those refined under DOE's program. These efforts trained a workforce of computational scientists in accelerator programming, with LLNL personnel applying Sierra-derived expertise to successor projects, including , thereby sustaining institutional and reducing onboarding barriers for exascale-era tools.

References

  1. https://www.llnl.gov/article/44786/lawrence-livermore-unveils-nnsas-sierra-worlds-third-fastest-supercomputer
  2. https://hpc.llnl.gov/hardware/compute-platforms/sierra
  3. https://developer.nvidia.com/blog/lawrence-livermore-unveils-sierra-worlds-third-fastest-supercomputer/
  4. https://www.datacenterdynamics.com/en/news/lawrence-livermore-unveils-worlds-third-fastest-supercomputer-sierra/
  5. https://str.llnl.gov/past-issues/august-2020/sierra-era
  6. https://top500.org/system/179398/
  7. https://www.llnl.gov/article/45846/new-top500-list-unveiled-sierra-stays-no-2-six-other-llnl-supercomputers-top-100
  8. https://www.newswise.com/articles/sierra-honored-with-top-supercomputing-achievement-from-hpcwire
  9. https://www.llnl.gov/article/52061/lawrence-livermore-national-laboratorys-el-capitan-verified-worlds-fastest-supercomputer
  10. https://energy.gov/sites/prod/files/2014/12/f19/CORAL%2520Fact%2520Sheet__FINAL%2520AS%2520ISSUED_UPDATED.pdf
  11. https://hpc.llnl.gov/sites/default/files/announcement.DOE_.2014.11.14_0.pdf
  12. https://asc.llnl.gov/exascale/coral-sierra
  13. https://www.nextplatform.com/2017/10/05/clever-machinations-livermores-sierra-supercomputer/
  14. https://www.hpcwire.com/2017/11/20/installation-sierra-supercomputer-steams-along-llnl/
  15. https://ieeexplore.ieee.org/document/8961159
  16. https://www.cnet.com/science/ibm-nvidia-land-325-million-supercomputer-deal/
  17. https://www.energy.gov/articles/timeline-60-years-computing-lawrence-livermore-national-lab
  18. https://top500.org/lists/top500/2018/06/
  19. https://top500.org/lists/top500/2018/11/
  20. https://hpc.llnl.gov/documentation/tutorials/using-lc-s-sierra-systems
  21. https://research.ibm.com/publications/summit-and-sierra-supercomputer-cooling-solutions
  22. https://research.ibm.com/publications/the-high-speed-networks-of-the-summit-and-sierra-supercomputers
  23. https://www.osti.gov/servlets/purl/1400194
  24. https://www.osti.gov/servlets/purl/1480422
  25. https://www.llnl.gov/article/53316/llnl-used-ai-predict-historic-fusion-ignition-shot
  26. https://insidehpc.com/2022/12/due-credit-sierra-jade-and-hpcs-role-in-livermores-fusion-ignition-breakthrough/
  27. https://par.nsf.gov/biblio/10298914-scalability-hybrid-spmv-hypergraph-partitioning-vertex-delegation-communication-avoidance
  28. https://ldrd-annual.llnl.gov/archives/ldrd-annual-2016/computing/failure
  29. https://www.llnl.gov/article/44471/llnls-sierra-third-fastest-supercomputer
  30. https://top500.org/lists/top500/2020/06/
  31. https://asc.llnl.gov/sites/asc/files/2020-06/sierra-fact-sheet.pdf
  32. https://www.youtube.com/watch?v=OoajYVQuIhA
  33. https://ieeexplore.ieee.org/ielaam/5288520/9093089/8930295-aam.pdf
  34. https://www.llnl.gov/article/44886/sierra-reaches-higher-altitudes-takes-no-2-spot-list-worlds-fastest-supercomputers
  35. https://www.energy.gov/articles/two-doe-supercomputers-top-list-worlds-fastest
  36. https://top500.org/lists/top500/2019/11/
  37. https://oakridgetoday.com/2020/06/22/japanese-supercomputer-displaces-ornls-summit-as-worlds-most-powerful/
  38. https://insidehpc.com/2020/06/arm-based-fugaku-stands-on-summit-of-new-top500/
  39. https://www.datacenterdynamics.com/en/news/fugaku-remains-worlds-fastest-supercomputer-in-latest-top500/
  40. https://top500.org/news/frontier-remains-sole-exaflop-machine-and-retains-top-spot-improving-upon-its-previous-hpl-score/
  41. https://www.networkworld.com/article/1246936/aurora-enters-top500-supercomputer-ranking-at-no-2-with-a-challenge-for-reigning-champ-frontier.html
  42. https://www.energy.gov/nnsa/articles/nnsa-supercomputers-recognized-worldwide-speed-and-performance
  43. https://link.aps.org/doi/10.1103/PhysRevFluids.8.083401
  44. https://www.pnas.org/doi/10.1073/pnas.2508608122
  45. https://www.researchgate.net/publication/342227486_Reduced-order_modelling_of_equations_of_state_using_tensor_decomposition_for_robust_accurate_and_efficient_property_calculation_in_high-pressure_fluid_flow_simulations
  46. https://nap.nationalacademies.org/read/26728/chapter/6
  47. https://www.llnl.gov/article/49911/high-performance-computing-ai-cognitive-simulation-helped-llnl-conquer-fusion-ignition
  48. https://www.hpcwire.com/2022/12/21/supercomputings-critical-role-in-the-fusion-ignition-breakthrough/
  49. https://www.osti.gov/servlets/purl/966897
  50. https://www.lanl.gov/media/publications/national-security-science/0424-beyond-the-blast
  51. https://www.osti.gov/servlets/purl/1497475
  52. https://www.energy.gov/nnsa/us-nuclear-weapons-stockpile
  53. https://www.energy.gov/nnsa/nnsas-high-performance-computing-achievements
  54. https://www.llnl.gov/article/49056/scientific-discovery-stockpile-stewardship
  55. https://www.llnl.gov/article/40566/next-generation-supercomputer-coming-lab
  56. https://www.sandia.gov/app/uploads/sites/79/2023/03/ASC_Report_Feb_2023_Lowres_3_3_23-1.pdf
  57. https://www.energy.gov/sites/default/files/2019/08/f65/FY2020__SSMP.pdf
  58. https://www.osti.gov/servlets/purl/1631911
  59. https://www.world-nuclear-news.org/Articles/US-funding-for-supercomputers
  60. https://insidehpc.com/2018/10/llnl-unveils-nnsas-sierra-worlds-third-fastest-supercomputer/
  61. https://physicstoday.aip.org/news/nnsa-touts-savings-from-supercomputing
  62. https://nap.nationalacademies.org/download/11148
  63. https://www.armscontrol.org/act/1997-08/arms-control-today/report-criticizes-stockpile-stewardship-program
  64. https://issues.org/mello/
  65. https://www.heritage.org/defense/report/why-the-administrations-stockpile-stewardship-will-harm-the-us-nuclear-deterrent
  66. https://www.prif.org/fileadmin/Daten/Publikationen/Prif_Reports/2004/prif68.pdf
  67. https://www.osti.gov/servlets/purl/1494162
  68. https://www.osti.gov/servlets/purl/2204463
  69. https://computing.llnl.gov/about/newsroom/road-el-capitan-8
  70. https://www.space.com/space-exploration/tech/worlds-fastest-supercomputer-el-capitan-goes-online-it-will-be-used-to-secure-the-us-nuclear-stockpile-and-in-other-classified-research
  71. https://www.exascaleproject.org/exascale-day-article-early-access-systems-at-llnl-mark-progress-toward-el-capitan/
  72. https://bssw.io/items/webinar-migrating-to-heterogeneous-computing-lessons-learned-in-the-sierra-and-el-capitan-centers-of-excellence
  73. https://str.llnl.gov/str-december-2024/introducing-el-capitan
  74. https://www.energy.gov/articles/does-nnsa-signs-600-million-contract-build-its-first-exascale-supercomputer
  75. https://asc.llnl.gov/exascale/el-capitan
  76. https://www.nextgov.com/emerging-tech/2024/11/lawrence-livermores-el-capitan-supercomputer-officially-fastest-world/401113/
  77. https://www.tomshardware.com/news/new-amd-instinct-mi300-details-emerge-debuts-in-2-exaflop-el-capitan-supercomputer
  78. https://www.hpcwire.com/off-the-wire/llnl-scientists-anticipate-el-capitans-potential-impact/
  79. https://www.exascaleproject.org/event/sierra_and_elcapitan_coe/
  80. https://top500.org/news/with-sierra-nnsa-embraces-heterogeneous-supercomputing/
  81. https://research.ibm.com/publications/sierra-center-of-excellence-lessons-learned
  82. https://str.llnl.gov/past-issues/march-2017/center-excellence-prepares-sierra
Add your contribution
Related Hubs
User Avatar
No comments yet.