Hubbry Logo
Advanced Matrix ExtensionsAdvanced Matrix ExtensionsMain
Open search
Advanced Matrix Extensions
Community hub
Advanced Matrix Extensions
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Contribute something
Advanced Matrix Extensions
Advanced Matrix Extensions
Advanced Matrix Extensions
View on Wikipedia
from Wikipedia

Advanced Matrix Extensions (AMX), also known as Intel Advanced Matrix Extensions (Intel AMX), are extensions to the x86 instruction set architecture (ISA) for microprocessors from Intel designed to work on matrices to accelerate artificial intelligence (AI) and machine learning (ML) workloads.[1] Especially they perform matrix multiplication at the hardware level, making them apt for problems and algorithms that use matrix multiplication as their core.

Extensions

[edit]

AMX was introduced by Intel in June 2020 and first supported by Intel with the Sapphire Rapids microarchitecture for Xeon servers, released in January 2023.[2][3] It introduced 2-dimensional registers called tiles upon which accelerators can perform operations. It is intended as an extensible architecture; the first accelerator implemented is called tile matrix multiply unit (TMUL).[4][5]

In Intel Architecture Instruction Set Extensions and Future Features revision 46, published in September 2022, a new AMX-FP16 extension was documented. This extension adds support for half-precision floating-point numbers. In revision 48 from March 2023, AMX-COMPLEX was documented, adding support for half-precision floating-point complex numbers. Both extensions are available in the Granite Rapids set of server processors (with AMX-COMPLEX support only being available in Granite Rapids-D [6]).

Tile matrix multiply unit

[edit]

TMUL unit supports BF16 and INT8 input types.[7] AMX-FP16 and AMX-COMPLEX also add support for real and complex FP16 numbers. The register file consists of 8 tiles, each with 16 rows of size of 64 bytes (32 BF16/FP16 or 64 INT8 elements). The only supported operation is matrix multiply and accumulate (MMA), which is the extension of the fused multiply–add (FMA) operation for scalar values as applied to matrix operands:[8]

[4]

4th Gen Intel Xeon Scalable processor core can perform 2048 INT8 or 1024 BF16 operations per cycle:[9][10] the maximal input sizes are for A and for B, where J is 64 for INT8 and 32 for BF16. The matrix multiplication requires multiplication and additions, thus performing operations in 16 cycles.[10]

Software support

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Advanced Matrix Extensions

View on Grokipedia
from Grokipedia
Advanced Matrix Extensions (AMX), formally known as Intel® Advanced Matrix Extensions, are a set of x86 extensions designed to accelerate operations for and workloads on processors. Introduced by in June 2020, AMX provides a dedicated hardware accelerator integrated into each CPU core, enabling efficient handling of dense matrix computations without requiring discrete accelerators. At its core, AMX introduces a tile-based architecture consisting of eight 1KB two-dimensional registers (known as XTILEs) that store matrix data in a compact, row-major format, allowing for high-throughput operations on matrices up to 16x64 elements depending on the . The primary computational unit is the Tile Matrix Multiply (TMUL) engine, which performs matrix multiplications using supported s including bfloat16 (BF16) for both and , int8 for tasks, and FP16 in later extensions such as those in ® ® 6 processors. Additional instructions handle loading, configuration, zeroing, and accumulation, facilitating seamless integration with existing vector extensions like while reducing data movement overhead. AMX first became available with the 4th Generation Intel® Xeon® Scalable processors (code-named ) in January 2023, marking Intel's third generation of Boost technologies following initial VNNI and subsequent enhancements. It has since been enhanced in the 5th Generation Intel® Xeon® Scalable processors () and Intel® Xeon® 6 processors with Performance-cores, delivering up to 14x improvements in training and inference performance compared to prior generations. Software support includes intrinsics in compilers like GCC (from version 11) and (from version 12), as well as optimizations in major frameworks such as and , enabling developers to leverage AMX for applications like , , and generative AI. By embedding matrix acceleration directly into the CPU, AMX lowers for AI deployments through reduced power consumption and simplified system design, while supporting scalable on standard server hardware without specialized GPUs. This positions AMX as a key enabler for edge-to-cloud AI processing, particularly in enterprise environments requiring high efficiency and flexibility.

History

Announcement and Development

Advanced Matrix Extensions (AMX) were first introduced by in June 2020, with further details presented at the company's Architecture Day event on August 13, 2020, where it was presented as a key component of built-in acceleration for future processors. Developed as an extension to the x86 , AMX aims to accelerate and accumulation operations central to and workloads on general-purpose CPUs. It specifically addresses the inefficiencies of prior vector-based instructions like in handling the dense, two-dimensional data patterns common in deep learning models, such as those involving training. The primary motivations for AMX's creation were to enable competitive CPU-based performance for AI training and inference, diminishing reliance on discrete accelerators and supporting cost-effective deployments in data centers and edge computing scenarios. From its inception, AMX was designed for integration into Intel's oneAPI software ecosystem to facilitate portable, high-level programming across heterogeneous hardware. Early development included prototypes in Intel's labs, with the company expressing readiness to sample hardware to partners shortly after the announcement, emphasizing a tile-based approach for improved data handling efficiency.

Initial Release

The initial hardware implementation of Advanced Matrix Extensions (AMX) debuted with the 4th Generation Scalable processors, codenamed , which were released on January 10, 2023. These server-grade processors marked the first integration of AMX into production silicon, providing dedicated acceleration for operations critical to AI and workloads. AMX is implemented as a dedicated hardware block embedded within each CPU core of the architecture, operating alongside existing execution units to handle tiled matrix computations efficiently. This integration relies on updated CPU for instruction decoding and execution, with activation controlled through operating system flags and configurations to ensure compatibility and security. The rollout of AMX in aligned with Intel's strategic emphasis on accelerating AI and in environments, enabling scalable performance gains for enterprise server deployments without requiring discrete accelerators. AMX typically involves selecting the "Intel Advanced Matrix Extensions" option in the system under processor features, followed by verification in supported operating systems like , where kernels version 5.16 and later detect and enable the feature at runtime without mandatory boot parameters.

Subsequent Extensions

Following the initial release of Advanced Matrix Extensions (AMX) with the 4th Generation Scalable processors () in early 2023, introduced enhancements to broaden its applicability in AI and workloads. In September 2022, documented the AMX-FP16 extension in revision 46 of the Instruction Set Extensions and Future Features Programming Reference, adding support for half-precision floating-point (FP16) operations to accelerate AI tasks while balancing precision and efficiency. This extension enables processing of FP16-trained models, complementing the original INT8 and BF16 formats by reducing demands in deployments. In December 2022, Intel further expanded AMX capabilities with the AMX-COMPLEX extension, documented in revision 047 of the same reference manual, which introduces operations on half-precision complex numbers. Designed for , scientific simulations, and applications, this extension supports complex matrix multiplications essential for Fourier transforms and other domain-specific computations. AMX-COMPLEX first became available in the Granite Rapids-D processors, a variant of the 6th Generation Scalable family launched in 2025. AMX support expanded across subsequent processor generations, enhancing scalability in data center environments. The 5th Generation Scalable processors (), announced in December 2023, integrated baseline AMX along with FP16 capabilities on every core, enabling seamless upgrades from prior generations while optimizing for AI acceleration. Building on this, the 6th Generation Scalable processors (Granite Rapids), released in September 2024, incorporated full AMX support including FP16 and COMPLEX extensions, delivering up to 2,048 FP16 operations per cycle per core for improved throughput in mixed-precision workloads. By 2025, AMX integration extended to 6 processors with P-cores, as seen in new SKUs like the 6776P announced in May 2025, which added enhanced FP16 support to facilitate on-chip AI processing in edge and cloud scenarios. In October 2025, and announced an agreement to support AMX in future x86 processors, promoting and wider adoption across the . As of November 2025, 's oneAPI Base Toolkit version 2025 includes optimizations for AMX interoperability with AVX10.2, such as early access to extended instruction sets in the oneDNN library, to streamline development for hybrid CPU-GPU AI pipelines. However, no major new AMX extensions have been announced, with focus shifting toward software maturity and compatibility with emerging standards like APX.

Architecture

Tile Registers

The tile registers form the core storage mechanism in Intel's Advanced Matrix Extensions (AMX), providing a dedicated for holding matrix data during computations. This consists of eight two-dimensional registers named tmm0 through tmm7, each capable of storing up to 1 KB ( bytes) of data, for a total capacity of 8 KB across all tiles. These registers are used to hold sub-matrices representing source operands A and B, as well as the accumulator matrix C, enabling efficient data staging for operations like . Each tile register supports a configurable two-dimensional layout, allowing flexible matrix shapes tailored to specific workloads. The dimensions are defined in terms of rows and columns measured in bytes, with a maximum of 16 rows and 64 bytes per row (e.g., supporting shapes like 16 rows by 64 bytes for matrix A). This configuration enables tiles to represent portions of larger matrices, with the actual element count depending on the data format specified in the configuration. The registers are volatile, meaning their contents are not preserved across context switches or interrupts unless explicitly saved. The configuration is managed through a 64-bit Tile Configuration Palette register (TILECFG), which specifies the row and column dimensions, as well as the , for each individual . This palette is loaded using the LDTILECFG instruction or restored via XRSTOR, and its settings can be queried through leaf 1DH (sub-leaf 1 with ECX=1). The palette supports different "palettes" (e.g., Palette 1 for standard AMX configurations), each defining maximum dimensions and total tile bytes to optimize for various matrix sizes. Data is loaded into the tile registers primarily from system memory using the TILELOADD instruction, which transfers elements into the specified tile based on the configured dimensions and starting address. For example, TILELOADD can load a rectangular block of data directly into tmm0 from a memory operand, supporting both contiguous and strided access patterns to accommodate row-major or column-major matrix layouts. This mechanism allows efficient population of tiles before computations, minimizing data movement overhead.

Matrix Multiply Unit

The Tile Matrix Multiply Unit (TMUL) serves as the dedicated hardware accelerator within Intel's Advanced Matrix Extensions (AMX) for executing dense matrix multiply-accumulate (MMA) operations, enabling efficient computation of large-scale matrix products essential for AI workloads. This unit processes data stored in the AMX registers, focusing on fused multiply-add operations to minimize latency and maximize throughput in deep learning training and inference tasks. In terms of operation flow, the TMUL takes two input tiles—A with dimensions M×KM \times K and B with dimensions K×NK \times N—and performs the accumulation C[M×N]+=A[M×K]×B[K×N]C[M \times N] += A[M \times K] \times B[K \times N], where the inner dimension KK aligns with the configured tile widths for seamless without intermediate transpositions. This design supports block-wise computation of larger matrices by iterating over tiled submatrices, leveraging the 2D structure of the s to handle up to 1 KB per in a single instruction. The TMUL is implemented as a pipelined execution block integrated directly into each CPU core of supported processors, such as the 4th Generation Scalable family ( ), allowing concurrent operation alongside scalar and vector execution pipelines. It functions independently of the vector units, dedicating separate hardware resources—including a grid of fused multiply-add units—to matrix operations, which isolates AI compute from general-purpose processing and reduces contention for shared resources like caches. Regarding throughput, in 4th Generation implementations, the TMUL delivers up to 2048 INT8 operations per cycle for inference-focused workloads or 1024 BF16 operations per cycle for , representing an 8x improvement for INT8 and 16x for BF16 over VNNI and FP32 equivalents, respectively. This performance stems from the unit's parallel processing of tile elements, with each cycle advancing the multiply-accumulate across the full dimensions without scalar overhead.

Data Formats

Advanced Matrix Extensions (AMX) primarily support bfloat16 (BF16) and 8-bit integer (INT8) as the core data formats for matrix elements, optimized for workloads. BF16 is a 16-bit floating-point format featuring 1 , 8 exponent bits, and 7 mantissa bits, preserving the exponent range of single-precision floating-point (FP32) to handle the dynamic ranges common in training and inference while reducing and computational overhead. INT8, available in signed and unsigned variants, facilitates quantized inference by representing weights and activations with lower precision, achieving significant speedups in post-training quantization scenarios without substantial accuracy degradation in many models. Subsequent extensions expand format support: AMX-FP16 introduces half-precision floating-point (FP16), a 16-bit format with 1 , 5 exponent bits, and 10 mantissa bits, suitable for scenarios requiring higher precision in the mantissa than BF16. These extensions, including AMX-FP16 and AMX-COMPLEX, were introduced in the 6th Generation Scalable processors (Granite Rapids) in 2024. AMX-COMPLEX enables operations on complex numbers, where each complex element packs two FP16 values—one for part and one for the imaginary part—into a 32-bit dword, targeting applications in and scientific computing that involve Fourier transforms or other complex-domain computations. Matrix elements are packed into tiles in row-major order, with each row spanning up to 64 bytes (32 BF16/FP16 elements or 64 INT8 elements), ensuring contiguous storage for efficient loading from memory. Accumulations in matrix multiply operations incorporate saturation modes for INT8 to clamp results within representable ranges, preventing overflow, and rounding to nearest, tie-even for BF16 outputs, with denormals flushed to zero independent of the floating-point environment. These formats are configured per tile to match the element type, influencing the interpretable dimensions and palette used for loading data. The selection of BF16 and INT8 aligns closely with requirements in major AI frameworks like and , where BF16 supports mixed-precision training to accelerate convergence and inference while mitigating the precision loss seen in traditional FP16, and INT8 enables deployment of compressed models on resource-constrained systems.

Instruction Set

Configuration Instructions

The configuration of Advanced Matrix Extensions (AMX) tiles begins with the LDTILECFG instruction, which loads a 64-byte tile configuration descriptor from memory into the TILECFG state register. This descriptor specifies the palette identifier (0 for initialization or 1 for standard configuration), the starting row index (initially 0), and per-tile parameters including the number of rows and bytes per row for each of the eight source or accumulator tiles. The palette determines the data format and maximum tile dimensions, such as up to 16 rows and 64 bytes per row for BF16 or INT8 formats, enabling flexible matrix shapes while adhering to the hardware's 1 KB per tile limit (8 tiles total, 8 KB aggregate). Upon execution, LDTILECFG validates the descriptor; invalid palettes or dimensions trigger a (#GP), and all tiles are zeroed if the palette is 0, preparing the architecture for subsequent operations. The complementary STTILECFG instruction stores the current TILECFG state to a 64-byte memory location, writing zeros if no configuration is active, which is essential for querying or persisting setup details. Data movement into and out of tiles is handled by dedicated load and store instructions that support both 1D (contiguous) and 2D (strided) memory layouts via Scale-Index-Base (SIB) addressing. The TILELOADD instruction loads tile data from memory into a specified tile register (TMM), using the base address plus an optional index register as the row stride; if no index is provided, a stride of zero implies 1D loading. It respects the current TILECFG dimensions, filling rows sequentially from the start_row index (updated on partial loads due to page faults for restartability), and zeros any unfilled upper rows beyond the configured size. A variant, TILELOADDT1, adds a temporal data hint to optimize caching for repeated accesses. For example, loading a 16x64 BF16 matrix uses a stride matching the row byte count (128 bytes), ensuring efficient transfer of up to 1 KB per tile. The TILESTORED instruction performs the reverse, storing a tile's contents to memory starting from the configured start_row, again using SIB for strided 2D access and resetting start_row to zero upon completion. Both load and store operations are synchronous with the instruction stream, maintain cache coherency, and raise AMX-E3 exceptions for tile architecture issues, with Intel TSX transactions aborting on execution. To prepare tiles for computation, particularly accumulators, the TILEZERO instruction clears all bytes in a specified tile register to zero, using the current palette to determine the extent (rows and bytes per row). This operation is atomic, affects no flags, and is crucial for initializing accumulators before matrix multiplications, ensuring no residual data interferes with results; it raises AMX-E5 exceptions if the tile state is invalid. For instance, zeroing an 8x64 INT8 accumulator resets its 512 bytes efficiently in a single instruction. State management for context switching in multitasking environments involves saving and restoring both the configuration and tile data using dedicated mechanisms beyond standard XSAVE/XRSTOR extensions. The TILECFG state is persisted via STTILECFG to memory, while tile data requires explicit stores with TILESTORED to capture the full 8 KB register file. Restoration reverses this: LDTILECFG reloads the configuration, followed by TILELOADD to repopulate tiles from saved memory. This approach, supported by OS-enabled XTILECFG and XTILEDATA components in XSAVE (via arch_prctl or equivalent), ensures complete AMX state migration without hardware DMA, though it incurs latency due to the large data volume; Intel recommends minimizing switches in performance-critical paths.

Compute Instructions

The compute instructions in Advanced Matrix Extensions (AMX) form the core of its arithmetic capabilities, enabling efficient matrix multiply-accumulate operations on tile registers. These instructions, part of the TDP (Tile Dot Product) family, perform the fundamental computation C←C+A×BC \leftarrow C + A \times B, where AA, BB, and CC represent matrices stored in 2D tile registers (TMM0 through TMM7), with dimensions configured via the tile palette. The accumulation occurs element-wise after multiplying corresponding rows of AA (size m×km \times k) and columns of BB (size k×nk \times n), producing an updated CC of size m×nm \times n. All operations use round-to-nearest-even (RNE) rounding and flush denormals to zero for both inputs and outputs, without raising exceptions or updating the MXCSR register. The TDP family includes variants tailored to specific data precisions, supporting low-precision formats common in AI and workloads. For bfloat16 (BF16) inputs, the TDPBF16PS instruction computes the of BF16 elements from two source s and accumulates the results into a single-precision floating-point (FP32) tile. For example, each pair of BF16 values from the source tiles is multiplied and summed across the inner dimension kk, with the output scaled implicitly by the precision conversion to FP32. Similarly, the TDPFP16PS instruction (part of the AMX-FP16 extension) handles FP16 inputs, performing the same multiply-accumulate semantics but treating FP16 pairs as the operands, again accumulating into FP32. These floating-point variants enable high-throughput operations for layers, where BF16 and FP16 reduce without significant accuracy loss. Integer variants target quantized models, with the AMX-INT8 instructions providing four signed-saturate options: TDPBSSD for signed-by-signed byte multiplications accumulating to 32-bit integers; TDPBSUD for signed-by-unsigned; TDPBUSD for unsigned-by-signed; and TDPBUUD for unsigned-by-unsigned. Each processes byte elements (four per 32-bit dword) from the source tiles, computing dot products with saturation to prevent overflow during accumulation into an INT32 tile. These instructions support signed saturation, ensuring results stay within the representable range of the accumulator. For complex arithmetic (AMX-COMPLEX), the instructions TCMMRLFP16PS and TCMMIMFP16PS handle real and imaginary parts separately using FP16 components, performing conjugate multiplications and accumulations into separate FP32 tiles for real and imaginary outputs, effectively realizing C←C+A×BC \leftarrow C + A \times B for complex matrices. Subsequent extensions as of November 2025 include AMX-FP8 for 8-bit floating-point operations (e.g., TDPBF8PS, TDPHF8PS instructions supporting E4M3 and E5M2 formats) and AMX-TF32 for tensor float 32 (e.g., TDPTF32PS), enabling further optimizations for AI workloads on newer processors like 6 ( Rapids). AMX-TRANSPOSE was proposed for efficient matrix transposition but later discontinued. All TDP-family instructions are encoded using the EVEX prefix, extending the AVX-512 framework, with tile registers specified via and VEX.vvvv fields (e.g., TDPBF16PS uses EVEX.512.66.0F38.W0 5F /r, where /r indicates tmm1 as destination, tmm2 as first source, and tmm3 as second source). The EVEX encoding ensures 512-bit vector operation modes while embedding the 2D semantics, distinguishing AMX from scalar or 1D vector instructions. Prior tile configuration via the configuration instructions is required to define the palette and dimensions before executing these compute operations. Support for each variant is enumerated via leaf 07H. Basic features (AMX-TILE in EDX bit 24, AMX-BF16 in EDX bit 22, AMX-INT8 in EDX bit 25) at sub-leaf ECX=0; AMX-FP16 in EDX bit 21 and AMX-COMPLEX in EDX bit 8 at sub-leaf ECX=1.

Software Support

Operating Systems

Linux kernel version 5.16, released in late 2021, introduced support for Advanced Matrix Extensions (AMX), enabling the operating system to manage AMX state components through the XSAVE feature and allowing per-process enablement via the arch_prctl(2) system call with ARCH_REQ_XCOMP_PERM to request permission for dynamic XSTATE components like TILE_DATA. This support ensures proper context switching and signal handling for AMX usage, with the kernel detecting AMX-capable processors at runtime and setting the AMX bit in XCR0 during boot. VMware ESXi 8.0 Update 1, released in 2023, provides full passthrough support for AMX instructions to virtual machines configured with virtual hardware version 20 or higher, allowing guest operating systems like 5.19 or later to utilize the extension without host intervention. Windows 11, along with , enables AMX support on compatible processors through kernel updates that handle AMX context saving and restoration, requiring driver-level feature detection to ensure compatibility; Windows 10 lacks this context save capability, preventing AMX usage. Oracle Linux's Unbreakable Enterprise Kernel 7 Update 1, released in 2023, includes runtime support for AMX on capable processors, automatically detecting the feature and enabling userspace access without additional configuration. provides basic enablement for AMX through updates via the cpu-microcode-intel port, which applies processor firmware updates at to ensure compatibility on supported hardware, though full kernel integration for AMX remains limited. Across these operating systems, AMX features are detected using the CPUID instruction with leaf 7 and subleaf 0, where EDX bit 24 indicates support for the core AMX-TILE functionality, including tile multiply accumulate (TMUL) operations, and bit 22 signals the AMX-BF16 variant.

Compilers and Libraries

Support for Advanced Matrix Extensions (AMX) in compilers primarily comes through intrinsic functions that map directly to the AMX instruction set architecture, enabling developers to write low-level code for tile-based operations. The GNU Compiler Collection (GCC) version 11 and later provides AMX support via the -mamx-tile flag, which exposes intrinsics such as _tile_loadconfig for loading tile configuration descriptors (TCFG). Similarly, LLVM-based Clang compilers, starting from version 13, include AMX intrinsics through headers like amxintrin.h, activated by the -mamx-tile option, allowing for tile loading, computation, and storage operations. These intrinsics facilitate direct access to AMX features without requiring assembly code, though they demand explicit management of tile registers and shapes. Optimized libraries abstract AMX usage for higher-level programming, reducing the need for manual intrinsic calls. Intel's oneDNN (oneAPI Deep Neural Network Library), formerly known as MKL-DNN, integrates automatic AMX offload for primitives like and , detecting hardware availability at runtime via its CPU dispatcher to select AMX-accelerated kernels. This enables seamless acceleration of workloads on supported processors without code modifications. For , the oneAPI DPC++ compiler supports AMX through its integration with oneMKL (), allowing SYCL-based code to target AMX for matrix operations alongside GPU or FPGA offloads. Major frameworks leverage AMX via dedicated extensions. The Extension for , integrated since TensorFlow 2.11 in early 2023, incorporates AMX-optimized kernels for operations like GEMM and , automatically dispatching to AMX on compatible hardware. Likewise, the Extension for , available from version 1.13 in 2023, provides AMX support for training and inference kernels, including fused operations that utilize tile registers for improved throughput. These extensions maintain framework compatibility while enabling AMX acceleration for models like transformers. As of 2025, the oneAPI Base Toolkit release enhances AMX integration with support for AVX10.2 instructions on upcoming processors such as Diamond Rapids, extending AMX capabilities to newer vector-matrix hybrid operations within the oneDNN and DPC++ environments. This update ensures with existing AMX code while optimizing for advanced ISA features in future architectures.

Performance and Applications

Performance Metrics

Advanced Matrix Extensions (AMX) provide significant throughput improvements for matrix operations on processors. On 4th-generation Scalable processors, each core can perform up to 2,048 INT8 operations per cycle and 1,024 BF16 operations per cycle, enabling high-efficiency AI workloads. These specifications represent an 8x increase in INT8 throughput and a 16x increase in BF16 throughput compared to VNNI, which delivers 256 INT8 and 64 FP32 operations per cycle, primarily due to AMX's 2D register that reduces data movement overhead. Benchmark results demonstrate substantial speedups in AI tasks. For instance, 5th-generation processors achieve up to 14x better and performance compared to 3rd-generation processors, leveraging AMX for accelerated matrix multiplications. Specific workloads show even higher gains: real-time (RNN-T) yields up to 10.7x speedup using BF16 on 5th-generation Platinum 8592+ versus FP32 on 3rd-generation models. Additionally, GPT-J-6B performance doubles on 6 processors compared to 5th-generation , with BF16 data format. Efficiency considerations highlight trade-offs in precision and power. BF16 operations offer a balance of precision and performance suitable for training and high-accuracy inference, while INT8 enables higher throughput for quantized models at reduced precision, often with lower power consumption. For example, AMX delivers up to 7.9x higher performance per watt in on 5th-generation versus 3rd-generation, emphasizing energy-efficient scaling for deployments.
Data TypeAMX Throughput (ops/cycle per core) VNNI Throughput (ops/cycle per core)Speedup Factor
INT82,0482568x
BF161,02464 (FP32 equivalent)16x
This table illustrates the core throughput advantages of AMX over , focusing on matrix multiply operations at peak utilization.

Use Cases

Advanced Matrix Extensions (AMX) have found significant application in and workloads, particularly for accelerating training and inference directly on CPUs without relying on dedicated GPUs. This capability is especially beneficial for transformer-based models such as BERT and GPT variants, where matrix multiplications form the core of operations like attention mechanisms. For instance, optimized implementations using AMX on processors enable efficient tasks, including text classification and generation, by leveraging low-precision formats like BF16 and INT8 to reduce computational overhead while maintaining model accuracy. In scientific computing, AMX supports matrix operations critical to simulations and numerical methods, such as those involving mixed-precision floating-point computations for high-fidelity modeling in fields like and physics. These extensions facilitate faster execution of linear algebra routines, which underpin algorithms for and predictive modeling, allowing researchers to process large datasets on standard server hardware. AMX enables efficient AI inference in both edge and cloud environments, particularly on Intel Xeon-based servers, where power and space constraints demand integrated acceleration. In cloud deployments, it supports scalable AI services for hyperscale data centers, reducing costs for lighter workloads like recommendation systems and image recognition by avoiding GPU provisioning. As of 2025, adoption has grown in enterprise and cloud infrastructures, with platforms like Nutanix and VMware integrating AMX for on-premises, hybrid, and edge AI applications, including Google's use of AMX with Intel TDX for confidential AI workloads as of July 2025, promoting cost-effective scaling for distributed computing needs.

References

  1. https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/what-is-intel-amx.html
  2. https://en.wikichip.org/wiki/x86/amx
  3. https://www.intc.com/news-events/press-releases/detail/1598/intel-launches-4th-gen-xeon-scalable-processors-max-series
  4. https://www.phoronix.com/review/intel-xeon-amx
  5. https://download.intel.com/newsroom/2020/client-computing/intel-2020-architecture-day-fact-sheet.pdf
  6. https://fuse.wikichip.org/news/3600/the-x86-advanced-matrix-extension-amx-brings-matrix-operations-to-debut-with-sapphire-rapids/
  7. https://www.crn.com/slide-shows/components-peripherals/6-big-announcements-from-intel-architecture-day-2020
  8. https://www.intel.com/content/www/us/en/developer/articles/technical/accelerate-pytorch-training-inference-on-amx.html
  9. https://www.intel.com/content/www/us/en/developer/articles/technical/fourth-generation-xeon-scalable-family-overview.html
  10. https://www.intel.com/content/www/us/en/support/articles/000101065/processors/intel-xeon-processors.html
  11. https://blogs.oracle.com/linux/intel-amx-with-uek
  12. https://cdrdv2-public.intel.com/790021/architecture-instruction-set-extensions-programming-reference.pdf
  13. https://www.phoronix.com/news/Intel-AMX-COMPLEX-GCC
  14. https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-02/xeon-6-granite-rapids-product-brief.pdf
  15. https://www.tomshardware.com/pc-components/cpus/intel-launches-three-new-xeon-6-p-core-cpus-will-debut-in-nvidia-dgx-b300-ai-systems
  16. https://www.hwcooling.net/en/intel-and-amd-agree-on-future-of-x86-cpus-amx-and-ram-tagging/
  17. https://www.intel.com/content/www/us/en/developer/articles/release-notes/oneapi-base-toolkit/2025.html
  18. https://www.intel.com/content/www/us/en/developer/articles/technical/next-gen-performance-gcc-15.html
  19. https://cdrdv2-public.intel.com/843860/architecture-instruction-set-extensions-programming-reference-dec-24.pdf
  20. https://cdrdv2-public.intel.com/814201/355308-Optimization-Reference-Manual-049-Changes-Doc.pdf
  21. https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-12/accelerate-ai-with-amx-sb.pdf
  22. https://arxiv.org/abs/1905.12322
  23. https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html
  24. https://docs.nrel.gov/docs/fy25osti/93622.pdf
  25. https://www.intel.com/content/www/us/en/docs/onednn/developer-guide-reference/2024-2/data-types-002.html
  26. https://www.intel.com/content/www/us/en/developer/articles/code-sample/advanced-matrix-extensions-intrinsics-functions.html
  27. https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus
  28. https://www.felixcloutier.com/x86/ldtilecfg
  29. https://www.felixcloutier.com/x86/sttilecfg
  30. https://www.felixcloutier.com/x86/tileloadd:tileloaddt1
  31. https://www.felixcloutier.com/x86/tilestored
  32. https://www.felixcloutier.com/x86/tilezero
  33. https://www.felixcloutier.com/x86/tdpbf16ps
  34. https://www.felixcloutier.com/x86/tdpbssd:tdpbsud:tdpbusd:tdpbuud
  35. https://cdrdv2-public.intel.com/869288/319433-060-architecture-instruction-set-extensions-programming-reference.pdf
  36. https://www.phoronix.com/news/Intel-Remove-AMX-TRANSPOSE-GCC
  37. https://www.felixcloutier.com/x86/cpuid
  38. https://www.phoronix.com/news/Intel-AMX-Lands-In-Linux
  39. https://docs.kernel.org/arch/x86/xstate.html
  40. https://lwn.net/Articles/863391/
  41. https://lenovopress.lenovo.com/lp2067-using-intel-advanced-matrix-extensions-with-vmware-esxi
  42. https://github.com/openvinotoolkit/openvino/issues/21078
  43. https://www.freshports.org/sysutils/cpu-microcode-intel/
  44. https://github.com/intel/Intel-AMX-Detection
  45. https://community.intel.com/t5/Intel-ISA-Extensions/AMX-support-in-SDE-incomplete/td-p/1193846
  46. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html
  47. https://gcc.gnu.org/gcc-11/changes.html
  48. https://www.intel.com/content/www/us/en/developer/articles/technical/whats-new-in-llvm-for-4th-gen-intel-xeon-processor.html
  49. https://clang.llvm.org/doxygen/amxintrin_8h_source.html
  50. https://blog.tensorflow.org/2023/01/optimizing-tensorflow-for-4th-gen-intel-xeon-processors.html
  51. https://cdrdv2-public.intel.com/785250/Intel-AMXBrief-Final-3.17.pdf
  52. https://www.intel.com/content/www/us/en/developer/articles/guide/bert-ai-inference-amx-4th-gen-xeon-scalable.html
  53. https://github.com/intel/intel-extension-for-transformers
  54. https://www.wwt.com/article/unlocking-ai-performance-with-intels-amx-accelerator-in-5th-gen-xeon-processors
  55. https://www.nutanix.com/blog/accelerating-ai-adoption-with-intel-amx-with-nutanix-enterprise-ai-2-4
  56. https://cdrdv2-public.intel.com/861663/confidential-computing-ai-whitepaper.pdf
Add your contribution
Related Hubs
Contribute something
User Avatar
No comments yet.