Hubbry Logo
InfiniBandInfiniBandMain
Open search
InfiniBand
Community hub
InfiniBand
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
InfiniBand
InfiniBand
InfiniBand
View on Wikipedia
from Wikipedia

Key Information

InfiniBand (IB) is a computer networking standard used in high-performance computing that features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also used as either a direct or switched interconnect between servers and storage systems, as well as an interconnect between storage systems. It is designed to be scalable and uses a switched fabric network topology. Between 2014 and June 2016,[1] it was the most commonly used interconnect in the TOP500 list of supercomputers.

Mellanox (acquired by Nvidia) manufactures InfiniBand host bus adapters and network switches, which are used by large computer system and database vendors in their product lines.[2]

As a computer cluster interconnect, IB competes with Ethernet, Fibre Channel, and Intel Omni-Path. The technology is promoted by the InfiniBand Trade Association.

History

[edit]

InfiniBand originated in 1999 from the merger of two competing designs: Future I/O and Next Generation I/O (NGIO). NGIO was led by Intel, with a specification released in 1998,[3] and joined by Sun Microsystems and Dell. Future I/O was backed by Compaq, IBM, and Hewlett-Packard.[4] This led to the formation of the InfiniBand Trade Association (IBTA), which included both sets of hardware vendors as well as software vendors such as Microsoft. At the time it was thought some of the more powerful computers were approaching the interconnect bottleneck of the PCI bus, in spite of upgrades like PCI-X.[5] Version 1.0 of the InfiniBand Architecture Specification was released in 2000. Initially the IBTA vision for IB was simultaneously a replacement for PCI in I/O, Ethernet in the machine room, cluster interconnect and Fibre Channel. IBTA also envisaged decomposing server hardware on an IB fabric.

Mellanox had been founded in 1999 to develop NGIO technology, but by 2001 shipped an InfiniBand product line called InfiniBridge at 10 Gbit/second speeds.[6] Following the burst of the dot-com bubble there was hesitation in the industry to invest in such a far-reaching technology jump.[7] By 2002, Intel announced that instead of shipping IB integrated circuits ("chips"), it would focus on developing PCI Express, and Microsoft discontinued IB development in favor of extending Ethernet. Sun Microsystems and Hitachi continued to support IB.[8]

In 2003, the System X supercomputer built at Virginia Tech used InfiniBand in what was estimated to be the third largest computer in the world at the time.[9] The OpenIB Alliance (later renamed OpenFabrics Alliance) was founded in 2004 to develop an open set of software for the Linux kernel. By February, 2005, the support was accepted into the 2.6.11 Linux kernel.[10][11] In November 2005 storage devices finally were released using InfiniBand from vendors such as Engenio.[12] Cisco, desiring to keep technology superior to Ethernet off the market, adopted a "buy to kill" strategy. Cisco successfully killed InfiniBand switching companies such as Topspin via acquisition.[13] [citation needed]

Of the top 500 supercomputers in 2009, Gigabit Ethernet was the internal interconnect technology in 259 installations, compared with 181 using InfiniBand.[14] In 2010, market leaders Mellanox and Voltaire merged, leaving just one other IB vendor, QLogic, primarily a Fibre Channel vendor.[15] At the 2011 International Supercomputing Conference, links running at about 56 gigabits per second (known as FDR, see below), were announced and demonstrated by connecting booths in the trade show.[16] In 2012, Intel acquired QLogic's InfiniBand technology, leaving only one independent supplier.[17]

By 2014, InfiniBand was the most popular internal connection technology for supercomputers, although within two years, 10 Gigabit Ethernet started displacing it.[1]

In 2016, it was reported that Oracle Corporation (an investor in Mellanox) might engineer its own InfiniBand hardware.[2]

In 2019 Nvidia acquired Mellanox, the last independent supplier of InfiniBand products.[18]

Specification

[edit]

Specifications are published by the InfiniBand trade association.

Performance

[edit]

Original names for speeds were single-data rate (SDR), double-data rate (DDR) and quad-data rate (QDR) as given below.[12] Subsequently, other three-letter initialisms were added for even higher data rates.[19]

InfiniBand unidirectional data rates
Year[20] Line code Signaling rate (Gbit/s) Throughput (Gbit/s)[21] Adapter latency (μs)[22]
1x 4x 8x 12x
SDR 2001, 2003 NRZ 8b/10b[23] 2.5 2 8 16 24 5
DDR 2005 5 4 16 32 48 2.5
QDR 2007 10 8 32 64 96 1.3
FDR10 2011 64b/66b 10.3125[24] 10 40 80 120 0.7
FDR 2011 14.0625[25][19] 13.64 54.54 109.08 163.64 0.7
EDR 2014[26] 25.78125 25 100 200 300 0.5
HDR 2018[26] PAM4 256b/257b[i] 53.125[27] 50 200 400 600 <0.6[28]
NDR 2022[26] 106.25[29] 100 400 800 1200 ?
XDR 2024[30] [to be determined] 200 200 800 1600 2400 [to be determined]
GDR TBA [to be determined] 400 400 1600 3200 4800
Notes
  1. ^ Using Reed-Solomon forward error correction

Each link is duplex. Links can be aggregated: most systems use a 4 link/lane connector (QSFP). HDR often makes use of 2x links (aka HDR100, 100 Gb link using 2 lanes of HDR, while still using a QSFP connector). 8x is called for with NDR switch ports using OSFP (Octal Small Form Factor Pluggable) connectors "Cable and Connector Definitions".

InfiniBand provides remote direct memory access (RDMA) capabilities for low CPU overhead.

Topology

[edit]

InfiniBand uses a switched fabric topology, as opposed to early shared medium Ethernet. All transmissions begin or end at a channel adapter. Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or quality of service (QoS).

Messages

[edit]

InfiniBand transmits data in packets of up to 4 KB that are taken together to form a message. A message can be:

  • a remote direct memory access read or write
  • a channel send or receive
  • a transaction-based operation (that can be reversed)
  • a multicast transmission
  • an atomic operation

Physical interconnection

[edit]
InfiniBand switch with CX4/SFF-8470 connectors

In addition to a board form factor connection, it can use both active and passive copper (up to 10 meters) and optical fiber cable (up to 10 km).[31] QSFP connectors are used.

The InfiniBand Association also specified the CXP connector system for speeds up to 120 Gbit/s over copper, active optical cables, and optical transceivers using parallel multi-mode fiber cables with 24-fiber MPO connectors.[citation needed]

Software interfaces

[edit]

Mellanox operating system support is available for Solaris, FreeBSD,[32][33] Red Hat Enterprise Linux, SUSE Linux Enterprise Server (SLES), Windows, HP-UX, VMware ESX,[34] and AIX.[35]

InfiniBand has no specific standard application programming interface (API). The standard only lists a set of verbs such as ibv_open_device or ibv_post_send, which are abstract representations of functions or methods that must exist. The syntax of these functions is left to the vendors. Sometimes for reference this is called the verbs API. The de facto standard software is developed by OpenFabrics Alliance and called the Open Fabrics Enterprise Distribution (OFED). It is released under two licenses GPL2 or BSD license for Linux and FreeBSD, and as Mellanox OFED for Windows (product names: WinOF / WinOF-2; attributed as host controller driver for matching specific ConnectX 3 to 5 devices)[36] under a choice of BSD license for Windows. It has been adopted by most of the InfiniBand vendors, for Linux, FreeBSD, and Microsoft Windows. IBM refers to a software library called libibverbs, for its AIX operating system, as well as "AIX InfiniBand verbs".[37] The Linux kernel support was integrated in 2005 into the kernel version 2.6.11.[38]

Ethernet over InfiniBand

[edit]

Ethernet over InfiniBand, abbreviated to EoIB, is an Ethernet implementation over the InfiniBand protocol and connector technology. EoIB enables multiple Ethernet bandwidths varying on the InfiniBand (IB) version.[39] Ethernet's implementation of the Internet Protocol Suite, usually referred to as TCP/IP, is different in some details compared to the direct InfiniBand protocol in IP over IB (IPoIB).

Ethernet over InfiniBand performance
Type Lanes Bandwidth (Gbit/s) Compatible Ethernet type(s) Compatible Ethernet quantity
SDR 1 2.5 GbE to 2.5 GbE 2 × GbE to 1 × 2.5 GbE
4 10 GbE to 10 GbE 10 × GbE to 1 × 10 GbE
8 20 GbE to 10 GbE 20 × GbE to 2 × 10 GbE
12 30 GbE to 25 GbE 30 × GbE to 1 × 25 GbE + 1 × 5 GbE
DDR 1 5 GbE to 5 GbE 5 × GbE to 1 × 5 GbE
4 20 GbE to 10 GbE 20 × GbE to 2 × 10 GbE
8 40 GbE to 40 GbE 40 × GbE to 1 × 40 GbE
12 60 GbE to 50 GbE 60 × GbE to 1 × 50 GbE + 1 × 10 GbE
QDR 1 10 GbE to 10 GbE 10 × GbE to 1 × 10 GbE
4 40 GbE to 40 GbE 40 × GbE to 1 × 40 GbE

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

InfiniBand

View on Grokipedia
from Grokipedia
InfiniBand is an industry-standard, channel-based, interconnect architecture designed for high-performance server-to-server and server-to-storage connectivity in data centers. It enables ultra-low latency data transfer, high bandwidth up to 800 Gb/s per , and efficient (RDMA) operations that bypass the CPU to reduce overhead and improve . Originally conceived as a replacement for traditional bus architectures like PCI, InfiniBand supports topologies with virtual lanes for (QoS) and , making it ideal for demanding environments. The technology originated in the late 1990s when competing proposals for next-generation I/O—such as Future I/O and Next Generation I/O—merged under the InfiniBand Trade Association (IBTA), founded in August 1999 by seven leading companies including Compaq, Dell, HP, IBM, Intel, Microsoft, and Sun Microsystems. The first specification, Volume 1 Release 1.0, was released in 2000, with initial products deploying around 2003 and early adopters like Mellanox (now part of NVIDIA) shipping over 100,000 ports by that time. Over the years, the IBTA—now with over 40 members—has iteratively advanced the architecture, incorporating enhancements like I/O virtualization, software-defined networking (SDN) support via subnet managers, and integration with RDMA over Converged Ethernet (RoCE) since 2010. Recent releases, such as Volume 1 Release 2.0 in 2025, focus on AI-driven workloads with improved memory placement for even lower latency and higher switch density. At its core, InfiniBand employs a layered including physical, , network, and layers, facilitating packet-switched communication over cables (up to 17 meters for lower speeds) or optical fibers (extending to 10 km). It supports multiple data rates, evolving from Single Data Rate (SDR) at 2.5 Gb/s to the latest XDR (Extreme Data Rate) at 800 Gb/s, with full-duplex operation doubling effective throughput. Key architectural strengths include to tens of thousands of nodes per (unlimited via routers), built-in reliability features like error correction and multipathing, and for secure, isolated channels—up to 16 million per host. These elements ensure high (RAS), outperforming Ethernet in latency-sensitive scenarios while complementing it in hybrid fabrics. InfiniBand's primary applications span high-performance computing (HPC) clusters, artificial intelligence (AI) training systems, cloud infrastructure, financial trading platforms, and enterprise storage networks, where it unifies local area networking (LAN) and storage area networking (SAN) functions. In HPC and AI, its low-latency RDMA enables massive parallel processing, as seen in supercomputers and large-scale GPU interconnects from . For storage, it replaces in many data centers, offering higher efficiency and bandwidth for workloads. Overall, InfiniBand delivers superior cost-performance ratios for bandwidth-intensive tasks, with ongoing IBTA roadmaps emphasizing even greater speeds and AI optimizations to meet future demands.

History

Origins and Early Development

In the late , the need for a high-performance interconnect to address the limitations of traditional bus architectures like PCI became evident amid growing demands for scalable server and storage networking in data centers. This led to the formation of the InfiniBand Trade Association (IBTA) on August 27, 1999, through the merger of two competing industry initiatives: Future I/O, backed by , , and , and Next Generation I/O (NGIO), led by with support from and . The IBTA, initially founded by seven leading companies and quickly growing to over 180 members, aimed to develop an for a interconnect that would enable low-latency, high-bandwidth communication across clusters of servers and storage devices. The IBTA released the initial InfiniBand Architecture Specification Version 1.0 on October 24, 2000, defining a channel-based architecture designed to replace the PCI bus with a more scalable, point-to-point serial interconnect supporting data rates up to 2.5 Gbit/s per direction in its single data rate (SDR) configuration. This specification targeted enterprise environments by providing (RDMA) capabilities, remote procedure calls, and reliable transport services, allowing direct data transfer between application memory spaces without CPU intervention or operating system involvement. Key early contributors to the specification's development included , , , , , , and , whose collaborative efforts shifted the industry from proprietary I/O solutions toward a unified . Despite its innovative design, InfiniBand faced significant early adoption challenges in the early , primarily due to from established technologies like Ethernet for general networking and [Fibre Channel](/page/Fibre Channel) for storage area networks, which offered lower initial costs and broader maturity. The technology's in deployment and , coupled with higher upfront hardware expenses compared to incumbents, slowed its penetration into mainstream enterprise markets, though it began gaining traction in clusters where its low-latency advantages proved critical. This transition from proprietary standards to InfiniBand's required substantial industry coordination, ultimately fostering a multi-vendor but initially hindering rapid .

Key Milestones and Industry Acquisitions

The InfiniBand Trade Association released the initial Single Data Rate (SDR) specification in 2000, with commercial products shipping in 2002 at 2.5 Gbit/s per , enabling 10 Gbit/s aggregate speeds for in (HPC) environments. This marked the technology's entry into the market, providing low-latency interconnects for clustered systems. By 2005, the (DDR) specification doubled performance to 10 Gbit/s per or 20 Gbit/s for , broadening in centers and supercomputing clusters through improved bandwidth efficiency. Subsequent generations accelerated InfiniBand's expansion in HPC. The Quad Data Rate (QDR) introduction in 2008 delivered 40 Gbit/s aggregate speeds, enhancing scalability for larger clusters and facilitating broader use in scientific simulations and enterprise storage. This was followed by Fourteen Data Rate (FDR) in 2011, offering 56 Gbit/s signaling rates with 40 Gbit/s effective throughput after encoding, which further reduced latency and supported more efficient in . These advancements solidified InfiniBand's role in supercomputing, where it achieved dominance on the list, powering over 50% of systems during its peak adoption period from 2014 to 2016. A pivotal industry shift occurred in 2019 when announced its acquisition of , the leading InfiniBand hardware provider, for $6.9 billion in cash, completed in 2020 at a value of $7 billion. This merger integrated Mellanox's networking expertise with 's GPU ecosystem, consolidating leadership in InfiniBand development and accelerating innovations for AI and HPC workloads. The High Data Rate (HDR) specification, rolled out in 2018 with 100 Gbit/s per port capabilities (scalable to 200 Gbit/s in full configurations), exemplified this synergy by enabling massive AI training clusters with reduced bottlenecks in data transfer, supporting distributed across thousands of GPUs. Following the acquisition, accelerated InfiniBand advancements, releasing the NDR specification at 400 Gb/s in 2021 for enhanced AI scalability. In 2024, the XDR rate at 800 Gb/s was introduced, with Volume 1 Release 2.0 in 2025 emphasizing AI-driven features like improved latency for large-scale GPU clusters.

Overview

Core Concepts and Design Principles

InfiniBand is an open-standard communications protocol developed for (HPC) and data center environments, enabling high-throughput and low-latency data transfers across clustered systems that can scale to thousands of interconnected nodes. As a networking technology, it addresses the demands of large-scale applications such as scientific simulations, AI training, and by providing a unified fabric for interconnecting servers, storage, and embedded systems over or links. This architecture supports seamless integration of compute, storage, and management traffic, fostering efficient resource utilization in expansive clusters. At its core, InfiniBand utilizes a topology consisting of point-to-point serial links managed by switches, which allows for non-blocking, full-mesh connectivity without the shared-medium limitations of bus-based systems. A key enabler is (RDMA), which permits direct data movement between the memory of participating nodes, circumventing the host CPU, operating system, and software protocol stacks to minimize latency and CPU overhead. This RDMA capability ensures efficient, kernel-bypass operations, where data is transferred with hardware-enforced reliability and ordering. InfiniBand's transport layer offers flexible services, including reliable connection (RC) for guaranteed, in-order delivery with acknowledgments and retransmissions, and unreliable datagram (UD) for lightweight, connectionless messaging suitable for broadcast scenarios. It also incorporates atomic operations for or functions across the fabric, enabling synchronized access to remote without software locks, as well as support for one-to-many data distribution in group communications. These features, implemented via queue pairs—paired send and receive work queues—facilitate diverse messaging patterns essential for parallel processing. The design principles of InfiniBand prioritize to support HPC workloads by allowing expansion through additional switches, maintaining across vast topologies without central bottlenecks. It achieves ultra-low end-to-end latency, measured at approximately 600 ns in modern implementations, through hardware-accelerated flow control, credit-based mechanisms, and virtual lanes for . Bandwidth aggregation is optimized via the switched fabric's non-contended paths, ensuring collective throughput scales linearly with node count and link widths, thus avoiding oversubscription issues common in hierarchical networks.

Advantages Over Competing Technologies

InfiniBand provides superior latency reduction through its support for (RDMA) and kernel bypass, allowing direct data transfers between application memory spaces without involving the operating system kernel or CPU interrupts. This results in end-to-end latencies as low as 1-3 microseconds, compared to 10-80 microseconds for traditional TCP/IP over Ethernet, enabling 2-5 times faster data transfers in environments. In terms of throughput, InfiniBand scales efficiently to Gbit/s per in its XDR generation, with aggregated fabric bandwidth reaching terabits per second in large-scale deployments, while avoiding through its credit-based flow control and virtual lane architecture. This makes it particularly suitable for workloads, where Ethernet can suffer from contention and reduced effective bandwidth under heavy loads. Unlike , which is optimized for storage with latencies around 5-10 microseconds and bandwidths up to 128 Gbit/s, InfiniBand delivers higher aggregate throughput for compute-intensive tasks without the protocol overhead of encapsulation. InfiniBand incorporates built-in (QoS) mechanisms and advanced congestion control, such as Forward and Backward (FECN/BECN), which ensure predictable performance and minimal in AI and workloads. This contrasts with Ethernet's potential for packet drops and retransmissions in congested networks without lossless mechanisms like Priority Flow Control; InfiniBand's lossless fabric prevents packet drops entirely, providing deterministic delivery. For cost-efficiency in large-scale deployments, InfiniBand reduces CPU overhead by offloading network processing to hardware, significantly reducing CPU utilization compared to Ethernet-based TCP/IP stacks, and offers lower energy consumption per bit than proprietary alternatives like due to its efficient design. This translates to reduced operational expenses in power and cooling for hyperscale clusters. A key example of InfiniBand's advantages is its role in enabling systems, where its low-latency RDMA and high-bandwidth scaling support synchronization across millions of nodes without the overhead that plagues Ethernet in massive parallel simulations; for instance, it powers approximately 50% of the world's top supercomputers as of November 2024, facilitating the transition to 1 exaFLOPS performance.

Architecture

Physical Layer Specifications

The InfiniBand physical layer defines the electrical and optical signaling characteristics, cabling, and hardware interfaces that enable high-speed, low-latency data transmission between devices. It supports serial, point-to-point connections using differential signaling over twisted-pair or multimode/single-mode , with provisions for multiple data rates to accommodate evolving needs in high-performance computing environments. InfiniBand employs different encoding schemes depending on the speed generation to balance , , and bandwidth . Single Data Rate (SDR) and (DDR) use 8b/10b encoding, which maps 8-bit data to 10-bit symbols for DC balance and sufficient transitions, achieving approximately 80% . Quad Data Rate (QDR) also utilizes 8b/10b encoding. Starting with Fourteen Data Rate (FDR), higher generations such as Enhanced Data Rate (EDR), High Data Rate (HDR), Next Data Rate (NDR), and XDR adopt , which improves to about 97% by adding only 2 sync bits to 64-bit blocks, reducing overhead while maintaining robust detection. Lane configurations in InfiniBand ports are denoted as 1x, , or 12x, referring to the number of parallel differential pairs (lanes) for transmit and receive, enabling scalable bandwidth. Each operates independently but synchronously, with auto-negotiation determining the active width during link initialization. Signaling rates vary by generation, as summarized in the following table for representative 1x configurations (full-duplex data rates are double the per-direction values):
GenerationSignaling Rate per Lane (Gb/s)EncodingEffective Data Rate per Lane (Gb/s)
SDR2.58b/10b2.0
DDR5.08b/10b4.0
QDR10.0 (raw, post-encoding adjustment)8b/10b8.0
FDR14.062564b/66b13.64
EDR25.7812564b/66b25.0
HDR53.12564b/66b50.0 (PAM4 for higher variants)
NDR100.0 (NRZ/PAM4 hybrid)64b/66b96.97 (effective)
XDR206.2564b/66b200.0 (PAM4)
For example, a 4x HDR achieves up to 200 Gb/s aggregate per direction. These rates support cabling for short distances and optical for longer reaches, with (FEC) integrated in higher generations for mitigation below 10^{-15}. Connector types for InfiniBand ports are standardized for multi-lane operations, primarily using Quad Small Form-factor Pluggable (QSFP) variants to accommodate 4x configurations common in modern deployments. QSFP (for QDR/FDR/EDR) and QSFP28 (for HDR) support up to four lanes, while QSFP-DD (Double Density) and OSFP enable eight lanes for NDR/XDR speeds up to 800 Gb/s aggregate. These connectors interface with direct-attach (DAC) cables for distances up to 5 meters in passive variants or 7-10 meters active, and active optical cables (AOC) or transceivers for up to 100 meters over multimode fiber, with single-mode options extending to 10 km via . Copper cables are cost-effective for intra-rack connections, while optical media provide scalability for inter-rack fabrics. The physical layer interfaces through Host Channel Adapters (HCAs) for server or host connections and Target Channel Adapters (TCAs) for storage or peripheral devices, both implementing the full protocol stack up to the physical layer. HCAs typically feature PCIe interfaces and one or two ports, while TCAs offer a subset of capabilities for I/O targets. Power consumption for these adapters ranges from 10-25 W per port, depending on speed and form factor; for instance, HDR HCAs draw around 15-20 W, emphasizing efficient serdes and low-jitter clocking to minimize thermal overhead in dense clusters. Backward compatibility across generations is ensured through auto-negotiation protocols during link training, allowing newer ports (e.g., NDR) to operate at lower speeds (e.g., HDR or FDR) when connected to legacy hardware or cables, provided the connector and lane width match. This process involves exchanging capabilities via training sequences and settling on the highest mutually supported rate and width, facilitating incremental upgrades without full fabric replacement. InfiniBand networks employ scalable topologies such as and to achieve non-blocking communication, ensuring consistent bandwidth across all levels without hotspots that plague traditional tree structures. The , a multi-level variant, maintains identical upstream and downstream bandwidth at each layer, supporting non-blocking for large clusters; for instance, a two-level configuration with 8-port switches can connect 16 endpoints using four leaf switches. topologies, often implemented as 2D or 3D tori, provide regular connectivity where each node links to four neighbors along axes, offering predictable latency in grid-like arrangements suitable for scientific simulations. These topologies enable InfiniBand to scale to up to 48,000 nodes in a single subnet, facilitating environments without performance degradation. Subnet management in InfiniBand is handled by the Subnet Manager (SM), a centralized or distributed software entity that discovers the , assigns Local Identifiers (LIDs) to each port, and programs tables for efficient . The SM performs a fabric scan to detect all switches, host channel adapters (HCAs), and links, then allocates 16-bit LIDs (ranging from 1 to 48,000 in practical deployments) to enable local within the subnet, ensuring unique addressing and path computation via algorithms like MinHop or Up/Down. tables are constructed to avoid loops and optimize paths, with support for up to 2048 nodes in typical SM implementations, though the architecture scales to larger fabrics through hierarchical management. Multiple SMs can operate for , with one acting as the master to coordinate LID assignments and table updates during changes. For larger deployments exceeding single-subnet limits, InfiniBand supports multi-subnet configurations interconnected via routers, which forward traffic between independent fabrics using global identifiers (GUIDs) for addressing. Each InfiniBand device possesses a 64-bit GUID burned into hardware, serving as a unique endpoint identifier similar to a ; in multi-subnet routing, packets include a Global Routing Header (GRH) with the destination GUID, allowing routers to resolve paths without local LIDs. Routers maintain separate forwarding tables per , enabling fault isolation and to over 40,000 end-ports across interconnected domains, while preserving autonomy for management and security. This global routing mechanism ensures seamless communication in expansive clusters, such as those in infrastructures. InfiniBand incorporates adaptive algorithms to manage failures and balance load, operating in deterministic or randomized modes to enhance reliability and throughput. Deterministic routing selects fixed paths based on source-destination pairs via static tables, minimizing latency in uniform traffic but risking congestion in fat-trees; adaptive modes, enabled at switches, dynamically choose egress ports using port-load feedback, such as round-robin randomization or tree-specific heuristics, to distribute flows and mitigate hotspots. These algorithms handle link or node failures by recomputing paths through SM notifications, improving bandwidth utilization by up to 25% in non-uniform workloads compared to deterministic approaches, while supporting features like SHARP for congestion avoidance. Within physical links, InfiniBand uses up to 16 virtual lanes (VLs) to multiplex independent data streams, providing traffic isolation and prioritization for (QoS). Each VL operates as a logical channel sharing the physical link's bandwidth, mapped from service levels (SLs) via switch tables to prevent interference; for example, high-priority HPC traffic can be assigned to dedicated VLs, ensuring low latency unaffected by bulk transfers on others. This mechanism supports weighted round-robin arbitration at switches, with credits for flow control per VL, enabling efficient resource allocation in mixed workloads without .

Protocols and Performance

Transport Layer and Message Handling

The InfiniBand transport layer manages end-to-end data transfer between communicating entities, utilizing Queue Pairs (QPs) as the primary mechanism for handling messages and ensuring reliable communication within the fabric. It operates above the network layer and provides services that support both connection-oriented and connectionless paradigms, enabling efficient data movement with minimal host CPU involvement. The layer processes packets by interpreting headers, managing acknowledgments, and coordinating retransmissions where required, all while adhering to the overall InfiniBand architecture defined in the specification. InfiniBand packets include structured headers for and transport control, with the Local Route Header (LRH) mandatory for all intra-subnet communications and the Global Route Header (GRH) used for inter-subnet or when global identifiers are necessary. The LRH, an 8-byte structure, contains fields such as Virtual Lane (VL) for prioritization, Service Level (SL) for , Destination Local Identifier (DLID) and Source Local Identifier (SLID) for local addressing, and packet length to facilitate switch forwarding within a single . The GRH, a 40-byte IPv6-compatible header, extends this with Source Global Identifier (SGID) and Destination Global Identifier (DGID) for broader , along with class and flow label fields to support subnet-to-subnet transfers. Following these headers, the can reach up to 4 KB in size, allowing for efficient segmentation of larger messages into multiple packets when exceeding the Path MTU. The defines several services to handle message delivery, with Reliable Connection (RC) providing a connection-oriented mechanism that guarantees ordered, error-free delivery through explicit acknowledgments and automatic retransmissions managed by the hardware. In contrast, Unreliable (UD) offers a lightweight, connectionless alternative for fire-and-forget transmissions, ideal for scenarios like where reliability is not assured but is prioritized. These services are selected based on the QP type during connection establishment, ensuring the appropriate level of overhead and guarantees for the application. Remote Direct Memory Access (RDMA) operations are a of the , enabling direct data placement into or from application without kernel or CPU intervention, thus reducing latency and overhead. The Send/Receive operation transfers data between sender and receiver buffers via work requests posted to the QP, with the receiver pre-allocating space for incoming messages. RDMA Write allows the initiator to directly write data to a remote specified by a virtual address and protected by a Remote Key (R_Key), supporting immediate data or descriptor-based transfers. RDMA Read pulls data from a remote location similarly secured by an R_Key, with the handling any necessary segmentation. Atomic operations, such as and , perform lock-free 64-bit manipulations on remote , ensuring atomicity across the fabric for synchronization primitives. Flow control in the prevents buffer overflows through a credit-based system, where receivers advertise available buffer space to senders via periodic updates. At the end-to-end level for RC services, the Acknowledged Extended Transport Header (AETH) conveys credit returns alongside acknowledgments, allowing senders to track and respect buffer limits. Link-level flow control complements this with Go/No-Go signals, where senders transmit only when credits indicate sufficient space per Virtual Lane, and mechanisms like Negative Acknowledgments (NAKs) or RESYNC packets enforce pauses during congestion or errors. Error handling ensures data integrity through (CRC) mechanisms embedded in packets, with the Variant CRC (VCRC) providing 2-byte link-layer protection and the Invariant CRC (ICRC) offering 4-byte end-to-end verification against corruption during transit. Detected errors trigger QP state transitions to an error condition, potentially initiating retransmissions for reliable services or silent discards for unreliable ones. In higher-speed generations, (FEC) augments this by proactively correcting bit errors at the using low-latency codes, as introduced in InfiniBand Volume 2 Release 1.3.1 to maintain reliability without impacting performance.

Performance Metrics and Benchmarks

InfiniBand's performance has evolved significantly across its generations, with effective bandwidths scaling from 8 Gbit/s in Single Data Rate (SDR) to 200 Gbit/s in High Data Rate (HDR) by 2020. SDR operates at a raw signaling rate of 2.5 GT/s per lane, delivering a raw bandwidth of 10 Gbit/s for a 4x configuration using 8b/10b encoding (effective bandwidth of 8 Gbit/s). Subsequent generations improved this progression: Double Data Rate (DDR) at 20 Gbit/s, Quad Data Rate (QDR) at 40 Gbit/s, Fourteen Data Rate (FDR) at 56 Gbit/s with 64b/66b encoding for higher efficiency, Enhanced Data Rate (EDR) at 100 Gbit/s, and HDR at 200 Gbit/s. Later developments include NDR at 400 Gbit/s (announced in 2020) and XDR at 800 Gbit/s (announced in 2023) by NVIDIA and the InfiniBand Trade Association to support AI and scientific computing demands. NDR achieves latencies around 0.5 μs for small messages in benchmarks, while XDR targets similar or lower with enhanced FEC and AI-specific features, as per the 2025 specifications (Volume 1 Release 2.0). Latency in InfiniBand remains a key strength, typically sub-microsecond for small messages, enabling rapid data exchange in environments. For instance, RDMA latency measures around 0.6 μs in EDR and HDR configurations using ConnectX adapters. In OSU Micro-Benchmarks, small-message latency hovers at 0.8 μs for both FDR and EDR, while 4 KB messages exhibit 2.75-2.84 μs. For larger transfers in 4x HDR setups, latency scales to 5-10 μs, maintaining efficiency through (RDMA) mechanisms that minimize software intervention. Bandwidth utilization approaches near 100% in aggregated fabrics, as demonstrated by RDMA benchmarks showing over 95 Gbit/s on FDR links and 99.2 Gbit/s unidirectional on EDR. In HDR environments, bidirectional throughput reaches 193.6 Gbit/s for 4 MB messages, reflecting high efficiency in multi-node setups. This scalability supports massive parallelism without significant bottlenecks. InfiniBand's kernel bypass via RDMA results in low CPU overhead, typically under 5% and often below 2% during high-throughput operations, compared to 20-30% or higher in traditional Ethernet due to protocol stack processing. This efficiency frees computational resources for application workloads, as evidenced in benchmarks saturating links with minimal processor involvement. Effective bandwidth can be calculated as: Effective BW=(Raw signaling rate×Lanes×Encoding efficiency)−Overhead\text{Effective BW} = (\text{Raw signaling rate} \times \text{Lanes} \times \text{Encoding efficiency}) - \text{Overhead} For a HDR configuration, with 50 GT/s raw rate per lane and (98% efficiency), this yields approximately 196 Gbit/s after minor protocol overhead.

Software and Interoperability

Host Software Interfaces and Drivers

The Verbs API serves as the foundational interface for applications to interact with InfiniBand hardware, enabling (RDMA) operations such as send/receive, RDMA read/write, and atomic operations directly from user space. Defined in Chapter 11 of the InfiniBand Architecture Specification by the InfiniBand Trade Association (IBTA), the API abstracts hardware details into a set of function calls, or "verbs," that manage queue pairs (QPs), completion queues (CQs), and domains for secure, low-latency data transfer. Recent updates in IBTA Volume 1 Release 1.8 (July 2024) and Release 2.0 (July 2025) include security enhancements to verbs like Modify QP to verify valid destination queue pairs and addresses, preventing user-mode attacks. The libibverbs library implements this API in user space, providing a portable interface across InfiniBand host channel adapters (HCAs) while the kernel module ib_uverbs handles system calls for and . On Linux systems, the OpenFabrics Enterprise Distribution (OFED) stack provides the primary software ecosystem for InfiniBand, encompassing kernel drivers, user-space libraries, and utilities to enable full fabric functionality. 's MLNX_OFED (now integrated into DOCA-OFED) includes the mlx5 driver for ConnectX-series HCAs, supporting InfiniBand speeds up to 800 Gb/s (XDR) with ConnectX-8 and later, RDMA and kernel bypass for reduced CPU overhead. This stack facilitates integration with workloads through bundled MPI implementations, such as , which leverages native InfiniBand transports for collective operations and point-to-point messaging. Additionally, IP over InfiniBand (IPoIB) support in OFED allows standard IP networking over InfiniBand fabrics, encapsulating packets for compatibility with legacy applications while maintaining RDMA capabilities for optimized traffic. Windows environments utilize the WinOF driver suite from for InfiniBand connectivity, offering RDMA-enabled support on ConnectX adapters for high-throughput applications. WinOF-2, applicable to ConnectX-4 and later cards, includes protocols for storage, cloud, and HPC scenarios, with installation via MSI packages that integrate seamlessly into editions. enhances this through extensions in Microsoft MPI (MS-MPI), enabling InfiniBand-optimized clustering for distributed workloads like simulations and data analytics on Azure VMs or on-premises setups. InfiniBand management relies on tools embedded in the OFED and WinOF stacks for diagnostics, configuration, and monitoring. OpenSM acts as the reference subnet manager, an IBTA-compliant daemon that discovers topology, assigns local identifiers (LIDs), routes traffic, and enforces quality-of-service policies across the fabric. For host-level diagnostics, ibstat queries basic HCA status, including port states, link widths, and physical connectivity, while ibv_devinfo provides detailed RDMA device attributes such as node GUIDs, firmware versions, and queue pair limits. Security in InfiniBand host software emphasizes isolation and authentication through built-in mechanisms like partition keys (P_Keys) and integration with for IP-based traffic. P_Keys, 16-bit values enforced at the HCA and switch levels, segment the fabric into isolated partitions, preventing unauthorized communication between groups; full members (P_Key with MSB=1) can access all ports, while limited members (MSB=0) are restricted, with the default P_Key of 0x7FFF ensuring basic separation. For IPoIB traffic, OFED integrates to provide encryption and integrity, offloading cryptographic operations to the HCA where supported, thus securing IP datagrams without compromising RDMA performance.

Extensions like RoCE and Ethernet Convergence

(RoCE) extends InfiniBand's (RDMA) capabilities to Ethernet networks by mapping the InfiniBand directly onto Ethernet frames, enabling low-latency, high-throughput data transfers without kernel involvement. RoCE requires a lossless Ethernet fabric to maintain performance, typically achieved through enhancements like Data Center Bridging (DCB). RoCEv1 operates at Layer 2 of the , using a dedicated of 0x8915 for the RDMA header, which limits it to communication within the same or . In contrast, RoCEv2 encapsulates RDMA packets within UDP/IP headers, adding routability at Layer 3 to support traffic across IP-routed networks while preserving the efficiency of InfiniBand's transport semantics. This mapping allows RoCE to deliver InfiniBand-like performance over standard Ethernet infrastructure, with end-to-end latencies typically in the low microseconds for point-to-point connections, slightly higher than native InfiniBand. Ethernet over InfiniBand (EoIB) provides a complementary extension by tunneling standard Ethernet frames over an InfiniBand fabric, facilitating the integration of legacy Ethernet-based applications and devices into InfiniBand environments without requiring hardware changes. EoIB encapsulates Ethernet packets within InfiniBand datagrams, supporting multiple virtual Ethernet networks and enabling scalable bandwidth aggregation based on the underlying InfiniBand link speed. Extensions like Single Root I/O Virtualization (SR-IOV) enhance InfiniBand's support for virtualized environments by allowing physical InfiniBand host channel adapters (HCAs) to present multiple virtual functions to , reducing virtualization overhead and enabling direct RDMA access from guest operating systems. Similarly, NVMe over Fabrics (NVMe-oF) leverages InfiniBand and RoCE transports to disaggregate NVMe storage across the network, providing converged I/O for storage and compute with sub-microsecond latencies and high scalability in multi-host setups. These features promote a unified fabric for networking and storage, minimizing the need for separate LAN and SAN infrastructures while supporting virtualization demands in data centers. The InfiniBand Trade Association (IBTA) standardizes these extensions, including RoCE protocols and mechanisms for Ethernet convergence, to ensure interoperability and compliance across implementations. A key enabler for RoCE's lossless operation is Priority Flow Control (PFC), an IEEE 802.1Qbb standard integrated into IBTA specifications, which applies pause frames on a per-priority basis to prevent frame drops and maintain low-latency performance akin to native InfiniBand's credit-based flow control. Overall, these adaptations achieve convergence benefits such as reduced cabling complexity and operational costs, with RoCE delivering low-microsecond latencies in optimized configurations.

Applications and Adoption

Role in High-Performance Computing

InfiniBand serves as a critical interconnect in (HPC), enabling efficient communication among thousands of nodes in supercomputing environments to support large-scale scientific simulations. Its low-latency and high-bandwidth capabilities make it ideal for workloads requiring massive parallelism, such as those in national laboratories and research institutions. In the list of the world's most powerful supercomputers, InfiniBand has maintained significant presence, powering 223 out of 500 systems in the November 2025 ranking (44.6% market share), down slightly from 50.8% (254 systems) in November 2024 and up from 40% (200 systems) in June 2023. This prevalence underscores InfiniBand's role in achieving petascale and exascale performance, as seen in systems like the JUPITER Booster supercomputer, which utilized Quad-Rail InfiniBand NDR200 to deliver 1.000 exaFLOPS on the High-Performance Linpack benchmark in 2025. InfiniBand's support for the (MPI) standard facilitates parallel processing across distributed nodes, allowing seamless data exchange in computationally intensive applications. Libraries like MVAPICH2 and OpenMPI, optimized for InfiniBand, enable efficient implementation of MPI collectives and point-to-point communications, reducing overhead in systems. This integration has been pivotal in petascale simulations, such as high-resolution climate modeling with the Community Atmosphere Model on InfiniBand clusters, achieving scalable performance for global atmospheric predictions. Similarly, in physics simulations like finite-difference modeling, InfiniBand's reliable transport supports the of complex multiphysics computations across thousands of cores. In the convergence of AI and HPC, InfiniBand provides low-latency interconnects for GPU clustering, essential for multi-node training of large-scale models. NVIDIA's DGX systems, for example, leverage InfiniBand in SuperPOD configurations to enable high-throughput data movement between GPUs, supporting distributed workloads with minimal synchronization delays. For storage fabrics, InfiniBand integrates with parallel filesystems like Lustre and GPFS, delivering aggregate I/O bandwidth exceeding 200 GB/s in HPC environments, as demonstrated in benchmarks on systems with multiple targets. This capability ensures efficient parallel access to massive datasets in simulation pipelines. The JUPITER Booster supercomputer exemplifies InfiniBand's impact in , where its non-blocking fabric minimized interconnect overhead, contributing to sustained performance near 1 exaFLOPS for scientific discovery in fields like and fusion energy.

Deployment in Data Centers and Cloud Environments

InfiniBand has seen significant adoption in enterprise data centers and cloud environments, particularly among hyperscalers seeking high-bandwidth, low-latency interconnects for compute-intensive workloads. , for instance, integrates InfiniBand in its ND-series virtual machines, such as the ND-H100-v5 instances, which provide each GPU with a dedicated 400 Gb/s Quantum-2 CX7 InfiniBand connection to support scalable AI and tasks. This deployment enables non-blocking fat-tree networks with up to 3.2 Tb/s per VM, optimizing resource utilization in cloud-based simulations and applications. In AI data centers, NVIDIA's BlueField Data Processing Units (DPUs) leverage InfiniBand to facilitate disaggregated computing architectures and enhance smart NIC capabilities. BlueField-3 and later models support up to 400 Gb/s InfiniBand connectivity, offloading networking, storage, and tasks from host CPUs to enable efficient resource pooling across distributed AI infrastructures. This integration allows data centers to scale AI factories by disaggregating compute, storage, and networking layers, reducing latency in GPU-to-GPU communications for training large-scale models. The InfiniBand market is experiencing robust growth, projected to expand from USD 25.74 billion in 2025 to USD 126.99 billion by 2030, driven by surging AI demand in cloud and environments, with a (CAGR) of 37.60%. This trajectory reflects InfiniBand's role in supporting bandwidth-hungry applications like generative AI and processing. Hybrid deployments combining InfiniBand spines with Ethernet leaves are emerging in 2025 architectures to balance performance and cost in large-scale . These configurations use InfiniBand for high-speed core interconnects in AI clusters while employing Ethernet for edge connectivity, enabling interoperability and optimized scaling in mixed workloads. Despite higher initial costs compared to Ethernet—often due to specialized hardware—InfiniBand delivers (TCO) savings in bandwidth-intensive scenarios like analytics, where its superior throughput and lower latency reduce overall cluster size and energy consumption.

Standards and Future Directions

Evolution of InfiniBand Specifications

The InfiniBand Trade Association (IBTA), founded in 1999 through the merger of the Future I/O and Next Generation I/O forums, has governed the development and maintenance of royalty-free InfiniBand specifications since the release of Version 1.0 in October 2000. The IBTA ensures to these specifications, promoting widespread adoption by making them available for download without licensing fees, while fostering collaboration among members to evolve the standard for high-performance interconnects. The progression of InfiniBand generations has focused on doubling data rates to meet escalating demands in and data centers. Enhanced Data Rate (EDR) at 100 Gbit/s per emerged in 2015, building on prior generations like FDR (56 Gbit/s) with improved signaling efficiency. High Data Rate (HDR) followed in 2018 at 200 Gbit/s, introducing with four levels (PAM4) to achieve higher bandwidth density while maintaining low latency. Next Data Rate (NDR) arrived in 2021 at 400 Gbit/s, enhancing for large-scale fabrics through refined error correction and link training. Extreme Data Rate (XDR) at 800 Gbit/s was specified in 2023, with commercial products released in 2025 to support next-generation AI workloads. Key specification updates have driven these advancements, with Volume 1 Release 1.3 (2015) formalizing EDR support, Release 1.4 (2020) enabling HDR via PAM4 modulation for denser port configurations, Release 1.5 (2021) adding NDR capabilities including updated virtual lane arbitration for , and Release 1.7 (2023) defining XDR alongside enhancements for high- switches. Volume 1 and 2 Release 2.0, published on July 31, 2025, further advances XDR with features like , support for large switches up to 64K ports, enhanced network probes for congestion control and in AI systems, and CMIS 5.3 for transceivers. These releases incorporate iterative refinements to signaling and transport protocols, prioritizing without disrupting existing deployments. To ensure ecosystem reliability, the IBTA's Compliance and Interoperability Working Group (CIWG) administers the Logo Program, which certifies products through rigorous testing at annual Plugfests. This program verifies adherence to Volumes 1 (general and transport layers) and 2 (physical layer) of the specification, focusing on link negotiation, error handling, and multi-vendor fabric integration to guarantee seamless operation. InfiniBand's design emphasizes backward and , enabling mixed-generation fabrics through auto-negotiation protocols that detect and fallback to the lowest common rate and width during link initialization. For instance, an NDR port can automatically operate at HDR or EDR speeds when connected to legacy components, supporting gradual upgrades in heterogeneous environments without requiring full fabric replacement. InfiniBand's Next Data Rate (NDR) specification, operating at 400 Gb/s, saw initial deployments in AI superclusters as early as 2022, with NVIDIA's Quantum-2 platform enabling connections for tens of thousands of GPUs in systems like Microsoft Azure's advanced supercomputing clusters. These implementations provided low-latency, high-bandwidth fabrics essential for large-scale AI training, supporting up to 16,000 GPU endpoints in non-oversubscribed topologies. By 2024, prototypes for the Extreme Data Rate (XDR) specification emerged, targeting 800 Gb/s throughput with 200 Gb/s per electrical lane via advanced technology and optics, as demonstrated in NVIDIA's Quantum Q3400 switch unveiled at GTC 2024. These XDR prototypes leverage integrated photonics to extend reach and reduce latency, paving the way for trillion-parameter AI models in hyperscale environments. Integration of InfiniBand with (CXL) is gaining traction for memory pooling in disaggregated computing systems, where InfiniBand handles inter-node interconnects while CXL enables intra-node coherent memory sharing. This hybrid approach allows compute nodes to access pooled memory resources across clusters with minimal latency overhead compared to traditional RDMA over InfiniBand alone, as shown in prototypes like KAIST's DirectCXL, which achieves up to 3x better performance in disaggregated workloads by combining CXL's cache-coherent fabric with InfiniBand's scalable bandwidth. Such integrations support dynamic resource allocation in AI and HPC setups, reducing underutilization in memory-intensive tasks like fine-tuning. Sustainability efforts in InfiniBand focus on power-efficient designs, with co-packaged (CPO) in recent platforms delivering a 3.5x improvement in energy efficiency over prior architectures by integrating lasers directly onto switches, minimizing electrical-to-optical conversion losses. These advancements align with initiatives, enabling higher throughput per watt in AI factories while supporting environmental goals through reduced cooling demands and resilient, low-failure-rate links. NVIDIA's Quantum-X series exemplifies this trend, incorporating CPO to cut power consumption in high-density deployments. Competition from Ethernet is intensifying, particularly with 800G standards, but NVIDIA's ecosystem promotes convergence by offering dual-support in SuperNICs like ConnectX-8, which handles both InfiniBand and Ethernet protocols for seamless hybrid fabrics. This push facilitates transitions in AI infrastructures, where InfiniBand's RDMA advantages complement Ethernet's cost scalability, potentially merging ecosystems for broader adoption in and edge environments. The InfiniBand market is projected to exceed $18 billion by the end of 2025, driven by surging demand in AI clusters and emerging quantum-HPC hybrid systems that leverage its low-latency interconnects for integrating classical and quantum workloads. Growth is fueled by AI's need for scalable, predictable networking, with InfiniBand maintaining dominance in 90% of large AI deployments despite Ethernet's rise.

References

  1. https://www.infinibandta.org/ibta-specification/
  2. https://network.nvidia.com/pdf/whitepapers/IB_Intro_WP_190.pdf
  3. https://docs.nvidia.com/networking/display/RDMAAwareProgrammingv17/InfiniBand
  4. https://network.nvidia.com/pdf/whitepapers/idc_whitepaper_0101.pdf
  5. http://buyya.com/superstorage/chap42.pdf
  6. https://www.afs.enea.it/asantoro/V1r1_2_1.Release_12062007.pdf
  7. https://www.hpcwire.com/2000/10/27/infiniband-technology-may-remake-servers/
  8. https://www.infinibandta.org/wp-content/uploads/2018/03/Taneja_Analyst_Report_-_InfiniBands_Data_Center_-_July_2012.pdf
  9. https://www.computerworld.com/article/1353576/infiniband.html
  10. https://network.nvidia.com/pdf/whitepapers/WP_FDR_InfiniBand_is_Here.pdf
  11. https://www.hpcwire.com/2008/11/25/qlogic_completes_home-grown_infiniband_strategy/
  12. https://www.infinibandta.org/infiniband-trade-association-announces-45-percent-of-worlds-most-powerful-computers-listed-on-top500-use-infiniband/
  13. https://www.infinibandta.org/infiniband-leads-the-top500-powering-more-than-50-percent-of-the-worlds-supercomputing-systems/
  14. https://nvidianews.nvidia.com/news/nvidia-to-acquire-mellanox-for-6-9-billion
  15. https://nvidianews.nvidia.com/news/nvidia-completes-acquisition-of-mellanox-creating-major-force-driving-next-gen-data-centers
  16. https://www.infinibandta.org/top500-list-results-hdr-200gb-s-infiniband-powers-the-worlds-fastest-supercomputer/
  17. https://www.infinibandta.org/about-infiniband/
  18. https://docs.nvidia.com/networking/display/RDMAAwareProgrammingv17/Transport%2BModes
  19. https://www.ufispace.com/company/blog/compare-infiniband-vs-ethernet
  20. https://www.whitefiber.com/blog/ethernet-vs-infiniband
  21. https://www.nas.nasa.gov/assets/nas/pdf/papers/NAS_Technical_Report_NAS-2014-01.pdf
  22. https://docs.nvidia.com/networking/display/RDMAAwareProgrammingv17/Comparison%2Bof%2BRDMA%2BTechnologies
  23. https://www.infinibandta.org/ibta-unveils-xdr-infiniband-specification-to-enable-the-next-generation-of-ai-and-scientific-computing/
  24. https://network.nvidia.com/pdf/whitepapers/deploying_qos_wp_10_19_2005.pdf
  25. https://www.fs.com/blog/infiniband-vs-ethernet-what-are-the-differences-2740.html
  26. https://www.fibermall.com/blog/what-is-infiniband-network-and-difference-with-ethernet.htm
  27. https://www.aptlytech.com/infiniband-vs-ethernet-which-fits-the-best-for-a/
  28. https://www.infinibandta.org/infiniband-and-roce-advances-further-in-the-top500-november-2024-list/
  29. https://www.weka.io/learn/enterprise-technology/nvidia-infiniband/
  30. https://docs.nvidia.com/networking/display/SB77X0EDR/Interfaces
  31. https://docs.nvidia.com/networking/display/QM97X0PUB/Interfaces
  32. https://www.infinibandta.org/wp-content/uploads/2025/11/Whats-New-in-IBTA-Specification-Release-2.0-2025-11-05.pdf
  33. https://www.infinibandta.org/infiniband-roadmap-charting-speeds-for-future-needs/
  34. https://massedcompute.com/faq-answers/?question=What%2520are%2520the%2520power%2520requirements%2520for%2520InfiniBand%2520connections%2520with%2520NVIDIA%2520H100%2520GPUs%2520in%2520a%2520data%2520center%2520environment.
  35. https://www.infinibandta.org/wp-content/uploads/2019/10/Integrators-List-PF32.pdf
  36. https://docs.nvidia.com/networking/display/ConnectX7Firmwarev28352000LTS/Firmware%2BCompatible%2BProducts
  37. https://support.hpe.com/hpesc/public/docDisplay?docId=a00048433en_us&page=GUID-B3AEDA98-5262-4BA6-89FB-E5840AD901B0.html&docLocale=en_US
  38. https://massedcompute.com/faq-answers/?question=Can%2520InfiniBand%2520HDR%2520and%2520NDR%2520be%2520used%2520with%2520existing%2520InfiniBand%2520FDR%2520switches?
  39. https://network.nvidia.com/pdf/whitepapers/IB_vs_Ethernet_Clustering_WP_100.pdf
  40. https://enterprise-support.nvidia.com/s/article/designing-an-hpc-cluster-with-mellanox-infiniband-solutions
  41. https://network.nvidia.com/pdf/whitepapers/InfiniBandFAQ_FQ_100.pdf
  42. https://docs.nvidia.com/networking/display/MLNXOSv3112006/Subnet%2BManager
  43. https://www.cs.fsu.edu/~xyuan/paper/08tpds.pdf
  44. https://docs.nvidia.com/networking/display/MLNXOSv3106004/IB%2BRouter
  45. https://enterprise-support.nvidia.com/s/article/ib-router-architecture-and-functionality
  46. https://web-backend.simula.no/sites/default/files/publications/files/adaptive_routing_in_infiniband_hardware.pdf
  47. https://htor.inf.ethz.ch/publications/img/hoefler-ib-routing.pdf
  48. https://docs.nvidia.com/doca/sdk/InfiniBand%2BQoS/index.html
  49. https://www.infinibandta.org/new-infiniband-specification-updates-expand-interoperability-flexibility-and-virtualization-support/
  50. https://www.advancedclustering.com/act_kb/infiniband-types-speeds/
  51. https://www.hpcwire.com/2020/11/16/nvidia-mellanox-debuts-ndr-400-gigabit-infiniband-at-sc20/
  52. https://nebius.com/blog/posts/what-is-infiniband
  53. https://network.nvidia.com/files/doc-2020/br-infiniband-product-guide.pdf
  54. https://dl.dell.com/manuals/all-products/esuprt_software/esuprt_it_ops_datcentr_mgmt/high-computing-solution-resources_white-papers77_en-us.pdf
  55. https://www.researchgate.net/publication/343269790_Analyzing_and_Understanding_the_Impact_of_Interconnect_Performance_on_HPC_Big_Data_and_Deep_Learning_Applications_A_Case_Study_with_InfiniBand_EDR_and_HDR
  56. https://network.nvidia.com/pdf/whitepapers/IBSCKeyMeasures.pdf
  57. https://wiki.archlinux.org/title/InfiniBand
  58. https://www.infinibandta.org/wp-content/uploads/2024/09/IBTA-Overview-of-IBTA-Volume-1-Release-1.8.pdf
  59. https://docs.kernel.org/infiniband/user_verbs.html
  60. https://www.rdmamojo.com/2012/05/18/libibverbs/
  61. https://docs.nvidia.com/doca/sdk/DOCA%2BRelease%2BNotes/index.html
  62. https://docs.nvidia.com/networking/display/MLNXOFEDv581121LTS/Introduction
  63. https://docs.nvidia.com/networking/display/OFEDv502180/IP%2Bover%2BInfiniBand%2B%28IPoIB%29
  64. https://network.nvidia.com/products/adapter-software/ethernet/windows/winof-2
  65. https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/enable-infiniband
  66. https://docs.nvidia.com/networking/display/MLNXOFEDv461000/OpenSM
  67. https://docs.nvidia.com/networking/display/OFEDv502180/InfiniBand%2BFabric%2BUtilities
  68. https://docs.nvidia.com/networking/display/NVIDIAInfiniBandSecurityOverviewandGuidelines/Security%2Bin%2BInfiniBand
  69. https://apps.dtic.mil/sti/trecms/pdf/AD1166855.pdf
  70. https://docs.nvidia.com/networking/display/MLNXOFEDv23070512/RDMA%2Bover%2BConverged%2BEthernet%2B%28RoCE%29
  71. https://www.infinibandta.org/wp-content/uploads/2019/05/IBTA_WhitePaper_May-20-2019.pdf
  72. https://docs.nvidia.com/doca/sdk/RDMA%2Bover%2BConverged%2BEthernet/index.html
  73. https://cloudswit.ch/blogs/roce-or-infiniband-technical-comparison/
  74. https://www.openfabrics.org/images/2018workshop/presentations/114_ESmirnov_EthernetoverInfiniband.pdf
  75. https://www.rfwireless-world.com/terminology/ethernet-over-infiniband
  76. https://docs.nvidia.com/networking/display/mlnxofedv23103211lts/Single%2BRoot%2BIO%2BVirtualization%2B%28SR-IOV%29
  77. https://nvmexpress.org/nvme-over-fabrics-part-two/
  78. https://docs.nvidia.com/networking/display/MLNXOFEDv461000/NVME-oF%2B-%2BNVM%2BExpress%2Bover%2BFabrics
  79. https://www.infinibandta.org/wp-content/uploads/2024/07/Demartek_IBTA_RoCE_Deployment_Guide_2018-07.pdf
  80. https://top500.org/lists/top500/2025/11/
  81. https://www.sdxcentral.com/news/supercomputer-champ-repeats-infiniband-tops-ethernet-and-linux-dominates/
  82. https://top500.org/news/el-capitan-retains-1-as-jupiter-becomes-europes-first-exascale-system-in-the-66th-top500-list/
  83. https://nowlab.cse.ohio-state.edu/projects/16/
  84. https://www.researchgate.net/publication/224017952_Petascale_Atmospheric_Models_for_the_Community_Climate_System_Model_New_developments_and_evaluation_of_scalable_dynamic_cores
  85. https://kbolsen.sdsu.edu/PUBL_dir/SC13-SCEC-FINAL.pdf
  86. https://www.nvidia.com/en-us/data-center/dgx-superpod/
  87. https://www.alcf.anl.gov/files/Zamora_Parallel_IO_Storage_Oct4_2018.pdf
  88. https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/ndh100v5-series
  89. https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nd-family
  90. https://www.nvidia.com/en-us/networking/products/data-processing-unit/
  91. https://www.mordorintelligence.com/industry-reports/infiniband-market
  92. https://vitextech.com/infiniband-vs-ethernet-for-ai-clusters-2025/
  93. https://www.juniper.net/content/dam/www/assets/white-papers/us/en/2024/juniper-artificial-intelligence-data-center-comparison-of-infiniband-and-rdma-over-converged-ethernet.pdf
  94. https://www.infinibandta.org/about-the-ibta/
  95. https://www.naddod.com/blog/what-is-infiniband
  96. https://www.nextplatform.com/2021/09/30/infiniband-innovation-is-about-more-than-bandwidth-and-latency/
  97. https://www.infinibandta.org/new-infiniband-and-roce-specification-introduces-memory-placement-extensions-to-significantly-reduce-persistent-memory-latency/
  98. https://www.hpe.com/psnow/doc/a50009229enw
  99. https://www.infinibandta.org/ibta-expands-interoperability-with-new-infiniband-architecture-specification-release-extends-virtualization-support/
  100. https://www.infinibandta.org/methods-of-implementation/
  101. https://www.roceinitiative.org/ibta-plugfest-33-sees-first-infiniband-hdr-200gbs-compliance-testing/
  102. https://docs.nvidia.com/networking/display/ConnectX7VPI/Specifications
Add your contribution
Related Hubs
User Avatar
No comments yet.