Hubbry Logo
Ethernet flow controlEthernet flow controlMain
Open search
Ethernet flow control
Community hub
Ethernet flow control
logo
8 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Ethernet flow control
Ethernet flow control
Ethernet flow control
View on Wikipedia
from Wikipedia
Wireshark screenshot of an Ethernet pause frame

Ethernet flow control is a mechanism for temporarily stopping the transmission of data on Ethernet family computer networks. The goal of this mechanism is to avoid packet loss in the presence of network congestion.

The first flow control mechanism, the pause frame, was defined by the IEEE 802.3x standard. The follow-on priority-based flow control, as defined in the IEEE 802.1Qbb standard, provides a link-level flow control mechanism that can be controlled independently for each class of service (CoS), as defined by IEEE P802.1p and is applicable to data center bridging (DCB) networks, and to allow for prioritization of voice over IP (VoIP), video over IP, and database synchronization traffic over default data traffic and bulk file transfers.

Description

[edit]

A sending station (computer or network switch) may be transmitting data faster than the other end of the link can accept it. Using flow control, the receiving station can signal the sender requesting suspension of transmissions until the receiver catches up. Flow control on Ethernet can be implemented at the data link layer.

The first flow control mechanism, the pause frame, was defined by the Institute of Electrical and Electronics Engineers (IEEE) task force that defined full duplex Ethernet link segments. The IEEE standard 802.3x was issued in 1997.[1]

Pause frame

[edit]

An overwhelmed network node can send a pause frame, which halts the transmission of the sender for a specified period of time. A media access control (MAC) frame (EtherType 0x8808) is used to carry the pause command, with the Control opcode set to 0x0001 (hexadecimal).[1] Only stations configured for full-duplex operation may send pause frames. When a station wishes to pause the other end of a link, it sends a pause frame to either the unique 48-bit destination address of this link or to the 48-bit reserved multicast address of 01-80-C2-00-00-01.[2]: Annex 31B.3.3  The use of a well-known address makes it unnecessary for a station to discover and store the address of the station at the other end of the link.

Another advantage of using this multicast address arises from the use of flow control between network switches. The particular multicast address used is selected from a range of address which have been reserved by the IEEE 802.1D standard which specifies the operation of switches used for bridging. Normally, a frame with a multicast destination sent to a switch will be forwarded out to all other ports of the switch. However, this range of multicast address is special and will not be forwarded by an 802.1D-compliant switch. Instead, frames sent to this range are understood to be frames meant to be acted upon only within the switch.

A pause frame includes the period of pause time being requested, in the form of a two-byte (16-bit), unsigned integer (0 through 65535). This number is the requested duration of the pause. The pause time is measured in units of pause quanta, where each quanta is equal to 512 bit times.

By 1999, several vendors supported receiving pause frames, but fewer implemented sending them.[3][4]

Issues

[edit]

One original motivation for the pause frame was to handle network interface controllers (NICs) that did not have enough buffering to handle full-speed reception. This problem is not as common with advances in bus speeds and memory sizes. A more likely scenario is network congestion within a switch. For example, a flow can come into a switch on a higher speed link than the one it goes out, or several flows can come in over two or more links that total more than an output link's bandwidth. These will eventually exhaust any amount of buffering in the switch. However, blocking the sending link will cause all flows over that link to be delayed, even those that are not causing any congestion. This situation is a case of head-of-line (HOL) blocking, and can happen more often in core network switches due to the large numbers of flows generally being aggregated. Many switches use a technique called virtual output queues to eliminate the HOL blocking internally, so will never send pause frames.[4]

Subsequent efforts

[edit]

Congestion management

[edit]

Another effort began in March 2004, and in May 2004 it became the IEEE P802.3ar Congestion Management Task Force. In May 2006, the objectives of the task force were revised to specify a mechanism to limit the transmitted data rate at about 1% granularity. The request was withdrawn and the task force was disbanded in 2008.[5]

Priority flow control

[edit]

Ethernet flow control disturbs the Ethernet class of service (defined in IEEE 802.1p), as the data of all priorities are stopped to clear the existing buffers which might also consist of low-priority data. As a remedy to this problem, Cisco Systems defined their own priority flow control extension to the standard protocol. This mechanism uses 14 bytes of the 42-byte padding in a regular pause frame. The MAC control opcode for a Priority pause frame is 0x0101. Unlike the original pause, Priority pause indicates the pause time in quanta for each of eight priority classes separately.[6] The extension was subsequently standardized by the Priority-based Flow Control (PFC) project authorized on March 27, 2008, as IEEE 802.1Qbb.[7] Draft 2.3 was proposed on June 7, 2010. Claudio DeSanti of Cisco was editor.[8] The effort was part of the data center bridging task group, which developed Fibre Channel over Ethernet.[9]

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Ethernet flow control

View on Grokipedia
from Grokipedia
Ethernet flow control is a link-level mechanism defined in the IEEE 802.3x-1997 standard that enables full-duplex Ethernet devices to temporarily pause transmission to avoid buffer overflows and frame loss during local congestion on a . This protocol operates within the Media Access Control (MAC) sublayer of the Ethernet frame structure, providing a simple yet effective way for receivers to signal senders to halt traffic, thereby supporting reliable delivery in environments with varying traffic loads without relying on higher-layer protocols for error recovery. Introduced as a supplement to the IEEE 802.3 standard, it addresses the needs of switches and other devices with finite buffer capacities, allowing them to manage ingress traffic dynamically while maintaining full-duplex performance across speeds like 10 Mb/s, 100 Mb/s, and beyond. The core mechanism of IEEE 802.3x flow control relies on PAUSE frames, which are special MAC control frames inserted between the MAC client and the physical layer. These frames use a reserved multicast destination address (01-80-C2-00-00-01), an EtherType of 0x8808 for MAC control, and an opcode of 0x0001 to indicate a pause request, followed by a pause time parameter measured in slot times (each equivalent to 512 bit times at the link speed). Upon receiving a PAUSE frame, the sender ceases transmitting data frames for the specified duration or until a subsequent frame with zero pause time (effectively resuming transmission) is received; this process can be repeated to extend pauses as needed. Flow control activation is typically negotiated automatically during link establishment on twisted-pair media via extensions to the auto-negotiation protocol in IEEE 802.3 Clause 28, though manual configuration may be required for fiber optic links. While IEEE 802.3x provides symmetric or asymmetric flow control suitable for general-purpose Ethernet networks, it applies a global pause to all traffic on the link, which can introduce latency for non-congested flows. To mitigate this limitation, particularly in data center environments requiring lossless operation for specific traffic classes, the priority-based flow control (PFC) extension was standardized in IEEE 802.1Qbb-2011 as part of the Data Center Bridging (DCB) enhancements to IEEE 802.1Q. PFC builds on the PAUSE frame format by incorporating 802.1p priority values (from 0 to 7), allowing independent pausing of up to eight virtual lanes or traffic classes without affecting others, thus enabling zero-loss transport for protocols like Fibre Channel over Ethernet (FCoE) while permitting best-effort traffic to continue. This advancement has become crucial in modern high-performance computing and storage networks, where mixed workloads demand both reliability and efficiency. As of 2025, further enhancements like credit-based flow control in the Ultra Ethernet Consortium's Specification 1.0 support lossless operation in large-scale AI networks.

Fundamentals

Definition and Purpose

Ethernet flow control is a mechanism standardized in IEEE 802.3x that regulates the rate of data frame transmission between directly connected full-duplex Ethernet devices to prevent buffer overflows at the receiver. In Ethernet networks, particularly switched local area networks (LANs), occurs when the ingress traffic rate to a device's receive buffer exceeds its processing or forwarding capacity, potentially leading to frame discards if unchecked. This mechanism enables receivers to signal senders to pause transmission temporarily when buffers near capacity, ensuring frames are not lost at the link layer. The primary purpose of Ethernet flow control is to provide reliable delivery of Ethernet frames in full-duplex links without depending on retransmission protocols at higher layers, such as TCP, thereby maintaining end-to-end data integrity in environments with constrained buffer sizes. By implementing backpressure through pause signals, it avoids the performance degradation from frame drops, which would otherwise trigger error recovery mechanisms and increase latency. Key benefits include enhanced network efficiency, as it reduces the overhead of higher-layer retransmissions, and supports the deployment of cost-effective switches with limited memory while preserving lossless operation on point-to-point links. Ethernet flow control predominantly relies on reactive methods, where the receiver detects congestion and responds by instructing the sender to halt transmission until buffers recover. Proactive approaches, which forecast and mitigate congestion in advance using techniques like credit allocation, contrast with this but are not inherent to the core IEEE 802.3x framework. This reactive nature is realized primarily through PAUSE frames sent between peers.

Historical Context

Early Ethernet , standardized under in 1983, operated in half-duplex mode using the with (CSMA/CD) protocol, which managed access to the shared medium through collision detection and backoff mechanisms but did not incorporate explicit flow control to regulate transmission rates between sender and receiver. This sufficed for the 10 Mbps speeds of the , where collisions served as a natural limiter, but as Ethernet evolved toward higher speeds and full-duplex operation in the 1990s—eliminating the shared medium and CSMA/CD—the absence of built-in congestion became problematic, particularly with the advent of switched that could lead to buffer overflows. The need for flow control intensified with the development of , where link speeds reached 1 Gbps, outpacing the buffer capacities of contemporary switches and risking frame loss during bursts. In response, the initiated proposals in to introduce a frame-based mechanism for full-duplex links, aiming to enable cost-effective switches with limited memory without resorting to packet drops. These efforts culminated in the approval of as an amendment by the IEEE Standards Board on , , with following on , , which was incorporated into the IEEE 802.3-1998 revision, standardizing PAUSE frames as the foundational tool for pausing transmission on congested links. Following IEEE 802.3x, advancements addressed limitations in diverse environments, notably through the IEEE 802.1Qbb ratified in 2011, which introduced Priority-based Flow Control (PFC) as part of the Data Center Bridging (DCB) enhancements to support per-priority pausing for lossless operation in converged . Adoption progressed steadily: by the early , IEEE 802.3x flow control had become widespread in enterprise switches alongside Gigabit Ethernet deployment, enabling reliable performance in office and campus environments. In the , focus shifted to data centers, where DCB and PFC gained traction to accommodate storage traffic like Fibre Channel over Ethernet, ensuring zero-loss fabrics amid 10 Gbps and higher speeds.

Core Mechanism

IEEE 802.3x PAUSE Frames

IEEE 802.3x specifies the MAC Control sublayer for full-duplex operation in networks, providing a mechanism to implement flow control through PAUSE frames. This standard, incorporated into IEEE Std 802.3 since 1998, applies to full-duplex at speeds ranging from 10 Mbps to 1 Gbps and beyond as evolved. Support for 802.3x flow control is optional for devices operating in full-duplex mode, while it is not applicable to half-duplex , which rely on CSMA/CD for congestion . The PAUSE frame is a special 64-byte MAC control frame designed to request the temporary suspension of frame transmission from the receiving device. It uses the value 0x8808 to indicate a MAC control frame and the 0x0001 specifically for the PAUSE operation. The destination is fixed as the 01-80-C2-00-00-01, ensuring it is recognized by all compliant full-duplex Ethernet devices on the link. The source is the MAC of the sending device, and the frame includes a 2-byte pause time parameter that specifies the duration of the pause in units of 512 bit times (quanta), ranging from 0 to 65535. A pause time of 0x0000 signals the resumption of transmission, while non-zero values instruct the receiver to halt sending frames for the indicated period. The structure of the PAUSE frame adheres to the minimum size of 64 bytes, including if necessary. The pause quanta duration in seconds is calculated as (pause_time × 512) / link_speed, where link_speed is in bits per second; for example, at 1 Gbps, one quanta equates to approximately 0.512 microseconds.
FieldSize (bytes)Description
7Synchronization (0xAAAAAAAAAAAAA, not transmitted on all media).
Start Frame Delimiter (SFD)1Marks the end of (0xAB).
Destination Address6Multicast: 01-80-C2-00-00-01.
Source Address6MAC address of the sender.
Length/Type20x8808 (MAC control).
Opcode20x0001 (PAUSE).
Parameters4Pause time (2 bytes: 0-65535 quanta); reserved (2 bytes: 0x0000).
Pad0-44Zeros to reach minimum frame size.
Frame Check Sequence (FCS)4CRC-32 checksum.
This format ensures compatibility across full-duplex Ethernet implementations, with the total frame length fixed at 64 bytes to align with the slot time for collision detection in legacy contexts, though collisions do not occur in full-duplex mode.

PAUSE Frame Operation

When a receiving device detects that its input buffer is approaching congestion, typically upon reaching a predefined high threshold such as 80% of its capacity, it generates and transmits a PAUSE frame to the sending device over the full-duplex Ethernet link. This threshold is configurable and serves to prevent buffer overflow and frame loss by proactively signaling the need to halt incoming traffic. Upon receiving the PAUSE frame, the sending device immediately stops transmitting Ethernet frames for the duration specified in the frame's pause quanta field, which represents a timer value in slot times (each slot time being the transmission time of 512 bits at the link speed). Transmission resumes automatically once this timer expires, or earlier if a subsequent PAUSE frame with a quanta value of zero is received, indicating that the receiver's buffer has sufficiently cleared below a low threshold. If congestion persists, the receiver may send additional PAUSE frames to extend the pause period; the latest frame received always overrides any previous pause timer on the sender, allowing dynamic adjustment of the halt duration. This mechanism supports sustained control by enabling repeated or prolonged pauses without requiring frame drops. PAUSE frame operation is strictly link-level and applies only to the point-to-point full-duplex connection between two directly attached devices, such as a switch port and an upstream host or another switch; it does not propagate across multi-hop Ethernet paths or through switching fabrics. In a typical scenario within an Ethernet switch, if incoming traffic to a port exceeds the output buffer's processing capacity—such as during a burst from an upstream device—the port detects the buffer nearing its high threshold and sends a PAUSE frame upstream to throttle the source, preventing downstream congestion while the switch clears the backlog. Once the buffer level drops below the resume threshold, a zero-quanta PAUSE frame is sent to unpause the link, restoring normal traffic flow.

Limitations

Head-of-Line Blocking

Head-of-line (HOL) blocking is a significant limitation of Ethernet flow control using IEEE 802.3x PAUSE frames, occurring in multi-queue ports where the mechanism halts all traffic on the link, thereby preventing high-priority queued behind low-priority from being transmitted. This arises because PAUSE operates at the link level without distinguishing between traffic classes, causing unrelated packets to accumulate and delay forwarding. The root cause lies in the global scope of 802.3x PAUSE, which signals congestion for the entire physical link rather than targeting specific priorities or flows, leading to indiscriminate pausing that affects all queues. For example, in converged networks combining voice over IP (VoIP) with bulk data transfers, a PAUSE frame issued due to buffer overflow from large file downloads can stall low-latency VoIP packets, resulting in jitter and degraded call quality. Basic 802.3x provides no mitigation through VLAN tagging or priority awareness, allowing such cross-traffic interference to persist unchecked. This blocking was prominently observed in early deployments of , where increasing diversity highlighted the need for more granular control mechanisms. In mixed-traffic scenarios, HOL can amplify latency by factors of 5 times, as round-trip times rise from baseline levels like 240 µs to over 1 ms due to prolonged pauses and buffer buildup.

Latency and Throughput Issues

The use of IEEE 802.3x PAUSE frames introduces variable delays into Ethernet networks, as the receiving device halts transmission for a specified quanta duration (up to 65,535 slot times, equivalent to approximately 33.6 ms on a 1 Gbps link) when buffers approach overflow. These pauses create unpredictable latency spikes, rendering PAUSE-based flow control unsuitable for real-time applications such as networked storage (e.g., iSCSI) or voice traffic, where consistent low delays are critical to maintain performance. In congested scenarios, empirical measurements on Gigabit Ethernet setups have shown baseline latencies increasing from 3 µs to 12 µs under background traffic loads, with flow control exacerbating these effects through enforced idle periods. The oscillatory of PAUSE operation—alternating between pause and resume signals—further degrades network , particularly in bursty traffic patterns common to data centers and storage systems. During bursts, frequent pauses lead to underutilization of link bandwidth, with simulations indicating losses of 20-50% as devices repeatedly stop and restart transmission, wasting cycles on control and times. Studies from the early on links under congestion reported throughput drops of up to 30%, as PAUSE mechanisms struggle to sustain full utilization amid rapid traffic fluctuations, dropping from near-line-rate (e.g., 1 Gbps) to as low as 700 Mbps in oversubscribed topologies. This reduction stems from the coarse-grained link-level pausing, which halts all traffic indiscriminately, amplifying oscillations and preventing smooth recovery to peak rates. Unpredictable pause durations also introduce significant jitter, compounding timing sensitivities in time-aware protocols like Audio Video Bridging (AVB) and Time-Sensitive Networking (TSN), where bounded end-to-end delays are essential for synchronized delivery. Measurements in Gigabit Ethernet environments reveal jitter widening from a standard deviation of 1.1 µs in unloaded conditions to several microseconds under moderate congestion with PAUSE active, disrupting periodic streams and increasing variance in packet arrival times. While PAUSE achieves link-level lossless operation by preventing immediate buffer overflows, it fails to guarantee end-to-end losslessness in multi-hop networks, as upstream congestion can propagate pauses without addressing downstream bottlenecks, potentially leading to indirect frame drops elsewhere. Compared to no flow control (which results in lossy behavior and retransmissions under congestion), PAUSE improves reliability by avoiding packet discards at the link layer, yet it underperforms finer-grained alternatives like explicit congestion notification, which maintain higher utilization (e.g., over 99% in simulations) without widespread pausing. Head-of-line blocking contributes to these issues by stalling non-congested queues during pauses.

Advanced Techniques

Priority-Based Flow Control

Priority-based flow control (PFC), defined in the IEEE 802.1Qbb-2011 standard, extends the basic Ethernet PAUSE mechanism to provide independent flow control for each of up to eight traffic priorities, as identified by the 802.1p class of service field in VLAN-tagged frames. Approved on June 16, 2011, this standard enables receivers to pause transmission selectively for specific priorities on full-duplex links, thereby mitigating head-of-line blocking while preserving overall link utilization for non-paused traffic classes. By operating at the link level within point-to-point connections, PFC ensures that congestion in one priority does not indiscriminately halt all frames, allowing concurrent support for latency-sensitive and best-effort traffic. The core mechanism relies on specialized PFC frames, which share the same destination MAC address (01-80-C2-00-00-01) and opcode (0x0001) as standard 802.3x PAUSE frames but include additional fields for per-priority control. Following the opcode, the frame contains a 2-octet priority enable vector, where the least significant 8 bits (e to e) form a bitmask indicating which priorities are affected—set to 1 for enabled priorities and 0 otherwise. This is followed by an 8-entry time vector, with each 2-octet entry specifying the pause duration (in quanta of 512 bit times) for the corresponding priority (0 to 65,535 quanta maximum). Upon receiving a PFC frame, the MAC client pauses the specified priority queues for the indicated time, using MAC control primitives (M_CONTROL.request and M_CONTROL.indication) to enforce the pause without discarding frames. Pause timers decrement based on the link transmission rate, resuming transmission once they reach zero, with a maximum delay constraint of 614.4 nanoseconds at 10 Gb/s to minimize latency impact. A primary benefit of PFC is enabling lossless Ethernet operation for protocols requiring zero packet loss, such as (FCoE), by dedicating specific priorities (e.g., priority 3 for storage) to pause only when necessary, while lower-priority best-effort traffic like IP flows continues unimpeded. This selective pausing reduces buffer requirements and supports converged network infrastructures, lowering operational costs in environments mixing storage, clustering, and general traffic. For instance, in storage networks, PFC ensures reliable delivery without retransmissions, enhancing for applications sensitive to even minor frame drops. Implementation requires devices to support PFC on at least one priority per port, with full compliance typically involving all eight 802.1p priorities to avoid partial blocking; negotiation of supported priorities occurs via the Bridging Capability eXchange (DCBX) protocol. Ports must process incoming PFC frames and generate them reactively based on buffer thresholds, ensuring symmetric behavior between sender and receiver. Since its ratification around 2010, PFC has become a cornerstone of Ethernet fabrics, particularly for and storage convergence, and is mandatory for reliable operation of version 2 (RoCEv2) to maintain lossless paths across the network. In 2025, Amendment 38 introduced automated PFC headroom calculation and support for Media Access Control Security (MACsec) with PFC to enhance and buffer management in high-speed networks. Its adoption has facilitated the deployment of unified fabrics in enterprise and environments, where it underpins protocols demanding sub-millisecond latency and zero loss.

Data Center Bridging Integration

Data Center Bridging (DCB) is a suite of enhancements to the standards designed to adapt Ethernet for environments, enabling converged networking that supports lossless and low-latency for diverse applications such as storage, clustering, and IP-based communications over a single . Key components include IEEE 802.1Qbb for Priority-based Flow Control (PFC), which provides per-priority lossless operation; IEEE 802.1Qaz for Enhanced Transmission Selection (ETS), which manages bandwidth allocation; and IEEE 802.1Qau for Quantized Congestion Notification (QCN), which addresses end-to-end congestion. These features collectively transform standard Ethernet into a unified fabric capable of handling protocols like Fibre Channel over Ethernet (FCoE) alongside traditional IP without requiring separate networks. The Data Center Bridging Capability Exchange (DCBX) protocol facilitates integration by of DCB parameters between directly connected devices, building on the (LLDP) defined in IEEE 802.1AB. DCBX advertises and configures capabilities for PFC, ETS, and congestion , ensuring consistent policies across peers to prevent mismatches that could lead to or suboptimal . For instance, it exchanges type-length-value (TLV) to enable PFC on specific priorities and allocate bandwidth via ETS, allowing devices to dynamically agree on flow control and without manual intervention. Within DCB, ETS (IEEE 802.1Qaz) complements PFC by organizing traffic into priority groups (PGs) and assigning bandwidth limits or guarantees , ensuring fair sharing while unused bandwidth from paused or low-utilization groups can be borrowed by others. This integration allows PFC to pause specific priorities without globally halting the link, maintaining throughput for non-paused traffic classes. A primary use case for DCB integration is the convergence of storage traffic with Ethernet networks, such as iSCSI and FCoE, where PFC ensures zero packet loss for storage protocols that cannot tolerate retransmissions, while coexisting with loss-tolerant IP traffic. In these scenarios, DCBX negotiates PFC enablement on the storage priority (e.g., VLAN 0 for FCoE), guaranteeing lossless delivery alongside ETS-managed bandwidth for LAN traffic. DCB has evolved from its initial specification in version 1.01 around 2010, which focused on core enhancements like PFC and ETS, to later amendments incorporating Edge Virtual Bridging (EVB) concepts. A notable advancement is IEEE 802.1BR, which standardizes Bridge Port Extension for virtualized environments, allowing edge devices to extend bridging functions and integrate DCB capabilities more seamlessly in multi-tenant data centers.

Congestion Management

Explicit Congestion Notification

(ECN) serves as a proactive congestion avoidance mechanism in Ethernet , enabling switches to signal impending overload to endpoints by marking packets rather than dropping them or issuing pauses. Defined initially for IP in RFC 3168, ECN integrated into Ethernet environments, particularly centers, where switches monitor buffer queues and markings when thresholds are met. This leverages the two-bit ECN field in the (bits 6 and 7 of the DiffServ code point). The operation involves senders marking outgoing packets as ECN-Capable Transport (ECT) using codepoints 01 or 10 in the ECN field to indicate support. Upon detecting congestion—typically when a queue exceeds a configured threshold—switches probabilistically set the Congestion Experienced (CE) codepoint (11), with marking probability increasing based on queue depth via a Random Early Detection (RED)-like algorithm. Receivers detect the CE mark and feedback the congestion signal to senders, such as through TCP ACKs with the ECN-Echo (ECE) flag, prompting rate reduction by halving the congestion window; senders then acknowledge this with the Congestion Window Reduced (CWR) flag to prevent repeated feedback. This end-to-end feedback loop ensures timely adjustment without packet loss. Compared to IEEE 802.3x PAUSE frames, ECN offers key advantages: it is proactive, intervening before severe congestion causes drops or pauses, avoids halting all transmissions on a link, and provides per-flow granularity ideal for TCP and RDMA over Converged Ethernet (RoCE) protocols. In practice, ECN enhances throughput and latency for bursty traffic by enabling smoother rate adaptations. ECN integrates with Priority-based Flow Control (PFC) in hybrid setups under Bridging (DCB), assigning ECN to lossy traffic classes for best-effort handling while reserving PFC for lossless priorities like storage traffic. However, its effectiveness depends on end-to-end support across all devices in the path, as non-supporting nodes may strip or ignore marks, rendering it less suitable for heterogeneous networks unlike the link-local PAUSE mechanism.

Quantized Congestion Notification

Quantized Congestion Notification (QCN) is a rate-based congestion control protocol defined in the IEEE 802.1Qau standard for managing long-lived data flows in Ethernet networks, particularly suited for data centers with limited bandwidth-delay products. It operates by enabling switches to signal congestion levels to sending devices, allowing proactive rate adjustments to prevent buffer overflows and frame loss. QCN is integrated into Data Center Bridging (DCB) frameworks to support low-latency applications alongside traditional Ethernet traffic. An extension known as Data Center QCN (DCQCN) adapts QCN for RDMA over Converged Ethernet (RoCE) in modern high-performance computing environments as of 2025. The protocol distinguishes between two primary components: the reaction point (RP) at the source device, which limits transmission rates, and the congestion point (CP) at the switch or bridge, which detects and reports congestion. Feedback messages are generated at the CP and transmitted back to the RP, carrying quantized congestion information to guide rate changes without requiring end-to-end acknowledgments. These messages use a 6-bit field to encode speed-up or slow-down factors, providing a compact representation of network state that balances precision and overhead. At the CP, congestion is assessed by monitoring queue relative to a target equilibrium level. The feedback value is derived as Q = \frac{\text{queue_length} - \text{target}}{\text{scale}}, which captures the deviation from the desired and is further refined to include queue change over time for . This value is quantized into the 6-bit feedback field and inserted into sampled or dedicated messages sent upstream to the RP. Sampling occurs probabilistically based on the congestion severity to minimize signaling overhead while ensuring timely notifications. Upon receiving a feedback message at the RP, the transmission rate is updated multiplicatively to alleviate congestion. The adjustment follows the formula ratenew=rateold×(1−α×feedback),\text{rate}_{\text{new}} = \text{rate}_{\text{old}} \times (1 - \alpha \times \text{feedback}), where α\alpha is computed as an exponential moving average of prior feedback values to dampen oscillations and stabilize convergence. This mechanism allows rapid rate decreases during congestion while incorporating gradual increases through periodic probing to probe available bandwidth. QCN demonstrates strong in incast scenarios, where multiple senders converge on a single receiver, by coordinated that avoids deep buffer usage. Compared to (ECN), it achieves lower latencies through rate feedback, reducing the risk of secondary bottlenecks in multi-hop paths. These benefits stem from its hardware-friendly , supporting in switches with shallow buffers. Deployment of QCN has been used in converged networks requiring low-latency operation, such as those supporting RoCE for storage and compute traffic, and in early cloud fabrics to enhance reliability in DCB-enabled Ethernet domains, though adoption has evolved with advancements in higher-layer protocols.

Modern Applications

Role in Data Centers

In modern data centers, Ethernet flow control has become integral to the evolution from siloed local area networks (LANs) for compute and management traffic and storage area networks (SANs) for block storage to unified Ethernet fabrics. This shift, driven by the need for cost efficiency and simplified infrastructure, leverages Data Center Bridging (DCB) standards, particularly Priority-based Flow Control (PFC), to converge storage, compute, and management workloads over a single Ethernet layer without performance degradation. By enabling lossless Ethernet, DCB eliminates the need for dedicated Fibre Channel fabrics, reducing cabling complexity and operational overhead while supporting higher bandwidth demands in scale-out environments. Key applications of Ethernet flow control in data centers center on high-performance, low-latency protocols. RDMA over Converged Ethernet (RoCE) depends on PFC to deliver lossless transport, preventing packet drops that could disrupt remote direct memory access operations critical for distributed computing and storage. For broader cloud-scale congestion management, Explicit Congestion Notification (ECN) and Quantized Congestion Notification (QCN) provide end-to-end feedback mechanisms, allowing endpoints to dynamically adjust transmission rates and avoid widespread bottlenecks in multi-tenant fabrics. These techniques ensure reliable performance for east-west traffic patterns, where data flows predominantly between servers within the data center rather than to external networks. Practical implementations often feature top-of-rack (ToR) switches with 40 Gbps and 100 Gbps ports configured for lossless operation, supporting protocols like for IP-based storage and (FCoE) for legacy SAN integration. These switches PFC on specific priority queues to isolate storage from best-effort compute flows, maintaining zero-loss guarantees across the rack while scaling to higher speeds. In rack-scale deployments, such configurations enable converged networking without retransmission overhead, optimizing utilization in dense server environments. Hyperscale operators have integrated hybrid flow control strategies combining traditional PAUSE frames with PFC to manage east-west traffic in their vast clusters. This approach balances lossless requirements for RDMA-based workloads with efficient handling of bursty application flows. Such adaptations address the unique challenges of hyperscale fabrics, where synchronized traffic bursts can overwhelm links. A prominent challenge addressed by these mechanisms is incast collapse in rack-scale systems, where multiple upstream servers simultaneously transmit to a single downstream receiver, causing switch buffer overflows and severe throughput degradation. QCN mitigates this by providing quantized feedback from congested switches, enabling rapid rate limiting at senders to stabilize flows without relying solely on loss-based recovery. This proactive control is particularly effective in storage-heavy scenarios, reducing tail latencies and improving overall cluster efficiency. By 2020, over 70% of Ethernet supported DCB features like PFC and ECN, reflecting widespread adoption driven by the growth in converged infrastructures and high-speed shipments exceeding 50 million units annually. This penetration underscores Ethernet flow control's role in enabling scalable, reliable fabrics for contemporary operations.

Developments for AI Networking

The Ultra Ethernet (UEC), formed in 2023 by founding members including , Arista, , , HPE, , Meta, and , aims to enhance Ethernet's capabilities for high-performance AI and high-performance computing (HPC) workloads through an open, interoperable . This includes improvements to flow control at the , such as credit-based extensions to traditional mechanisms like Priority Flow Control (PFC), to address the stringent requirements of AI clusters, including ultra-low latency and lossless transmission. A key development from UEC is Credit-Based Flow Control (CBFC), proposed as an optional to the Ethernet in 2024 specifications, which uses virtual credits tracked via cyclic counters at sender and receiver ends to enable precise and buffer . Unlike reactive pause-based approaches, CBFC operates proactively by confirming available buffer before transmission, reducing and improving in large-scale AI where collective operations demand synchronized, high-bandwidth communication. This mechanism builds on PFC foundations while offering backward compatibility and enhanced efficiency for scale-out AI fabrics. Integration of Link-Layer Retry (LLR) with flow control protocols like PFC and CBFC further tailors Ethernet for AI environments, providing rapid, local recovery from without involving higher-layer protocols such as TCP. LLR retransmits corrupted or lost frames at the , minimizing tail latency in GPU clusters where even minor losses can all-reduce operations; it validated in high-speed Ethernet deployments Gbps, ensuring near-lossless critical for AI model training. This combination addresses the zero-loss tolerance of AI workloads beyond traditional Data Center Bridging (DCB) requirements, enabling reliable interconnects in massive accelerator pods. In 2025, Broadcom's 6 switch series, delivering 102.4 Tbps capacity on a single 3 nm chip, incorporates AI-optimized enhancements to congestion , including variants of (ECN) and Quantized Congestion Notification (QCN) tuned for low-latency collective communications in AI clusters. These switches support UEC features like CBFC and LLR, facilitating flat, non-oversubscribed topologies that reduce latency for scale-up AI networking while maintaining compatibility with standards-based Ethernet. Industry efforts culminated at the Open Compute Project (OCP) Global Summit 2025, where announcements advanced standards-based Ethernet mechanisms for AI scale-up, including the formation of the Ethernet for Scale-Up Networking (ESUN) collaboration to promote open innovations in switching and framing. These initiatives, building on QCN and PFC, focus on reducing tail latency in AI infrastructures through enhanced flow control, with demonstrations showing improvements in predictability for hyperscale training environments.

References

  1. https://standards.ieee.org/ieee/802.3x/1082/
  2. https://www.ieee802.org/3/z/public/presentations/nov1996/RS8023x.pdf
  3. https://www.intel.com/content/www/us/en/docs/programmable/683221/20-3-19-1-0/flow-control-52885.html
  4. https://documentation.extremenetworks.com/exos_32.5/GUID-10385782-9CA8-4D4F-B5CD-E193B00152E5.shtml
  5. https://www.smallnetbuilder.com/lanwan/lanwan-features/when-flow-control-is-not-a-good-thing/
  6. https://standards.ieee.org/ieee/802.1Qbb/4361/
  7. https://1.ieee802.org/dcb/802-1qbb/
  8. https://www.juniper.net/documentation/us/en/software/junos/cos-ex/topics/concept/cos-priority-flow-control.html
  9. https://ultraethernet.org/ultra-ethernet-consortium-uec-launches-specification-1-0-transforming-ethernet-for-ai-and-hpc-at-scale/
  10. https://cdrdv2-public.intel.com/635330/635330_800%20Series%20Linux%20Flow%20Control%20Guide%20for%20RDMA_Rev1.1.pdf
  11. https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/concept/cos-qfx-series-congestion-notification-understanding.html
  12. https://www.cse.wustl.edu/~jain/papers/ftp/icc08.pdf
  13. https://www.iol.unh.edu/sites/default/files/knowledgebase/ethernet/ethernet_evolution.pdf
  14. https://www.oreilly.com/library/view/ethernet-the-definitive/1565926609/ch04.html
  15. https://ine.com/blog/2008-07-08-802-3x-flow-control
  16. http://ethernetalliance.org/wp-content/uploads/2011/10/document_files_DCB_Whitepaper_v21.pdf
  17. https://standards.ieee.org/ieee/802.3/10422/
  18. https://www.cs.emory.edu/~cheung/Courses/558/Syllabus/00/CSMA/00-Others/ethernet3.htm
  19. https://grouper.ieee.org/groups/802/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdf
  20. https://ieeexplore.ieee.org/document/9768038
  21. https://documentation.extremenetworks.com/VOSS/SW/87/VOSSUserGuide/GUID-2A8D3BF7-1689-4B96-8744-5C92A5373AC8.shtml
  22. https://www.ieee802.org/1/files/public/docs2008/bb-pelissier-pfc-proposal-0508.pdf
  23. https://blog.ipspace.net/2010/09/introduction-to-8021qbb-priority-flow/
  24. https://community.cisco.com/t5/switching/qos-and-flowcontrol/td-p/967279
  25. https://users.cs.utah.edu/~brent/docs/tcp-bolt.infocom14.pdf
  26. https://www.ieee802.org/802_tutorials/04-November/802_3_cmsg_tutorial_1104.pdf
  27. https://cds.cern.ch/record/684025/files/daq-2000-032.pdf
  28. https://www.cse.wustl.edu/~jain/theses/ftp/jinjing.pdf
  29. 1Qbb specifies protocols for flow control per traffic class, aiming to eliminate frame loss due to congestion, similar to 802.3x PAUSE.
  30. https://standards.ieee.org/ieee/8802-1Q_2024_Amd_38/12263/
  31. https://grouper.ieee.org/groups/802/1/pages/dcbridges.html
  32. https://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbx-capability-exchange-discovery-protocol-1108-v1.01.pdf
  33. https://1.ieee802.org/dcb/802-1qaz/
  34. https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus6000/sw/fcoe/6x/b_6k_FCoE_Config_6x/b_6k_FCoE_Config_602N11_chapter_01.pdf
  35. https://www.juniper.net/documentation/us/en/software/junos/fcoe-ex/topics/concept/dcbx-ex-series.html
  36. https://datatracker.ietf.org/doc/html/rfc3168
  37. https://doc.dpdk.org/guides-24.03/prog_guide/traffic_management.html
  38. https://www.juniper.net/documentation/us/en/software/junos/cos/topics/topic-map/explicit-congestion-notification.html
  39. https://www.keysight.com/blogs/en/tech/traf-gen/2022/10/24/how-to-control-congestion-in-your-data-center
  40. https://www.juniper.net/documentation/us/en/software/junos/cos/topics/concept/cos-qfx-series-congestion-notification-understanding.html
  41. https://1.ieee802.org/dcb/802-1qau/
  42. https://people.csail.mit.edu/alizadeh/papers/qcn-allerton08.pdf
  43. https://www.openfabrics.org/images/eventpresos/workshops2013/2013_Workshop_Tues_1100_liss_qcn.pdf
  44. https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-dcqcn-04
  45. https://www.ieee802.org/1/files/public/docs2008/au-rong-qcn-serial-hai-pseudo-code%20rev2.0.pdf
  46. https://blogs.cisco.com/datacenter/nexus-9300-cloud-scale-switches-ready-for-unified-lan-and-san-architectures-migration
  47. https://www.usenix.org/sites/default/files/conference/protected-files/schwartz_lisa12_slides.pdf
  48. https://docs.nvidia.com/networking/display/MLNXOFEDv451010/Flow%2BControl
  49. https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p523.pdf
  50. https://www.cisco.com/c/en/us/products/collateral/switches/nexus-5624q-switch/datasheet-c78-733100.html
  51. https://lenovopress.lenovo.com/tips1273-lenovo-rackswitch-g8264cs
  52. https://ieeexplore.ieee.org/document/5935217
  53. https://researchwith.njit.edu/en/publications/on-mitigating-tcp-incast-in-data-center-networks
  54. https://omdia.tech.informa.com/om021092/data-center-ethernet-switch-tracker--2q21-analysis
  55. https://www.hpcwire.com/2025/09/09/ultra-ethernet-has-arrived-one-network-to-rule-them-all/
  56. https://ultraethernet.org/
  57. https://blogs.cisco.com/datacenter/ultra-ethernet-for-scalable-ai-network-deployment
  58. https://community.cadence.com/cadence_blogs_8/b/fv/posts/uec-cbfc-credit-based-flow-control-for-next-gen-ethernet-in-ai-and-hpc
  59. https://ultraethernet.org/ultra-ethernet-specification-update/
  60. https://www.synopsys.com/articles/ultra-ethernet-ualink-ai-networks.html
  61. https://community.cadence.com/cadence_blogs_8/b/fv/posts/uec-llr-the-future-of-loss-recovery-in-ethernet-for-ai-hpc
  62. https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm78910-series
  63. https://www.broadcom.com/company/news/product-releases/63146
  64. https://www.edge-core.com/blogs/broadcom-tomahawk-6-powering-the-latest-generation-of-ai-and-hyperscale-networks/
  65. https://www.opencompute.org/blog/introducing-esun-advancing-ethernet-for-scale-up-ai-infrastructure-at-ocp
  66. https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/
  67. https://www.broadcom.com/blog/2025-ocp-global-summit-broadcom-advances-industry-support-for-ethernet-for-ai-networking
Add your contribution
Related Hubs
User Avatar
No comments yet.