Hubbry Logo
Distributed shared memoryDistributed shared memoryMain
Open search
Distributed shared memory
Community hub
Distributed shared memory
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Distributed shared memory
Distributed shared memory
Distributed shared memory
View on Wikipedia
from Wikipedia
Distributed shared memory implementation.

In computer science, distributed shared memory (DSM) is a form of memory architecture where physically separated memories can be addressed as a single shared address space. The term "shared" does not mean that there is a single centralized memory, but that the address space is shared—i.e., the same physical address on two processors refers to the same location in memory.[1]: 201  Distributed global address space (DGAS), is a similar term for a wide class of software and hardware implementations, in which each node of a cluster has access to shared memory in addition to each node's private (i.e., not shared) memory.

Overview

[edit]

DSM can be achieved via software as well as hardware. Hardware examples include cache coherence circuits and network interface controllers. There are three ways of implementing DSM: page-based approach using virtual memory, shared-variable approach using routines to access shared variables and object-based approach, ideally accessing shared data through object-oriented discipline.

DSM scales well with a large number of nodes and its message passing is hidden. It can handle complex and large databases without replication or sending the data to processes and is generally cheaper than using a multiprocessor system. It provides large virtual memory space, programs are more portable due to common programming interfaces and shield programmers from sending or receiving primitives.

It is generally slower to access than non-distributed shared memory and must provide additional protection against simultaneous accesses to shared data. DSM may incur a performance penalty, there is little programmer control over actual messages being generated and consistency models are needed to write correct programs.

Comparison with message passing

[edit]
Message passing Distributed shared memory
Variables have to be marshalled Variables are shared directly
Cost of communication is obvious Cost of communication is invisible
Processes are protected by having private address space Processes could cause error by altering data
Processes should execute at the same time Executing the processes may happen with non-overlapping lifetimes

Software DSM systems also have the flexibility to organize the shared memory region in different ways. The page-based approach organizes shared memory into pages of fixed size. In contrast, the object based approach organizes the shared memory region as an abstract space for storing shareable objects of variable sizes. Another commonly seen implementation uses a tuple space, in which the unit of sharing is a tuple.

Shared memory architecture may involve separating memory into shared parts distributed amongst nodes and main memory, or distributing all memory between nodes. A coherence protocol, chosen in accordance with a consistency model, maintains memory coherence.

Directory memory coherence

[edit]
Main article: Directory-based cache coherence

Memory coherence is necessary such that the system which organizes the DSM is able to track and maintain the state of data blocks in nodes across the memories comprising the system. A directory is one such mechanism which maintains the state of cache blocks moving around the system.

States

[edit]
A state diagram of a block of memory in a DSM. A block is "owned" if one of the nodes has the block in state EM.

A basic DSM will track at least three states among nodes for any given block in the directory.[2] There will be some state to dictate the block as uncached (U), a state to dictate a block as exclusively owned or modified owned (EM), and a state to dictate a block as shared (S). As blocks come into the directory organization, they will transition from U to EM (ownership state) in the initial node. The state can transition to S when other nodes begin reading the block.

There are two primary methods for allowing the system to track where blocks are cached and in what condition across each node. Home-centric request-response uses the home to service requests and drive states, whereas requester-centric allows each node to drive and manage its own requests through the home.

Home-centric request and response

[edit]

In a home-centric system, the DSM will avoid having to handle request-response races between nodes by allowing only one transaction to occur at a time until the home node has decided that the transaction is finished—usually when the home has received every responding processor's response to the request. An example of this is Intel's QPI home-source mode.[3] The advantages of this approach are that it is simple to implement but its request-response strategy is slow and buffered due to the home node's limitations.

Requester-centric request and response

[edit]
Sequential invocations and responses in DSM.

In a requester-centric system, the DSM will allow nodes to talk at will to each other through the home. This means that multiple nodes can attempt to start a transaction, but this requires additional considerations to ensure coherence. For example: when one node is processing a block, if it receives a request for that block from another node it will send a NAck (Negative Acknowledgement) to tell the initiator that the processing node cannot fulfill that request right away. An example of this is Intel's QPI snoop-source mode.[3] This approach is fast but it does not naturally prevent race conditions and generates more bus traffic.

Consistency models

[edit]

The DSM must follow certain rules to maintain consistency over how read and write order is viewed among nodes, called the system's consistency model.

In a situation where there are n processes and Mi memory operations for each process i, and that all the operations are executed sequentially. (M1 + M2 + ... + Mn)!/(M1! M2!... Mn!) are possible interleavings of the operations. The issue with this conclusion is determining the correctness of the interleaved operations. Memory coherence for DSM defines which interleavings are permitted.

Replication

[edit]

There are two types of replication algorithms. Read replication and Write replication: in Read replication multiple nodes can read at the same time but only one node can write, while in Write replication multiple nodes can read and write at the same time. The write requests are handled by a sequencer. Replication of shared data in general tends to reduce network traffic, promote increased parallelism and result in fewer page faults. Preserving coherence and consistency may become more challenging.

Release and entry consistency

[edit]

Release consistency is when a process exits a critical section, new values of the variables are propagated to all sites. Entry consistency is when a process enters a critical section, it will automatically update the values of the shared variables. View-based consistency is a variant of entry consistency, except the shared variables of a critical section are automatically detected by the system.

Examples

[edit]

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Distributed shared memory

View on Grokipedia
from Grokipedia
Distributed (DSM) is a abstraction that implements a shared memory model across a distributed , simulating a single, logical address space over physically separate local memories in multiple networked nodes. This approach allows parallel programs to access shared data transparently, as if running on a traditional shared-memory multiprocessor, while leveraging the scalability and cost-effectiveness of distributed hardware. DSM emerged in the early 1980s as researchers sought to combine the programming simplicity of shared-memory systems with the and expandability of message-passing distributed architectures. Early motivations included enabling direct without explicit inter-node communication, supporting multilevel memory hierarchies for better locality, and facilitating the of existing shared-memory applications to larger-scale systems. Pioneering work, such as Kai Li's 1986 dissertation on software DSM and systems like IVY (developed in 1988), laid the foundation by using mechanisms to manage shared pages across machines. At its core, DSM relies on protocols to ensure memory coherence and consistency, addressing the challenges of data replication and synchronization in a distributed environment. Coherence protocols, such as write-invalidate (which invalidates copies on writes) and write-update (which broadcasts updates), maintain a consistent view of shared data, often integrated with caching and network interconnects. Consistency models vary to balance performance and correctness: strict models like enforce a global of operations, while weaker ones like release consistency relax ordering except at synchronization points, reducing communication overhead. Implementation strategies include software-only approaches using operating system support for page migration or replication, hardware-assisted designs with dedicated coherence hardware, and language-based systems that extend programming models (e.g., Linda or ) to handle distribution implicitly. Notable DSM systems illustrate these concepts: software examples include Munin (with release consistency for object-based sharing) and TreadMarks (using lazy release consistency for page-level updates), while hardware prototypes like (Stanford's directory-based system) and Alewife (MIT's cache-coherent design) demonstrated scalable coherence over networks. Commercial efforts, such as Apollo Domain in the 1980s, brought DSM to practical use, though adoption has been tempered by ongoing challenges like network latency, contention for shared data, and the complexity of in heterogeneous environments. Despite these hurdles, DSM remains influential in modern , influencing hybrid systems and cloud-scale parallelism. In recent years, DSM concepts have influenced advancements like (CXL) for memory pooling in data centers and software DSM for distributed AI workloads.

Introduction

Definition and Core Principles

Distributed shared memory (DSM) is an abstraction in systems that provides a virtual shared across physically distributed nodes, enabling processes to read and write remote memory locations as if they were local, while transparently managing the underlying network communication and data movement. This approach combines the programming simplicity of shared-memory models with the of distributed architectures, allowing unmodified parallel applications to execute without explicit . At its core, DSM operates on the principle of creating an illusion of uniform memory access in a distributed setting, akin to (NUMA) but applied to loosely coupled multiprocessors where physical memories are not directly interconnected. This uniformity is maintained by organizing shared data into units such as local memory pages or objects that are mapped to remote nodes, with the system exploiting to minimize communication overhead. Access to non-local data triggers mechanisms like page faults or software traps to intercept references and fetch the required data, ensuring the abstraction remains seamless to the . To preserve data consistency across replicas, DSM incorporates coherence protocols that manage updates and invalidations, though the specifics vary by implementation. The primary abstraction layers in DSM involve either hardware extensions to the memory management unit (MMU) for efficient trapping of remote accesses or purely software-based mechanisms that rely on virtual memory paging to handle interception and resolution. In software DSM, for instance, a memory reference to a missing page generates a fault, prompting the system to transfer the page from another node's memory, effectively "paging" data between processors much like traditional virtual memory pages between main memory and disk. These layers build on virtual memory techniques to hide distribution complexities. Early conceptual foundations for DSM trace back to 1980s research on shared virtual memory, with Kai Li's IVY system (1988) as the pioneering prototype—a page-based implementation on a token-ring network of Apollo workstations that demonstrated a shared virtual address space for parallel computing in loosely coupled environments.

Historical Development

The concept of distributed shared memory (DSM) emerged from early research on multiprocessor systems in the 1970s, which explored ways to provide shared access to memory across multiple processors despite physical distribution. The first formal proposal for DSM arrived in Kai Li's 1986 PhD dissertation at , which proposed shared virtual memory on loosely coupled systems; this was followed by the Ivy system's 1988 implementation of a page-based shared virtual memory abstraction on a network of loosely coupled workstations using virtual memory mechanisms to simulate sharing. Ivy demonstrated the feasibility of software-based DSM by handling page faults to fetch shared pages over the network, laying the groundwork for subsequent systems. During the 1990s, research advanced rapidly with several influential prototypes. The Munin system, developed at the University of Wisconsin starting in 1990, introduced release consistency as a relaxed memory model to reduce communication overhead in software DSM, allowing ordinary accesses to be unordered while synchronization points enforced visibility. At Stanford, the Midway project around 1993 pioneered an object-based DSM approach, where shared data was managed at the granularity of language-level objects rather than pages, integrating compiler support for annotations and entry consistency to minimize invalidations. Commercial adoption began with efforts like Digital Equipment Corporation's (DEC) Memory Channel interconnect in 1994, which enabled hardware-assisted across clusters of Alpha servers by providing low-latency remote memory access and coherence support. Key surveys in the mid-1990s synthesized these developments, with Protić et al. (1996) providing a comprehensive overview of DSM concepts, systems, and algorithms, including classifications of coherence protocols and implementation trade-offs. This period also marked a shift from purely software-based implementations to hardware-assisted designs, exemplified by Stanford's multiprocessor in , which used directory-based to scale across up to 64 processors with distributed physical memory. By the early 2000s, DSM concepts consolidated into broader frameworks, such as integrations with Java-based systems like JavaSpaces (introduced in 1998 as part of ), which extended tuple-space models to support distributed shared state for coordination in networked environments. Standalone DSM research declined as message-passing paradigms like MPI gained dominance for due to better on commodity clusters. However, interest resurged around 2015–2020 with disaggregated memory architectures in cloud data centers, where remote memory pooling over high-speed networks revived DSM-like abstractions. This trend continued into the 2020s with technologies like (CXL), enabling cache-coherent memory sharing across servers, and software systems like MegaMmap (2024), which uses tiered DRAM and storage to expand effective memory capacity in distributed environments.

Fundamental Mechanisms

Page-Based and Object-Based Approaches

Distributed shared memory (DSM) systems implement shared address spaces across networked nodes using two primary approaches based on the granularity of data sharing: page-based and object-based methods. Page-based DSM treats operating system pages, typically 4 KB in size, as the fundamental unit of sharing, leveraging mechanisms to manage data distribution and coherence. When a processor accesses a non-local page, a occurs, triggering the transfer of the entire page over the network from the owning node, which simplifies implementation by aligning with existing OS paging infrastructure. A representative example of page-based DSM is TreadMarks, developed in 1994, which employs a lazy release consistency protocol to update pages only upon synchronization points, reducing unnecessary communication by deferring diff computations until releases. This approach minimizes overhead in applications with infrequent sharing but can incur costs from transferring unmodified portions of pages. In contrast, object-based DSM shares individual data objects or variables as the unit of granularity, enabling finer control over data access and synchronization through compiler or language-level support. This method requires instrumentation to track object boundaries and invocations, allowing systems to transfer only relevant data structures rather than fixed-size pages. , developed in the 1990s at , exemplifies object-based DSM by providing a programming language where shared objects are explicitly declared, with operations on them triggering atomic invocations for consistency . Object-based approaches integrate with object-oriented paradigms, using runtime systems to replicate and migrate objects across nodes, which supports efficient handling of irregular access patterns common in parallel applications. However, this necessitates modifications to the , such as avoiding direct pointer arithmetic on shared objects, to ensure portability and correctness. Mapping strategies in DSM systems handle the translation from virtual addresses in the to physical locations across distributed nodes, often using distributed translation tables or directories to locate data copies. These tables maintain mappings for pages or objects, updated dynamically during migrations or replications, with mechanisms like hashing or hierarchical directories to scale with the number of nodes. To mitigate latency from remote accesses, prefetching techniques anticipate data needs by issuing advance requests based on access patterns, while multiple-copy protocols allow read-only replicas on multiple nodes to reduce contention and fetch times. Prefetching in page-based systems, for instance, can preload adjacent pages or predicted objects, overlapping communication with , as demonstrated in compiler-assisted schemes that analyze loop structures for proactive transfers. Coherence maintenance, essential for both approaches, ensures updates propagate correctly but is handled separately through protocols that interact with these mappings. The choice of granularity involves key trade-offs: page-based methods offer simplicity and low overhead for coarse-grained sharing but suffer from , where unrelated variables on the same page trigger unnecessary invalidations and transfers, potentially degrading performance in fine-grained workloads. Object-based methods reduce by aligning sharing units with semantic boundaries, improving efficiency for applications with localized accesses, but introduce complexity in , object , and , increasing development effort and runtime costs.

Translation and Mapping Strategies

In distributed shared memory (DSM) systems, address translation extends the local memory management unit (MMU) to manage a unified shared across multiple nodes. In software-based DSM implementations, such as user-level systems, this is typically handled by software trap handlers that intercept page faults and perform remote lookups or migrations to resolve references to data not present locally. Hardware-assisted DSM approaches, by contrast, employ dedicated directories or specialized translation hardware to facilitate efficient remote address resolution, often integrating translation closer to the to reduce latency. In page-based approaches, these translation mechanisms are invoked upon virtual-to-physical mapping faults for fixed-size pages. Mapping techniques in DSM determine how shared virtual addresses are assigned to physical locations on specific nodes, balancing locality, scalability, and overhead. Centralized directories concentrate location information on a single node, simplifying management but creating bottlenecks under high contention. Distributed directories, where each node maintains mapping data for a subset of the address space, offer better scalability by spreading the load. Hashing functions are commonly used to locate the home node of data; for instance, simple hash tables distribute pages across nodes in systems like IVY, while consistent hashing enhances scalability by minimizing remapping disruptions during node additions or failures. Data location protocols in DSM specify how shared data is fetched or updated across nodes, optimizing for access patterns and consistency needs. The on-demand migration, or pull model, fetches data to the requesting node only upon an access fault, as implemented in TreadMarks to minimize unnecessary transfers. In contrast, push updates proactively propagate modifications from the owner to other cached copies, reducing fault frequency but increasing network traffic in write-heavy scenarios. Multi-versioning protocols maintain multiple copies of data with associated timestamps, allowing readers to access consistent versions without invalidating others, as seen in Munin's support for release consistency on shared objects. Performance in DSM translation and mapping is influenced by latency components and sharing artifacts, often modeled to guide optimizations. A basic access latency equation captures the overhead as the sum of local hit time and network round-trip time scaled by communication : Access time=tlocal+RTT×h\text{Access time} = t_{\text{local}} + \text{RTT} \times h where tlocalt_{\text{local}} is the local memory access time and hh is the number of network required for resolution or migration. False sharing introduces additional overhead, quantified by the page thrashing rate—the frequency of unproductive page migrations due to unrelated accesses within the same unit—which can degrade throughput by factors of 2–10 in fine-grained workloads.

Advantages and Comparisons

Key Benefits

Distributed shared memory (DSM) systems provide a unified that abstracts the complexities of distributed architectures, allowing developers to use familiar such as shared variables, locks, and barriers without explicit management of data movement across nodes. This approach simplifies the development of parallel applications, as programs written for multiprocessors can often execute on distributed systems with minimal or no modifications, reducing the on programmers compared to explicit message-passing paradigms. A key advantage of DSM lies in its , which effectively hides hardware heterogeneity by presenting a single, coherent across diverse processors and networks. Dynamic mechanisms like page migration enable automatic load balancing, redistributing data to optimize utilization and accommodate varying computational demands without manual intervention. This abstraction facilitates the extension of models to large-scale clusters, combining the ease of shared memory programming with the cost-effectiveness of distributed hardware. In terms of performance, DSM minimizes communication overhead for fine-grained data sharing through techniques such as data replication and lazy consistency protocols, which defer updates until necessary and leverage local caching to enhance data locality. For instance, protocols like Lazy Release Consistency in systems such as Munin reduce message traffic by up to an order of magnitude for certain workloads, achieving performance within 10% of optimized message-passing implementations while maintaining the benefits of shared access. These gains are particularly pronounced in applications with good temporal and spatial locality, where the overhead of coherence maintenance is offset by fewer remote accesses. DSM proves especially beneficial for use cases involving irregular access patterns or scientific simulations, such as solvers, where predicting data needs in advance is challenging. In these scenarios, DSM's on-demand page faulting and multiple consistency protocols—tailored for read-only, migratory, or write-shared data—efficiently handle unpredictable without the inefficiencies of bulk data transfers common in message-passing systems. Examples include graph algorithms, radiative models, linkage analysis in , and irregular computations in physics simulations, where DSM supports effective parallelism by abstracting distribution details. In modern contexts as of 2025, DSM concepts continue to offer advantages in disaggregated data centers and distributed AI, enabling efficient memory sharing across tiered DRAM and storage to enlarge effective capacity and support scalable workloads.

Comparison with

Distributed shared memory (DSM) and represent two fundamental paradigms for in parallel and distributed systems, with DSM providing an abstraction of a unified across nodes and relying on explicit data exchange via libraries like MPI. While DSM simplifies programming by allowing direct memory accesses as in shared-memory models, it incurs notable disadvantages compared to . Remote memory accesses in DSM often exhibit higher latency due to the need for network traversal and protocol handling, contrasting with the potentially optimized point-to-point transfers in . Additionally, DSM introduces complexity through coherence overhead, as maintaining consistent views of shared data requires invalidations, updates, or acknowledgments that can amplify communication costs, particularly in the presence of . Scalability in DSM is further limited by directory-based coherence mechanisms, where full-map directories require O(N) storage per cache line for N nodes to track sharers, leading to contention and increased access times as system size grows. Message passing, by contrast, offers strengths in scenarios demanding explicit control over data movement. Programmers can optimize bulk transfers—such as large copies—using collective operations or buffered sends, reducing unnecessary overhead compared to DSM's fine-grained, potentially frequent remote reads and writes. This paradigm suits loosely coupled applications, where processes operate independently with infrequent synchronization, avoiding the pitfalls of implicit shared . Moreover, message passing eliminates bugs arising from unintended shared state modifications, such as race conditions on global variables, since data ownership is explicit and local. Performance comparisons between the paradigms depend heavily on application characteristics. In tasks like generating the , DSM and (e.g., via PVM or MPI) achieve similar speedups, with near-linear scaling up to 24-32 nodes on clusters. However, for applications involving more frequent , such as mergesort, DSM underperforms due to higher network —up to 60% more than MPI at 8 nodes—stemming from coherence traffic and , while maintains better efficiency. DSM tends to excel in shared-data-intensive workloads with irregular or fine-grained accesses, where the reduces programming effort, though outperforms in bulk-synchronous or loosely coupled scenarios by minimizing hidden communications. To address the limitations of pure paradigms, hybrid approaches emerged in the 2000s, combining DSM's ease with 's control. Unified Parallel C (UPC), for instance, provides a that allows shared memory-style accesses while enabling explicit data distribution and locality control akin to , supporting scalable performance on clusters without full coherence overhead.

Implementation Paradigms

Software DSM Systems

Software distributed shared memory (DSM) systems implement the DSM abstraction entirely in software, typically as user-level libraries or runtime environments layered on top of commodity hardware and operating systems, without relying on specialized hardware support. These systems provide programmers with a shared memory model across distributed nodes, handling data distribution, coherence, and communication transparently. Core components often include mechanisms for tracking page modifications through diffing techniques, where only changed portions of memory pages are exchanged during updates to reduce communication overhead. For instance, TreadMarks, developed in 1994, employs a user-level library on standard Unix systems like SunOS, using diff-based multiple-writer protocols to manage page updates efficiently. Additionally, some systems incorporate compiler support to facilitate object migration, enabling finer-grained sharing by analyzing and relocating objects across nodes to minimize remote accesses, as seen in Jackal, a Java-based DSM that optimizes object graphs through source-level analysis. Key algorithms in software DSM focus on balancing consistency with performance through release consistency variants. Lazy release consistency protocols, such as those in TreadMarks, defer the propagation of updates until an acquire operation, reducing unnecessary invalidations and data transfers compared to eager protocols that push modifications immediately at release points. This approach significantly reduces the volume of communicated data compared to eager protocols, as demonstrated in benchmarks like SPLASH. For efficiency in read-heavy scenarios, many systems adopt multiple readers/single writer (MRSW) protocols, allowing multiple nodes to hold read copies of a page while restricting writes to a single owner, thereby minimizing coherence traffic. Notable examples include open-source projects like J-DSM, a Java-based framework from the early 2000s that supports sharing of both mobile and stationary objects via DSM interfaces, enabling finer control over granularity in distributed applications. Integration with virtual machines, such as JVM clustering, extends this to multithreaded programs, where DSM layers abstract shared state across JVM instances. More recently, cloud-oriented systems like Apache Ignite, with its in-memory features introduced around 2014 and extended post-2018, provide DSM-like abstractions through off-heap regions and distributed caching, supporting scalable data sharing in cloud environments without custom hardware. Software DSM addresses challenges like portability across operating systems by relying on standard APIs and avoiding kernel modifications, as exemplified by TreadMarks' compatibility with multiple Unix variants. Overhead is mitigated through application-specific optimizations, such as selective handling or compiler-directed prefetching, which can reduce latency by tailoring coherence to workload patterns.

Hardware-Assisted DSM

Hardware-assisted distributed shared memory (DSM) systems incorporate specialized hardware to manage and sharing across distributed nodes, reducing latency and overhead compared to purely software-based approaches. These systems typically feature custom caches and directory structures to track locations and ensure consistency without relying on operating system traps for every access. A seminal example is the (Directory Architecture for Shared Memory) prototype developed at Stanford in 1992, which used hardware directories to support scalable in a multiprocessor environment with up to 64 processors. This design allowed shared to be cached locally, minimizing remote memory access latencies and improving overall system utilization. Key hardware elements include (RDMA) mechanisms, such as those enabled by interconnects, which provide low-latency, direct memory transfers between nodes without CPU intervention. InfiniBand's RDMA capabilities facilitate efficient DSM by allowing applications to access remote memory as if it were local, supporting high-bandwidth operations in cluster environments. Protocols in hardware-assisted DSM often extend snooping mechanisms to clusters, where coherence requests are propagated via hardware networks rather than broadcasts limited to small symmetric multiprocessors. The Scalable Coherent Interface (SCI), standardized in 1992, exemplifies this by using distributed directories and request-response flows over point-to-point links to maintain coherence in large-scale DSM setups. Additionally, NUMA-aware interconnects like Intel's QuickPath Interconnect (QPI), introduced around 2008 and used until the late 2010s, optimized memory access in multi-socket systems by routing coherence traffic efficiently across nodes. Performance benefits of hardware assistance stem from drastically reduced software overhead; for instance, coherence operations bypass costly page traps, achieving up to 10 times faster response times in benchmarks compared to software DSM equivalents. The SGI Origin series, deployed in the , demonstrated this scalability in (HPC) environments, supporting hundreds of processors with cc-NUMA architecture for shared-memory applications. In modern contexts, (CXL), emerging in the 2020s, enables disaggregated memory pooling across devices, allowing dynamic allocation of remote memory resources while maintaining coherence. The evolution of hardware-assisted DSM in HPC reflects a shift toward integrated, low-latency interconnects to handle growing scale and heterogeneity. Early systems like and Origin laid the foundation for directory-based coherence in clusters, while contemporary advancements incorporate GPU integration via technologies like or CXL extensions, enabling unified address spaces across CPU-GPU fabrics for accelerated computing workloads. This progression enhances scalability in supercomputing, where hardware offloads coherence management to support exascale simulations and data-intensive tasks.

Memory Coherence

Directory-Based Protocols

Directory-based protocols maintain in distributed shared memory (DSM) systems by using a directory to track the sharing status of individual blocks across nodes, enabling selective communication rather than broadcasting to all processors. This approach, first proposed in the late , addresses the limitations of snooping protocols, which become inefficient in large-scale systems due to excessive network traffic from broadcasts. By requests point-to-point to the relevant nodes, directory protocols reduce contention and support in DSM environments with dozens or hundreds of processors. The directory structure varies between flat and hierarchical designs to balance storage efficiency and performance. Flat directories associate each memory block with a fixed home node, where the directory entry—often a bit-vector indicating which nodes cache the block—is stored alongside the block's data in main memory. For instance, a simple bit-vector uses one bit per node to mark presence, limiting support to systems with up to 64 nodes using a 64-bit vector, while more advanced variants employ pointers or masks to handle larger configurations with reduced precision for widespread sharing. Hierarchical directories, in contrast, organize entries in a multi-level tree, with each level tracking coarse-grained presence information for subgroups or subtrees of nodes; this reduces per-block storage from O(N) bits (for N nodes) to O(log N) by propagating queries up the hierarchy. Operations in directory-based protocols center on request handling at the home node to enforce coherence. For a read request, the requesting node sends a to the home, which checks the directory and either supplies the block directly or forwards a copy from another holder if shared; no action is needed if the block is already local. Write requests similarly route to the home node, which examines the directory to issue invalidation to all sharers, ensuring exclusive access before granting permission, thus preventing stale data propagation. These invalidations are selective, targeting only nodes listed in the directory, which minimizes unnecessary compared to broadcast methods. Scalability in these protocols stems from their avoidance of global broadcasts, with flat designs offering constant-time directory access at the cost of linear storage growth, while hierarchical variants achieve O(log N) lookup latencies through , suitable for DSM systems. However, this comes with trade-offs in memory overhead, as full bit-vector directories can consume space comparable to the total cache size—around 12% extra for 64-node systems—prompting optimizations like limited-pointer schemes that track only a small number of sharers per block. These structures integrate with various coherence states within the directory to manage block permissions efficiently.

Coherence States

In directory-based coherence protocols for distributed shared memory (DSM) systems, memory blocks are maintained in one of several standard states to ensure consistency across distributed nodes. The primary states include Uncached (or Invalid), where no node holds a copy of the block and the home memory location contains the most recent value; Shared, where multiple nodes may hold read-only copies and the memory is up-to-date; and Exclusive (or Dirty/Modified), where a single node holds the only copy, which may be writable, and the memory may be stale until updated. Extensions to these states often incorporate a to track whether a block in the Exclusive state has been modified, signaling that the home memory needs updating upon or replacement. Additionally, transient pending states, such as Pending or Transient, are used during ongoing operations to handle race conditions or incomplete transactions, preventing premature state changes until all actions resolve. These states are stored in the associated with the block's home node. Protocol transitions between states are triggered by access requests. For instance, a write request to a block in the Shared state initiates an invalidation phase, where the directory sends invalidation messages to all sharing nodes, transitioning the block to Exclusive in the requesting node upon receiving acknowledgments from sharers, ensuring no conflicting copies remain. Similarly, a read request to an Exclusive block may prompt the owner to supply and transition to Shared, updating the directory's sharer information, while write-backs from Exclusive to Uncached clear the directory entry. Acknowledgments are critical in these transitions to confirm invalidations or transfers, reducing the risk of stale propagation. To optimize storage in large-scale DSM systems, directory variants balance accuracy and space efficiency. Full-bit directories use a bit vector per block, with one bit per potential sharer to precisely track all nodes in Shared or Exclusive states, though this scales poorly with node count (O(N) space for N nodes). Coarse directories mitigate this by employing limited pointers or chained lists that reference only active sharers, such as a fixed number of slots (e.g., 4-8) with overflow chains, approximating the full set while reducing overhead for sparsely shared blocks.

Request and Response Flows

In directory-based protocols for distributed shared memory systems, request and response flows manage data access and maintain coherence by routing messages through the node, which holds the directory tracking block locations and permissions. These flows can follow a -centric approach, where the node centralizes all decisions by directing interventions to owners or sharers and coordinating responses, or a requester-centric (also called requester-assisted) approach, where the requester node directly communicates with owners or sharers after receiving initial directory information from the , thereby reducing the 's bottleneck at the cost of added complexity in request ordering. The -centric model simplifies implementation but incurs higher latency due to multiple hops through the , while the requester-centric model enables overlapping transactions by allowing the requester to poll or for data, with the updating the directory only after responses confirm coherence. For a read miss on a clean block (not cached elsewhere), the requester sends a GetS (shared access) request to the node, which supplies the from local memory, updates the directory to mark the requester as a sharer, and transitions the block state to shared if necessary; in a -centric flow, the awaits an acknowledgment from the requester before closing the transaction, whereas in requester-centric flows, the sends the and immediately closes, leaving the requester to buffer any conflicting requests. If the block is dirty (cached exclusively by an owner), the forwards an intervention request to the owner in both approaches; the owner supplies the directly to the requester and acknowledges the to update the directory to shared state, forming a reply chain that typically involves 4-5 point-to-point messages and transitions the owner's state from modified to shared. Write misses follow a similar initial routing to the home but require exclusive access, prompting the directory to identify and invalidate all sharers before granting permission. In the flow, the requester issues a GetM (modified access) or ReadX request; the home sends invalidation messages to sharers (or their list to the requester in requester-centric variants), collects acknowledgments to ensure no lingering copies, supplies or forwards the latest data, and updates the directory to exclusive state for the requester, often serializing the process to avoid races. This invalidation phase scales with the number of sharers but is efficient in practice, as applications like Barnes-Hut exhibit few concurrent sharers in 64-processor systems, resulting in 3-7 messages total. Optimizations mitigate latency in these flows: bypassing allows local cache hits to skip directory involvement entirely, while request forwarding (or ) in requester-centric protocols directs the read request past the home to the current owner, reducing the critical path by one hop and enabling direct data transfer, as demonstrated in the protocol where such interventions cut latency by up to 20% compared to home-routed alternatives.

Consistency Models

Sequential Consistency

Sequential consistency is the strongest memory consistency model in distributed shared memory (DSM) systems, ensuring that all memory operations appear to execute atomically and in the order specified by each processor's program across all processors. Formally, a DSM system is sequentially consistent if the result of any execution is the same as if the operations of all processors were executed in some sequential order that respects the program order for each individual processor. This model, introduced by in 1979, provides programmers with an intuitive shared-memory , as if all operations occur on a single, globally visible timeline. In DSM environments, achieving requires maintaining a global of all memory accesses while preserving per-processor program order. This necessitates explicit mechanisms, such as barriers or locks, to enforce ordering between operations on different processors, and strict protocols for invalidating or updating remote copies upon writes to ensure atomicity. For instance, if processor P executes operation A_i before B_j in its program order, then in the global serialization, A_i must precede B_j for all processors to observe consistent results. Mathematically, for operations from processor P denoted as O_{P,k}, the model requires that for any i < j, O_{P,i} appears before O_{P,j} in the total order σ of all operations across processors. Implementing in DSM typically relies on eager protocols, where writes trigger immediate invalidations or updates to all relevant copies to propagate changes synchronously, avoiding delayed visibility. However, this approach incurs high overhead due to frequent network communications and synchronization, often resulting in 2-5x slowdowns compared to weaker models in page-based systems, primarily from and manager bottlenecks. Early DSM systems like Ivy enforced using a central manager for page ownership and write-invalidate protocols, demonstrating the model's feasibility but highlighting its scalability limitations in distributed settings.

Weak Consistency Variants

Weak consistency variants in distributed shared memory (DSM) systems relax the stringent requirements of to improve by allowing greater flexibility in the ordering of operations across processors. These models permit optimizations such as operation reordering and buffering, which reduce the overhead of maintaining strict global ordering, at the cost of increased programmer responsibility for explicit to ensure correct program behavior. Processor consistency maintains the order of writes issued by a single processor as seen by other processors, ensuring that writes from one processor do not appear out of order to others, but allows arbitrary interleaving of operations from different processors without enforcing transitivity between reads and writes. This model avoids the need for immediate global visibility of writes, enabling processors to proceed without waiting for acknowledgments from all others, but it does not guarantee that a read by one processor will see the most recent write from another unless is used. Unlike , which imposes a single on all operations, processor consistency trades some ordering guarantees for reduced synchronization overhead. Weak consistency further relaxes ordering by distinguishing between ordinary memory operations and synchronization operations, such as locks or barriers, enforcing strict ordering only at synchronization points while allowing asynchronous reordering of non-synchronized accesses to different memory locations. In this model, writes may be buffered and propagated lazily until a synchronization event, after which all prior writes become globally visible, and subsequent reads reflect those updates. This approach, as classified by Adve and Gharachorloo, provides a safety net through synchronization to restore consistency without requiring atomicity for all operations. The primary benefits of these weak consistency variants in DSM systems include substantially reduced coherence traffic and fewer unnecessary cache invalidations, as systems can defer propagation of updates and avoid immediate broadcasting to all nodes. For instance, by hiding write latencies through reordering, weak models enable and hardware optimizations that improve overall throughput, often at the expense of programming complexity due to the need for explicit . Variants such as release consistency build on weak consistency by associating consistency guarantees more tightly with release operations at synchronization points, allowing even more aggressive optimizations while preserving —ensuring that dependent operations appear in order—though detailed implementations are addressed separately. Causal consistency, another variant, preserves the order of causally related operations across processors to maintain intuitive program semantics without full sequential ordering.

Replication Techniques

Replication Strategies

In distributed shared memory (DSM) systems, replication strategies aim to enhance availability and performance by maintaining multiple copies of shared data across nodes, reducing latency for concurrent accesses while managing coherence overheads. These strategies distinguish between data access patterns to optimize resource utilization, leveraging techniques that allow multiple nodes to hold copies without constant synchronization for reads. Seminal systems like IVY and Munin exemplify early implementations, where replication is integrated with virtual memory mechanisms to provide the illusion of a unified address space. Read-only replication permits multiple read copies of data across nodes, enabling parallel reads without invalidations triggered by read operations themselves. In this approach, updates—if any occur after designation as read-only—are propagated to a designated home node or all copies, but since the data is immutable post-initialization, no further coherence actions are typically required, avoiding the need for frequent messaging. For instance, in Munin, read-only objects are replicated on demand during initialization, allowing local access without runtime write permissions, which simplifies consistency and reduces network traffic for applications with high read ratios. This strategy is particularly effective for static data like code segments or constants, where the absence of writes eliminates costs. For read-write replication, multiple copies support both reads and writes, often employing quorum-based mechanisms to ensure fault tolerance and detect conflicts. A majority quorum for writes requires acknowledgments from a subset of replicas (e.g., more than half) before committing updates, while reads may use a smaller intersecting quorum to retrieve the latest version, preventing stale data. Conflict detection relies on versions or timestamps attached to copies; upon write, a node compares timestamps to identify divergences and resolves them via rollback or merging at synchronization points. This approach, while increasing availability, introduces coordination overhead, as seen in systems extending directory protocols to track replica sets and propagate updates selectively. Replication strategies vary in adaptability, with static approaches assigning fixed locations at initialization based on predicted access patterns, minimizing reconfiguration costs but risking suboptimal if workloads shift. Dynamic strategies, in contrast, migrate s or adjust copy counts in response to runtime access frequencies, such as promoting frequently read data to additional nodes or relocating write copies to hot-spot processors. Surveys of DSM algorithms highlight that dynamic methods, like those in competitive schemes, can achieve significant reductions in average latency, such as 5-10x speedups over static methods in variable workloads through adaptive distribution, though they incur monitoring overhead. Directory-based protocols may be extended briefly to track dynamic locations, facilitating efficient forwarding of requests. The scope of replication—whether full-page or object-level—impacts granularity and efficiency. Page-based replication, as in IVY, duplicates entire fixed-size pages (e.g., 4KB), leveraging hardware page faults for access but suffering from false sharing where unrelated data within a page causes unnecessary invalidations. Object-based replication, exemplified by Munin, targets programmer-defined objects with finer boundaries, allowing independent coherence for components like arrays or structures, which often reduces communication volume significantly, such as order-of-magnitude reductions in message counts for certain protocols, but requires additional runtime support for object boundaries. This choice balances transparency with overhead, prioritizing object granularity for complex data structures. Replication introduces overheads, including storage duplication where each copy consumes local , potentially doubling usage for highly replicated across N nodes. Consistency adds costs through mechanisms like gossip protocols for asynchronous update (incurring O(log N) messages per node) or chain-based updates that sequentially forward changes along a path, amplifying latency in linear topologies. These trade-offs are evident in early DSM prototypes, where replication improved read throughput significantly but increased write costs due to .

Consistency in Replicated Environments

In replicated distributed shared memory (DSM) environments, maintaining consistency across multiple copies of data introduces significant challenges. Stale copies arise when updates to one are not immediately propagated, potentially allowing processors to read outdated data after a write has occurred elsewhere. Write is required to ensure that concurrent writes do not conflict, often restricting access to a single writer at a time while permitting multiple readers, which can limit parallelism and . Additionally, partition tolerance during failures complicates consistency, as network partitions or node crashes can isolate replicas, risking divergent states unless explicit recovery mechanisms are in place. To address these issues, the primary-copy protocol designates one as writable, with all write operations directed to it, while other replicas remain read-only until updated by the primary. This approach serializes writes at the primary node, ensuring consistency by propagating changes to secondary copies upon completion, though it introduces latency for remote writers. Concurrency control in replicated DSM further distinguishes between pessimistic and optimistic strategies: pessimistic methods, such as eager invalidation or update protocols, enforce immediate to prevent conflicts by invalidating or changes at write time, minimizing the risk of stale data but increasing communication overhead. In contrast, optimistic approaches, like lazy release consistency, defer until points (e.g., lock acquires), performing commit-time checks to detect and resolve conflicts via if necessary, which reduces contention in low-conflict workloads. Weaker consistency models in replicated DSM facilitate these solutions by permitting delayed propagation of updates, allowing replicas to operate independently until convergence is needed, thus balancing performance and correctness. Version vectors track causality and detect conflicts in such settings; each replica maintains a vector of counters for updates from each node, and upon merging, a replica ii computes its version vector as VVi=max⁡(VVj ∀ j∈replicas)VV_i = \max(VV_j \ \forall \ j \in \text{replicas}), enabling identification of concurrent modifications or stale states. Fault tolerance in replicated DSM relies on mechanisms like replica recovery through backup copies and state replication on separate hosts to tolerate single failures. For availability, quorum-based reads and writes ensure progress despite failures; reads may access a single local copy or a quorum, while writes require acknowledgment from a write quorum (e.g., all or a of replicas), drawing from consensus principles to maintain consistency under partitions. Recovery involves detecting failures via timeouts and reconstructing state by polling or copying from surviving quorate replicas, typically within seconds.

Specialized Protocols

Release Consistency

Release consistency (RC) is a memory consistency model for distributed shared memory systems that relaxes the ordering requirements of by enforcing consistency only at explicit points, such as and release operations. In this model, ordinary memory operations (loads and stores) are grouped into sets (pending before an ) and release sets (pending before a release), allowing implementations to buffer and reorder these operations freely as long as accesses maintain proper ordering. RC serves as a refinement of weak consistency, providing programmers with explicit control over when memory views are synchronized while enabling hardware and software optimizations to reduce communication overhead. The formal rules of release consistency, known as R1 through R4, define the ordering guarantees:
  • R1: Before an ordinary load or store is performed, all previous accesses must have completed.
  • R2: Before a release access is performed, all previous ordinary loads and stores must have completed.
  • R3: Special () accesses are processor-consistent with respect to one another, meaning acquires and releases are ordered globally.
  • R4: Local data dependences within a processor are preserved.
These rules ensure that operations before a release are visible to subsequent acquires, but permit aggressive reordering otherwise. A key variant, lazy release consistency (LRC), defers invalidations and updates until necessary, typically at acquire points, rather than propagating them eagerly at releases; this reduces unnecessary network traffic in software distributed shared memory systems by pulling only relevant changes on demand. In the Munin software distributed shared memory system, release consistency is implemented by having the insert synchronization primitives around critical sections, buffering writes in a delayed update queue until release points, which merges and sends updates only to relevant processors. Optimizations in RC implementations distinguish between eager and lazy release strategies: eager RC pushes updates or invalidations to all potential readers at release time, ensuring immediate visibility but increasing bandwidth use, while lazy RC delays this until acquires, trading potential latency for reduced traffic. Additionally, processor-consistent RC (pc-RC) binds ordering to lock-based , treating ordinary accesses as processor-consistent (ordered per processor) and relying on acquires/releases for global , which simplifies hardware support compared to sequentially consistent RC (RCsc).

Entry Consistency

Entry consistency (EC) is a relaxed designed for distributed shared memory (DSM) systems, where coherence is enforced on a per-shared-object basis specifically at the points of acquisition, such as lock acquires. In this model, a processor sees a consistent view of a shared object only after acquiring the synchronization variable (e.g., lock) that explicitly protects it, ensuring that updates to that object are propagated or invalidated at those entry points. This finer-grained approach builds on release consistency by requiring programmers to associate specific shared objects with their guarding synchronization primitives, thereby limiting the scope of consistency guarantees to relevant data and synchronization pairs. The core rules of entry consistency dictate that operations on a shared object O become visible to a processor P only if P has d the lock protecting O subsequent to the write operations in question. Specifically, a read by P to O must return the value of the most recent write to O by any processor that precedes P's of the lock in the global order of events. Writes to O are not required to be immediately visible to other processors until they the associated lock, and the model supports both exclusive () and shared (reader) modes for acquires to optimize concurrent access. By tying coherence actions directly to these events, EC avoids propagating updates or invalidations for unrelated objects, significantly reducing network traffic in DSM environments. Implementations of entry consistency, such as in the Midway DSM system developed at , rely on programmer annotations to declare associations between shared data and objects, with the runtime employing an update-based protocol to fetch or apply changes upon acquires. plays a key role in enhancing efficiency by identifying fine-grained object boundaries, enabling automatic generation of annotations and reducing page-level coherence overheads through sub-page tracking with version numbers. For instance, Midway supports multiple consistency models within the same program, using software mechanisms on networked workstations to handle caching and ownership of both data and objects. One primary advantage of entry consistency is its ability to minimize and unnecessary invalidations, as coherence is scoped to specific lock-protected objects rather than broader events, leading to lower communication costs. In benchmarks like on eight processors, EC implementations transferred approximately half the data volume (3.4 MB vs. 7.1 MB) compared to lazy release consistency variants, while also reducing message counts in applications with clustered data access patterns (e.g., 2,517 messages vs. 7,175 for 3D-FFT). This can yield up to 2x performance improvements over release consistency in scenarios with high , such as parallel sorting algorithms. However, challenges include the burden of explicit annotations and the need for careful lock granularity design, as overly fine-grained locks may increase contention without proportional coherence gains.

Practical Examples

One prominent historical example of a distributed shared memory (DSM) system is Munin, developed in the early , which implemented release consistency using multiple protocols tailored to data types, such as write-shared for producer-consumer patterns and conventional coherence for locks. Munin ran on machines like the BBN TC2000, employing a delayed update queue to buffer writes until synchronization points, thereby reducing communication overhead. Performance evaluations showed Munin achieving speeds within 10% of optimized message-passing implementations for applications like matrix multiply and on up to 16 processors, with multi-protocol optimizations further reducing execution times by up to 50% compared to single-protocol approaches. TreadMarks, introduced in 1994, was a software DSM system for standard Unix workstations connected via LANs, utilizing lazy release consistency with page-based diffing to track and propagate updates only at . It supported multiple writers and minimized through multiple-writer protocols, running applications like water simulation and integer sort on clusters of DECstations. Benchmarks indicated TreadMarks delivered performance comparable to message-passing systems like PVM and MPI for workloads, though it lagged by 20-30% for fine-grained applications due to higher communication costs. Orca, from 1995, pioneered object-based entry consistency on distributed systems, using compile-time analysis and runtime heuristics for object placement and a write-update protocol with function shipping to reduce data transfer. Implemented on platforms like transputers and later Myrinet clusters, it supported parallel programs in a custom language with abstract data types for . On a 32-node cluster, Orca achieved speedups of up to 30.6x for simulations like , outperforming page-based systems like TreadMarks by sending 6-12x fewer messages and data volumes, while matching or exceeding object-based peers like CRL in execution time for N-body simulations. In modern contexts, disaggregated DSM has gained traction in cloud environments, exemplified by InfiniSwap from 2017, which leverages RDMA for remote paging to pool cluster memory without application changes. Designed for low-latency networks like , it exposes remote memory as a block device, enabling efficient swapping and reducing CPU involvement in transfers. Benchmarks on workloads including VoltDB's TPC-C and showed throughput gains of 4-15x and tail latency reductions up to 61x over disk-based spilling, while boosting overall cluster memory utilization by 1.47x across big data applications like PowerGraph. FaRM, developed by in 2014, provided an RDMA-based DSM for in-memory , exposing cluster memory as a shared with support for transactions via lock-free protocols. It used one-sided RDMA reads for direct access, bypassing traditional messaging, on clusters of 20 machines with Ethernet or . Performance tests demonstrated 167 million key-value operations per second at 31 µs latency, achieving 10x higher throughput than TCP-based equivalents and 100x lower latency, making it suitable for large-scale data serving. For (HPC), extensions to OpenSHMEM have integrated disaggregated memory, allowing one-sided access to remote pools via fabrics like HPE for scalable PGAS programming. This enables applications to offload excess data to remote memory without MPI-style collectives, targeting irregular access patterns in simulations. In benchmarks, hybrid OpenSHMEM with disaggregated memory reduced end-to-end times by 45-55% for large-scale radix sorts compared to local-memory-only variants, with sparse matrix-vector multiplication showing substantial gains at scale due to faster I/O over traditional file systems. Emerging CXL-based DSM systems, such as CXL-SHM from 2023, utilize for cache-coherent memory pooling across servers, supporting reference-counting for automatic allocation and partial failure recovery. Implemented on PCIe-attached CXL devices, it enables low-latency remote access at 390 ns for sequential operations. Evaluations on HPC workloads like word count yielded 80-87% faster execution than non-CXL baselines, while key-value stores achieved 117 million operations per second, 1.4-2.6x slower than local allocators but 10-1000x faster than persistent alternatives. More recent advancements as of include DRust, a Rust-based DSM system that uses language-guided fine-grained consistency to support scalable distributed programming without explicit management. DRust outperforms prior systems like GAM and in benchmarks, achieving better on multi-core clusters by leveraging Rust's model for automatic coherence. These examples illustrate DSM's viability in diverse settings, with benchmarks highlighting its advantages in irregular HPC workloads; for instance, OpenSHMEM extensions have demonstrated up to 55% time reductions over local configurations, underscoring for data-intensive tasks where incurs higher overhead.

References

  1. https://www.cise.ufl.edu/~nemo/cop5615/chow/ch7.pdf
  2. https://dl.acm.org/doi/10.1109/88.494605
  3. https://webdocs.cs.ualberta.ca/~tony/TechnicalReports/TR96-22.pdf
  4. https://www.sigops.org/2024/revisiting-distributed-memory-in-the-cxl-era/
  5. https://ieeexplore.ieee.org/document/10938434/
  6. https://www.cs.uic.edu/~ajayk/Chapter12.pdf
  7. https://people.eecs.berkeley.edu/~prabal/resources/osprelim/LH89.pdf
  8. https://crystal.uta.edu/~kumar/cse6306/papers/17.pdf
  9. https://systems.cs.columbia.edu/ds2-class/papers/li-ivy.pdf
  10. https://css.csail.mit.edu/6.824/2014/papers/li-dsm.pdf
  11. https://people.eecs.berkeley.edu/~kubitron/cs258/handouts/papers/p13-keleher.pdf
  12. https://csis.pace.edu/~marchese/CS865/Papers/bershad00289730.pdf
  13. http://shiftleft.com/mirrors/www.hpl.hp.com/hpjournal/dtj/vol9num1/vol9num1art3.pdf
  14. https://sites.cc.gatech.edu/classes/AY2009/cs4210_fall/papers/DSM_protic.pdf
  15. http://rsim.cs.uiuc.edu/arch/qual_papers/arch/lenoski_dash.pdf
  16. https://akougkas.io/assets/pdf/megammap.pdf
  17. https://www.usenix.org/conference/usenix-winter-1994-technical-conference/tread-marks-distributed-shared-memory-standard
  18. https://dl.acm.org/doi/10.5555/1267074.1267084
  19. https://ieeexplore.ieee.org/document/1419873
  20. http://ieeexplore.ieee.org/document/495453
  21. https://ieeexplore.ieee.org/document/562899
  22. https://dl.acm.org/doi/10.1145/1556444.1556452
  23. https://ieeexplore.ieee.org/document/546211
  24. https://dl.acm.org/doi/10.1145/506378.506384
  25. https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.4330050409
  26. https://semiengineering.com/distributed-shared-memory-that-enlarges-effective-memory-capacity-through-intelligent-tiered-dram-and-storage-management-iit/
  27. https://dl.acm.org/doi/10.1145/195473.195501
  28. http://www.cs.ucr.edu/~bhuyan/CS213/2004/LECTURE11a.pdf
  29. https://dl.acm.org/doi/10.1145/155332.155338
  30. https://www.researchgate.net/publication/4037750_A_performance_comparison_of_DSM_PVM_and_MPI
  31. https://upc.lbl.gov/
  32. https://dl.acm.org/doi/10.1109/2.121510
  33. https://www.sciencedirect.com/science/article/pii/S0743731505001103
  34. https://link.springer.com/content/pdf/10.1007/10704208.pdf
  35. https://www.intel.cn/content/dam/doc/white-paper/quick-path-interconnect-introduction-paper.pdf
  36. https://people.eecs.berkeley.edu/~kubitron/papers/alewife/pdf/mgs-tocs.pdf
  37. https://www.cs.umd.edu/class/fall2019/cmsc714/readings/SGI-Origin-2000.pdf
  38. https://dl.acm.org/doi/10.1145/3723851.3723858
  39. https://www.researchgate.net/publication/3046972_A_New_Solution_to_Coherence_Problems_in_Multicache_Systems
  40. https://docs.lib.purdue.edu/context/ecetr/article/1724/viewcontent/TR_EE_90_33.pdf
  41. https://www.cs.cmu.edu/afs/cs/academic/class/15418-s19/www/lectures/13_directory.pdf
  42. https://www.researchgate.net/publication/3503862_The_directory-based_cache_coherence_protocol_for_the_DASHmultiprocessor
  43. https://passlab.github.io/CSCE513/notes/lecture24_TLP_DSMDirectory.pdf
  44. https://dl.acm.org/doi/10.1145/325096.325132
  45. https://pages.cs.wisc.edu/~markhill/papers/primer2020_2nd_edition.pdf
  46. https://www.csc2.ncsu.edu/faculty/efg/courses/506/s22/www/lectures/notes/lec23.pdf
  47. https://lamport.azurewebsites.net/pubs/lamport-how-to-make.pdf
  48. http://sadve.cs.illinois.edu/Publications/computer96.pdf
  49. https://crystal.uta.edu/~kumar/cse6306/papers/Ramesh.pdf
  50. http://www0.cs.ucl.ac.uk/staff/B.Karp/gz03/f2011/lectures/gz03-lecture5-Ivy.pdf
  51. https://doi.org/10.1109/2.546611
  52. https://students.mimuw.edu.pl/SO/Wyklady-html/15_dsm/munin-design.pdf
  53. http://nil.csail.mit.edu/6.824/2017/papers/munin.pdf
  54. https://dl.acm.org/doi/10.1007/978-3-642-25873-2_3
  55. https://groups.csail.mit.edu/tds/papers/Lynch/FTCS97.pdf
  56. https://apps.dtic.mil/sti/tr/pdf/ADA207322.pdf
  57. https://ijcaonline.org/volume1/number6/pxc387271.pdf
  58. https://www.researchgate.net/profile/David-Black-19/publication/3499552_Competitive_management_of_distributed_shared_memory/links/5d4063c24585153e592fdd1e/Competitive-management-of-distributed_shared_memory.pdf
  59. https://www.cs.vu.nl/~ast/Publications/Papers/cpe-1992.pdf
  60. https://publications.scss.tcd.ie/tech-reports/reports.98/TCD-CS-1998-24.pdf
  61. https://hjemmesider.diku.dk/~vinter/cc/DSM.pdf
  62. https://www.eecg.toronto.edu/~stumm/Papers/Stumm-SDPS90.pdf
  63. https://dl.acm.org/doi/10.1145/325096.325102
  64. https://dl.acm.org/doi/10.1145/146628.139676
  65. https://apps.dtic.mil/sti/tr/pdf/ADA264645.pdf
  66. http://infolab.stanford.edu/pub/cstr/reports/csl/tr/95/685/CSL-TR-95-685.pdf
  67. http://rsim.cs.uiuc.edu/Pubs/ps2pdf/hpca2.pdf
  68. https://ieeexplore.ieee.org/document/1236348
  69. https://research.vu.nl/files/74115260/10970
  70. https://www.cs.cornell.edu/raoul/papers/tocs98.pdf
  71. https://www.usenix.org/system/files/conference/nsdi17/nsdi17-gu.pdf
  72. https://www.usenix.org/system/files/conference/nsdi14/nsdi14-paper-dragojevic.pdf
  73. https://cug.org/proceedings/cug2024_proceedings/includes/files/pap142s2-file1.pdf
  74. https://madsys.cs.tsinghua.edu.cn/publication/partial-failure-resilient-memory-management-system-for-cxl-based-distributed-shared-memory/SOSP23-zhang.pdf
  75. https://www.usenix.org/system/files/osdi24-ma-haoran.pdf
Add your contribution
Related Hubs
User Avatar
No comments yet.