Memory-mapped file
Memory-mapped file
Main page
M

Memory-mapped file

logo
Community Hub0 subscribers
Read side by side
Memory-mapped file
View on Wikipedia
from Wikipedia

A memory-mapped file is a segment of virtual memory[1] that has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource. This resource is typically a file that is physically present on disk, but can also be a device, shared memory object, or other resource that an operating system can reference through a file descriptor. Once present, this correlation between the file and the memory space permits applications to treat the mapped portion as if it were primary memory.

History

[edit]

TOPS-20 PMAP

[edit]

An early (c. 1969)[2] implementation of this was the PMAP system call on the DEC-20's TOPS-20 operating system,[3] a feature used by Software House's System-1022 database system.[4]

SunOS 4 mmap

[edit]

SunOS 4[5] introduced Unix's mmap, which permitted programs "to map files into memory."[1]

Windows Growable Memory-Mapped Files (GMMF)

[edit]

Two decades after the release of TOPS-20's PMAP, Windows NT was given Growable Memory-Mapped Files (GMMF).

Since "CreateFileMapping function requires a size to be passed to it" and altering a file's size is not readily accommodated, a GMMF API was developed.[6][7] Use of GMMF requires declaring the maximum to which the file size can grow, but no unused space is wasted.

Benefits

[edit]

The benefit of memory mapping a file is increasing I/O performance, especially when used on large files. For small files, memory-mapped files can result in a waste of slack space[8] as memory maps are always aligned to the page size, which is mostly 4 KiB. Therefore, a 5 KiB file will allocate 8 KiB and thus 3 KiB are wasted. Accessing memory mapped files is faster than using direct read and write operations for two reasons. Firstly, a system call is orders of magnitude slower than a simple change to a program's local memory. Secondly, in most operating systems the memory region mapped actually is the kernel's page cache (file cache), meaning that no copies need to be created in user space.

Certain application-level memory-mapped file operations also perform better than their physical file counterparts. Applications can access and update data in the file directly and in-place, as opposed to seeking from the start of the file or rewriting the entire edited contents to a temporary location. Since the memory-mapped file is handled internally in pages, linear file access (as seen, for example, in flat file data storage or configuration files) requires disk access only when a new page boundary is crossed, and can write larger sections of the file to disk in a single operation.

A possible benefit of memory-mapped files is a "lazy loading", thus using small amounts of RAM even for a very large file. Trying to load the entire contents of a file that is significantly larger than the amount of memory available can cause severe thrashing as the operating system reads from disk into memory and simultaneously writes pages from memory back to disk. Memory-mapping may not only bypass the page file completely, but also allow smaller page-sized sections to be loaded as data is being edited, similarly to demand paging used for programs.

The memory mapping process is handled by the virtual memory manager, which is the same subsystem responsible for dealing with the page file. Memory mapped files are loaded into memory one entire page at a time. The page size is selected by the operating system for maximum performance. Since page file management is one of the most critical elements of a virtual memory system, loading page sized sections of a file into physical memory is typically a very highly optimized system function.[9]

Types

[edit]

There are two types of memory-mapped files:

Persisted

[edit]

Persisted files are associated with a source file on a disk. The data is saved to the source file on the disk once the last process is finished. These memory-mapped files are suitable for working with extremely large source files.[10]

Non-persisted

[edit]

Non-persisted files are not associated with a file on a disk. When the last process has finished working with the file, the data is lost. These files are suitable for creating shared memory for inter-process communications (IPC).[10]

Drawbacks

[edit]

The major reason to choose memory mapped file I/O is performance. Nevertheless, there can be tradeoffs. The standard I/O approach is costly due to system call overhead and memory copying. The memory-mapped approach has its cost in minor page faults—when a block of data is loaded in page cache, but is not yet mapped into the process's virtual memory space. In some circumstances, memory mapped file I/O can be substantially slower than standard file I/O.[11]

Another drawback of memory-mapped files relates to a given architecture's address space: a file larger than the addressable space can have only portions mapped at a time, complicating reading it. For example, a 32-bit architecture such as Intel's IA-32 can only directly address 4 GiB or smaller portions of files. An even smaller amount of addressable space is available to individual programs—typically in the range of 2 to 3 GiB, depending on the operating system kernel. This drawback however is virtually eliminated on modern 64-bit architecture.

mmap also tends to be less scalable than standard means of file I/O, since many operating systems, including Linux, have a cap on the number of cores handling page faults. Extremely fast devices, such as modern NVM Express SSDs, are capable of making the overhead a real concern.[12]

I/O errors on the underlying file (e.g. its removable drive is unplugged or optical media is ejected, disk full when writing, etc.) while accessing its mapped memory are reported to the application as the SIGSEGV/SIGBUS signals on POSIX, and the EXECUTE_IN_PAGE_ERROR structured exception on Windows. All code accessing mapped memory must be prepared to handle these errors, which don't normally occur when accessing memory.

Only hardware architectures with an MMU can support memory-mapped files. On architectures without an MMU, the operating system can copy the entire file into memory when the request to map it is made, but this is extremely wasteful and slow if only a little bit of the file will be accessed, and can only work for files that will fit in available memory.

Common uses

[edit]

Perhaps the most common use for a memory-mapped file is the process loader in most modern operating systems (including Windows and Unix-like systems.) When a process is started, the operating system uses a memory mapped file to bring the executable file, along with any loadable modules, into memory for execution. Most memory-mapping systems use a technique called demand paging, where the file is loaded into physical memory in subsets (one page each), and only when that page is actually referenced.[13] In the specific case of executable files, this permits the OS to selectively load only those portions of a process image that actually need to execute.

Another common use for memory-mapped files is to share memory between multiple processes. In modern protected mode operating systems, processes are generally not permitted to access memory space that is allocated for use by another process. (A program's attempt to do so causes invalid page faults or segmentation violations.) There are a number of techniques available to safely share memory, and memory-mapped file I/O is one of the most popular. Two or more applications can simultaneously map a single physical file into memory and access this memory. For example, the Microsoft Windows operating system provides a mechanism for applications to memory-map a shared segment of the system's page file itself and share data via this section.

Platform support

[edit]

Most modern operating systems or runtime environments support some form of memory-mapped file access. The function mmap(),[14] which creates a mapping of a file given a file descriptor, starting location in the file, and a length, is part of the POSIX specification, so the wide variety of POSIX-compliant systems, such as UNIX, Linux, Mac OS X[15] or OpenVMS, support a common mechanism for memory mapping files. The Microsoft Windows operating systems also support a group of API functions for this purpose, such as CreateFileMapping().[16]

Some free portable implementations of memory-mapped files for Microsoft Windows and POSIX-compliant platforms are:

The Java programming language provides classes and methods to access memory mapped files, such as FileChannel. Furthermore, Java uses memory-mapped approach to load specific classes to decrease the class loading time in JVM - Java Class Data Sharing.[21]

The D programming language supports memory mapped files in its standard library (std.mmfile module).[22].

Ruby has a gem (library) called Mmap, which implements memory-mapped file objects.

Rust does not provide any mmap functionality in the standard library but there exists a third-party crate (library) called memmap2.[23]

Since version 1.6, Python has included a mmap module in its Standard Library.[24] Details of the module vary according to whether the host platform is Windows or Unix-like.

For Perl there are several modules available for memory mapping files on the CPAN, such as Sys::Mmap[25] and File::Map.[26]

In the Microsoft .NET runtime, P/Invoke can be used to use memory mapped files directly through the Windows API. Managed access (P/Invoke not necessary) to memory mapped files was introduced in version 4 of the runtime (see Memory-Mapped Files). For previous versions, there are third-party libraries which provide managed API's.[27] .NET have the MemoryMappedFile class.[28][29]

PHP supported memory-mapping techniques in a number of native file access functions such as file_get_contents() but has removed this in 5.3 (see revision log).

For the R programming language there exists a library on CRAN called bigmemory which uses the Boost library and provides memory-mapped backed arrays directly in R. The package ff offers memory-mapped vectors, matrices, arrays and data frames.

The J programming language has supported memory-mapped files since at least 2005. It includes support for boxed array data, and single datatype files. Support can be loaded from 'data/jmf' J's Jdb and JD database engines use memory-mapped files for column stores.

The Julia programming language has support for I/O of memory-mapped binary files through the Mmap module within the Standard Library.[30]

References

[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Memory-mapped file

View on Grokipedia
from Grokipedia
A memory-mapped file is a mechanism in operating systems that associates the contents of a file or portion thereof with a segment of a process's virtual address space, enabling the file data to be accessed directly as if it were resident in memory.[1] This mapping is achieved through system calls or APIs, such as mmap() in Unix-like systems, which create a virtual memory region backed by the file on disk, allowing read and write operations via standard memory pointers without explicit I/O system calls.[2] In practice, the operating system handles paging between disk and physical memory transparently, loading only the accessed portions into RAM on demand.[3] Memory-mapped files offer significant advantages for handling large datasets, as they avoid the overhead of loading entire files into memory or managing buffers manually, making them ideal for applications like databases, image processing, and high-performance computing.[4] They support both shared and private mappings: shared mappings (MAP_SHARED in Linux) allow multiple processes to access and modify the same file region synchronously, with changes persisted to disk, while private mappings (MAP_PRIVATE) provide copy-on-write semantics for isolated modifications without affecting the underlying file.[2] This duality enables efficient inter-process communication (IPC) and data sharing, as seen in scenarios where processes collaborate on common resources like executable libraries or configuration files.[1] The concept is natively supported across major operating systems, including Windows via the Win32 API functions like CreateFileMapping and MapViewOfFile, and Unix variants through POSIX-compliant mmap.[1] Benefits include reduced I/O latency, simplified programming by treating files as arrays, and optimized resource usage for files exceeding available RAM, though limitations such as page alignment requirements and potential synchronization issues in multi-process environments must be managed.[3] Overall, memory-mapped files represent a foundational technique for bridging persistent storage and volatile memory in modern software systems.

Fundamentals

Definition and Core Concept

A memory-mapped file is a technique that associates a portion or the entirety of a file's contents with a segment of a process's virtual memory address space, enabling the operating system to manage file accesses transparently as if the data were resident in physical memory. This approach leverages the operating system's virtual memory subsystem to treat disk-based files as an extension of RAM, facilitating seamless integration between persistent storage and application memory.[5] At its core, the concept involves using system calls to establish a direct correspondence between file offsets and virtual addresses, allowing programs to perform reads and writes via standard memory operations rather than dedicated I/O functions like read() or write(). For instance, modifications to the mapped memory region are automatically propagated back to the underlying file (in shared mappings), or buffered for later synchronization, depending on the mapping flags specified. This abstraction simplifies file handling by eliminating the need for explicit buffer management and I/O synchronization in many scenarios.[5][5] Memory-mapped files depend on foundational virtual memory features, such as paging, where memory is organized into fixed-size pages (typically 4 KB). Access to an unmapped or unloaded page triggers a page fault exception, which the operating system's kernel handles by fetching the relevant data from the file into a physical page frame and updating the page table to reflect the mapping. This demand-paging mechanism ensures that only actively accessed portions of the file consume physical memory, optimizing resource usage for large or sparsely accessed files.[6] A basic example of setting up a memory mapping in C, using the POSIX mmap() function, illustrates this process: 
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int fd = open("example.txt", O_RDWR);  // Open [file descriptor](/page/File_descriptor)
struct stat sb;
fstat(fd, &sb);
void *addr = mmap(NULL, sb.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
// Now 'addr' points to the mapped file content; access via pointers
// Unmap when done: munmap(addr, sb.st_size);
This code opens a file, determines its size, and maps the entire file into the process's address space starting at an OS-chosen address (NULL), with shared permissions allowing changes to persist to disk.[5][2]

Mapping Mechanism

To map a file into a process's virtual address space, an application first opens the file using a system call such as open() in POSIX-compliant systems, obtaining a file descriptor that references the file on disk. The mapping is then established by invoking the mmap() system call, which takes parameters including the desired starting address (often specified as 0 to let the kernel choose), the length of the mapping in bytes, protection flags (e.g., PROT_READ for read access or PROT_READ | PROT_WRITE for read-write access), mapping flags (such as MAP_SHARED or MAP_PRIVATE), the file descriptor, and an offset into the file where the mapping begins.[7] These parameters define the portion of the file to map and the behavior of accesses to that region, ensuring alignment with page boundaries (typically 4 KB) for efficient kernel handling.[2] Upon invocation of mmap(), the operating system kernel creates a new virtual memory region in the process's address space, associating it with the specified file segment without immediately loading the data into physical RAM.[7] This region is backed by the file on disk, and the kernel updates the process's page tables to mark the virtual pages as valid but not yet resident in memory. When the process first accesses a byte within the mapped region—such as reading or writing—it triggers a page fault because the corresponding physical page is absent. The kernel's page fault handler then intervenes via demand paging: it allocates a physical page if needed, reads the relevant file data from disk into the kernel's page cache, and maps that page into the process's virtual address space, resolving the fault and allowing the access to proceed transparently.[2] Subsequent accesses to the same page hit the cache or physical memory, avoiding further disk I/O until the page is evicted under memory pressure. The mapping flags control how modifications interact with the underlying file and other processes. With MAP_SHARED, changes made by the process to the mapped memory are propagated back to the file on disk and visible to other processes sharing the same mapping, enabling inter-process communication or persistent updates.[7] In contrast, MAP_PRIVATE uses a copy-on-write mechanism: initial reads reflect the file's content, but any write operation creates a private copy of the page for the process, isolating modifications from the file and other sharers to prevent unintended side effects.[2] Protection flags enforce access controls at the hardware level through page table entries, raising segmentation faults for violations like writing to a read-only mapping. The mmap() call can fail under various conditions, returning a null pointer (or (void *)-1 in POSIX) and setting an error code via errno. Common failures include ENOMEM when the system lacks sufficient virtual or physical memory to establish the mapping, EINVAL for invalid parameters such as a non-page-aligned offset or length exceeding the file size, or ENODEV if the file descriptor does not support mapping (e.g., certain special files).[7] Applications must check the return value to handle these errors gracefully, often falling back to traditional file I/O methods.[2]

Historical Development

Early Systems and PMAP

The TOPS-20 operating system, developed by Digital Equipment Corporation (DEC) in the 1970s for the PDP-10 mainframe, introduced memory-mapped file capabilities through its PMAP monitor call, enabling efficient access to file contents by integrating them directly into a process's virtual address space.[8] Released in early 1976 as part of the initial TOPS-20 distribution, this feature built on prior TENEX innovations and addressed the demands of 36-bit computing environments where physical memory was limited to hundreds of kilobytes.[9] The PMAP call (JSYS #56) allowed users to map one or more complete pages from a disk file into a process's memory for input, from a process to a file for output, or between processes, without the overhead of explicit data copying.[10] Key features of PMAP included dynamic mapping of files as either executable code or data, supporting read, write, or execute access modes based on file attributes established via the OPENF call. In timesharing systems like TOPS-20, which supported dozens of concurrent users on a single PDP-10, PMAP facilitated efficient program loading by mapping executable or save files (such as SSAVE or SAVE formats) directly into virgin processes, bypassing traditional read protections for execute-only files when combined with the GET monitor call. This approach enabled random access to file pages beyond EOF limits and supported page-mode I/O, where pages could be preloaded into physical memory or linked with copy-on-write semantics to minimize resource use.[10] Unmapping or deleting pages was also handled via PMAP, ensuring processes could release resources cleanly before file closure. PMAP demonstrated significant benefits for accessing large files in the resource-constrained era of 1970s mainframes, where swapping entire programs into limited RAM was inefficient for multi-user workloads. By treating files as extensions of virtual memory, it reduced I/O latency and memory fragmentation in timesharing scenarios, influencing later systems' approaches to virtual memory management. For instance, in PDP-10 environments with up to 384K words of physical memory divided into 512-word pages, PMAP's ability to map specific page ranges or entire files streamlined data sharing and process communication without redundant copies.[8][10]

Unix Implementations

SunOS 4, released by Sun Microsystems in December 1988, marked the first widespread implementation of the mmap() system call in a Unix operating system, enabling programs to map files directly into their virtual address space for efficient access. This feature was part of a comprehensive virtual memory overhaul, as described in the seminal 1987 USENIX paper "Virtual Memory Architecture in SunOS" by Robert A. Gingell, Joseph P. Moran, and William A. Shannon, which outlined the segment-based mapping mechanism using drivers like seg_vn for file-backed segments.[11] The mmap() call supported both shared and private (copy-on-write) mappings at page granularity, allowing seamless integration of file I/O with memory management and reducing the need for explicit read/write operations. The adoption of mmap() in Unix-like systems drew significant influence from the Berkeley Software Distribution (BSD), where it was first documented in 4.2BSD (1983) but fully implemented in 4.4BSD (1993), providing a foundation for memory-mapped I/O in academic and research environments. SunOS 4 itself was derived from BSD, incorporating these concepts into a production-ready system. Concurrently, mmap() was integrated into UNIX System V Release 4 (SVR4), released in 1988 by AT&T, which extended the interface to support mapping of general objects into address spaces via new system calls like mmap(2) and munmap(2), as detailed in the process model enhancements for /proc. This convergence between BSD and System V lineages facilitated broader compatibility. Standardization efforts culminated in the inclusion of mmap() in the POSIX.1-2001 specification, defining a portable interface for mapping files, shared memory objects, or typed memory into a process's address space across Unix variants, with flags for shared (MAP_SHARED) and private (MAP_PRIVATE) behaviors.[7] POSIX ensured interoperability by specifying error conditions, protection modes (e.g., PROT_READ, PROT_WRITE), and advice parameters (e.g., MAP_SHARED for synchronization), building on Unix implementations to promote application portability. A key benefit of mmap() in these Unix systems was its support for zero-copy I/O, where file data is accessed directly via memory references without user-kernel data copies or additional system calls for reads/writes, thereby minimizing context switches and overhead in data-intensive applications. This capability extended to handling shared memory segments, allowing multiple processes to map the same file or anonymous region (via MAP_ANON in later extensions) for inter-process communication, with the kernel managing page faults and coherency through the virtual memory subsystem.

Windows and Other OS Evolutions

Memory-mapped files, known as general memory-mapped files (GMMF) in Windows, were introduced with Windows NT in 1993, providing robust support for mapping files into virtual address spaces through the Win32 API functions CreateFileMapping and MapViewOfFile. These APIs enable the creation of file-mapping objects that can represent files larger than physical memory, with dynamic expansion capabilities by specifying a maximum size exceeding the current file length in the CreateFileMapping call; upon writing to the extended region, the underlying file grows accordingly.[12] The evolution of memory mapping in Windows addressed significant limitations in earlier systems like MS-DOS and 16-bit Windows, which lacked virtual memory management and relied on rudimentary shared global memory blocks via flags such as GMEM_SHARE, restricting scalability and inter-process sharing. Full implementation arrived with the Win32 subsystem in Windows NT, leveraging kernel-level section objects—also called file-mapping objects—to facilitate shared mappings across processes, where multiple views of the same section can be mapped into different address spaces for efficient data interchange without explicit copying.[12][13] A key feature in Windows is the ability to map views beyond the current file size, particularly for sparse files on NTFS, where unaccessed regions are not allocated on disk until written to, enabling efficient handling of large, irregularly accessed files by treating gaps as zero-filled virtual space.[14][15] Beyond Windows, memory mapping concepts persisted in other systems, such as OpenVMS—a successor to the TOPS-20 operating system—which employed global sections to map files or pageable memory into shared virtual address spaces, supporting both private and global access modes for process communication and file-backed paging.[16] Early Linux kernels adopted similar functionality in the early 1990s through the mmap system call, with initial support appearing in kernel version 0.98.2 in 1992 and improving in subsequent releases to provide POSIX-compliant file and anonymous mappings.[2] This development paralleled Unix implementations like mmap, extending memory mapping to open-source Unix-like environments.

Advantages

Performance Enhancements

Memory-mapped files enable zero-copy I/O by directly mapping file contents into a process's virtual address space, eliminating the data copies between kernel buffers and user-space buffers that occur in traditional read or write system calls. This reduces CPU overhead from memory copying and context switches, allowing applications to treat file access as simple memory operations.[17] Demand paging further enhances efficiency in memory-mapped files, as the operating system loads only the pages accessed by the application on demand, rather than pre-loading the entire file into memory. This lazy loading minimizes initial startup time and memory usage for large files, where only relevant portions are brought into RAM via page faults, optimizing resource allocation for sparse or sequential access patterns.[17] Integration with the OS page cache provides additional performance gains, as mapped file regions leverage the system's unified caching mechanism, keeping frequently accessed data in physical memory for subsequent reads without disk I/O. Benchmarks demonstrate these benefits: for cached sequential reads, memory mapping can achieve up to 3x higher bandwidth (e.g., 200 MB/s vs. 62 MB/s for traditional file reads on certain Unix systems) compared to standard read calls, though results vary by workload and hardware due to factors like page fault handling. In microbenchmarks on 4 GB files, optimized memory mapping reduces I/O time to approximately 1.02 seconds, comparable to or slightly better than read calls at 1.06 seconds, while default implementations may incur higher latency from unoptimized paging.[18][17]

Programming Simplicity

Memory-mapped files simplify programming by abstracting file access into direct memory operations, allowing developers to treat the file as a contiguous array in the process's address space and use standard pointer arithmetic or array indexing for reading and writing data.[19] This eliminates the need for repetitive system calls like lseek(), read(), or write(), which are required in traditional file I/O to navigate and transfer data in chunks.[20] Instead of managing offsets and buffer sizes manually, programmers can operate on the mapped region as if it were native memory, streamlining code for tasks involving large or sequentially accessed files.[2] This abstraction also reduces common sources of errors, such as buffer overflows, misalignment issues, or incorrect offset calculations, by offloading data transfer and paging to the operating system kernel.[21] Synchronization challenges related to explicit I/O buffers are minimized, as the mapped memory integrates seamlessly with the program's existing memory model without requiring custom allocation or deallocation logic.[19] To illustrate, consider processing a large file in pseudocode. With traditional I/O using fread(), the code involves opening the file, allocating a buffer, and looping over reads while handling partial reads and errors: 
FILE *fp = fopen("data.bin", "rb");
char buffer[BUFSIZ];
size_t bytes_read;
while ((bytes_read = fread(buffer, 1, BUFSIZ, fp)) > 0) {
    // Process buffer [data](/page/Data), e.g., for (size_t i = 0; i < bytes_read; i++) { process(buffer[i]); }
    // Handle potential errors or [end-of-file](/page/End-of-file)
}
fclose(fp);
free(buffer);  // If dynamically allocated
In contrast, using mmap() maps the entire file (or a portion) into memory, enabling direct indexing without loops for sequential access or buffer management: 
int fd = open("data.bin", O_RDONLY);
off_t length = lseek(fd, [0](/page/0), SEEK_END);
void *ptr = mmap(NULL, length, PROT_READ, MAP_SHARED, fd, [0](/page/0));
if (ptr != MAP_FAILED) {
    // [Process](/page/Process) directly, e.g., for (off_t i = [0](/page/0); i < length; i++) { process(((char*)ptr)[i]); }
    munmap(ptr, length);
}
close(fd);
This mapping approach requires fewer lines of code and avoids explicit error-prone buffer handling.[2] For multi-threaded applications, memory-mapped files further enhance simplicity by providing inherent shared access to the mapped region across threads within the same process, as threads naturally share the virtual address space.[21] This allows concurrent reads or coordinated writes without the overhead of inter-thread communication primitives solely for I/O, treating the file-backed memory as a unified data structure visible to all threads.[19]

Types of Mappings

Persistent Mappings

Persistent mappings in memory-mapped files refer to configurations where the mapped memory region is shared and modifications directly affect the underlying file on disk, ensuring durability beyond the lifetime of the mapping process. These mappings are typically established using flags such as MAP_SHARED in Unix-like systems, which allow updates to the mapped area to be visible to other processes accessing the same file and to propagate changes to the persistent storage.[2] In Windows environments, equivalent functionality is achieved through file-backed memory-mapped files created via APIs like CreateFileMapping or the .NET MemoryMappedFile.CreateFromFile, where the mapping is tied to an existing file handle.[22] The behavior of persistent mappings ensures that writes to the mapped memory are eventually flushed to the disk file, providing a mechanism for long-term data storage. In Unix systems, changes are carried through to the file automatically under MAP_SHARED, but precise control over synchronization—such as immediate or asynchronous flushing—is managed via the msync() system call to guarantee data integrity before unmapping or process exit.[2] Similarly, in Windows, modifications to the mapped view update the source file upon closure of the last referencing process, without requiring explicit flushing in many cases, though APIs like FlushViewOfFile can enforce immediate persistence.[22] This design makes persistent mappings suitable for scenarios requiring durable storage, as the file remains updated even after process termination or system restarts, provided synchronization has been properly invoked to avoid data loss from caching.[2] Persistent mappings are commonly employed for applications needing reliable, file-based persistence, such as database engines that treat large datasets as mappable files for efficient querying and updates, or system configuration stores that maintain state across sessions.[23] For instance, storage systems leveraging memory-mapped files for object persistence benefit from the unified memory-file interface, enabling seamless data durability in distributed environments. These use cases highlight their role in ensuring data integrity over process lifecycles, with proper syncing preventing inconsistencies during failures.[24]

Non-Persistent Mappings

Non-persistent mappings, often referred to as private mappings, involve creating a virtual memory region that maps to a file but does not propagate modifications back to the underlying file system. In Unix-like operating systems, these are typically established using the MAP_PRIVATE flag with the mmap() system call, which implements a copy-on-write (COW) mechanism: initial reads access the file directly, but any write operation triggers the creation of a private copy of the affected page in the process's address space, leaving the original file unchanged.[2] This approach ensures that updates remain isolated to the mapping and are not visible to other processes or persisted to disk.[5] The behavior of non-persistent mappings makes them particularly suitable for scenarios where read-only access or temporary in-memory edits are required without risking alteration of the source file, such as in data analysis tools that parse large datasets for processing or validation. For instance, a program might map a configuration file privately to experiment with modifications in memory before deciding whether to save changes separately. Unlike shared mappings, which synchronize changes across processes and to the file, private mappings prioritize isolation to prevent unintended side effects.[2] A key limitation of non-persistent mappings is the absence of automatic file synchronization; any changes made during the mapping's lifetime are discarded upon unmapping with munmap(), with no option to flush them to the original file without additional explicit handling. This design enforces their temporary nature but requires developers to manage persistence through alternative means if needed. In Windows, equivalent functionality is provided via the MapViewOfFile() function with the FILE_MAP_COPY access right, which also employs copy-on-write semantics: writes result in private page copies that do not affect the mapped file, ensuring the original remains unmodified.[25] These mappings are commonly used in applications needing process-specific isolation, such as secure parsing of sensitive files where source integrity must be preserved.[5]

Disadvantages

Memory and Resource Overhead

Memory-mapped files impose significant memory pressure on a process because the entire mapped region is reserved in the process's virtual address space, regardless of whether all pages are physically resident in RAM. This reservation counts toward the process's virtual memory limit, potentially leading to failures when creating additional mappings if the limit is exceeded, as enforced by mechanisms like RLIMIT_DATA on Linux systems. For instance, on 32-bit architectures, the total virtual address space is constrained to around 4 GB, limiting the aggregate size of all mappings. Even with demand paging, where pages are loaded only on access, the upfront reservation can fragment the address space and complicate memory management for applications handling multiple large files. Large memory mappings exacerbate risks of swapping and thrashing in environments with insufficient physical memory. When RAM is oversubscribed, the operating system may evict frequently accessed pages from memory-mapped regions to disk, causing excessive page faults and I/O operations that degrade performance. This thrashing occurs as the system spends more time managing virtual memory than executing application code, particularly for mappings exceeding available RAM, such as multi-gigabyte files in data processing workloads. To mitigate this, some systems allow flags like MAP_NORESERVE on Unix-like OSes to avoid pre-reserving swap space, though this increases the risk of segmentation faults during writes if memory is unavailable. Each memory mapping consumes kernel resources, including virtual memory areas (VMAs) that track the mapping in the process's address space. On Linux, the number of VMAs is limited by /proc/sys/vm/max_map_count, defaulting to 65,536, beyond which new mappings fail with ENOMEM; excessive mappings can thus exhaust this quota even if physical memory is available. Additionally, creating a mapping requires an open file descriptor, which, while closable post-mapping without invalidating the region, still incurs temporary resource use and contributes to per-process open file limits. Due to page-level granularity—typically 4 KB on x86 systems—memory mappings introduce overhead for small files, where the last incomplete page results in slack space allocation. For example, mapping a 1 KB file reserves a full 4 KB page in virtual memory, wasting 3 KB per such mapping, which accumulates in applications processing many tiny files like metadata indexes. This alignment requirement stems from hardware page sizes and ensures efficient translation but amplifies inefficiency for non-page-aligned data sizes.

Portability and Compatibility Issues

Memory-mapped files exhibit significant API variations across operating systems, complicating portability. In POSIX-compliant systems, the mmap function provides a single system call to map a file into memory, specified by parameters including address hint, length, protection modes (e.g., PROT_READ, PROT_WRITE), flags (e.g., MAP_SHARED for shared modifications or MAP_PRIVATE for copy-on-write), file descriptor, and offset.[7] Conversely, Windows employs a two-step process: first creating a file mapping object with CreateFileMapping, then mapping a view using MapViewOfFile, which takes a handle to the mapping object, desired access (e.g., FILE_MAP_READ, FILE_MAP_WRITE), offset components, and byte count, but lacks direct equivalents to POSIX flags like MAP_SHARED, instead achieving sharing through the mapping handle passed between processes.[25] These differences in invocation, parameters, and semantics—such as POSIX's optional support for MAP_FIXED versus Windows' granularity requirements—require conditional compilation or abstraction layers for cross-platform code.[7][25] Operating system limitations further hinder compatibility, particularly in older or constrained environments. While mmap has been supported in Linux since kernel version 0.98.2 in 1992, 32-bit systems impose strict file size constraints on mappings, often limited to 2 GB due to virtual address space boundaries, necessitating special handling like open64 for larger files in POSIX environments.[26][27] Similarly, Windows on 32-bit architectures restricts individual memory-mapped views to 2 GB, even if the underlying file exceeds this, requiring multiple views for larger data sets.[22] These address space limitations persist in legacy 32-bit deployments, contrasting with 64-bit systems where mappings can span terabytes, but demand careful size management to avoid failures.[22] Security concerns arise from memory mapping's interaction with system policies, potentially exposing vulnerabilities. In Linux, overcommitment—enabled by default via /proc/sys/vm/overcommit_memory=0—allows mmap to allocate virtual memory beyond physical availability, assuming delayed usage; however, upon page access, if memory is exhausted, the OOM killer may terminate processes based on a badness score factoring usage and adjustability, risking unintended data loss or denial-of-service.[28] Access control relies on underlying file permissions: POSIX mmap requires the file descriptor to be opened with at least read access, and write protections (PROT_WRITE) demand matching file write permissions, enforcing discretionary access control (DAC) without additional mapping-specific ACLs.[7] On Windows, mappings inherit file permissions but use dedicated security descriptors on the file mapping object, supporting granular rights like FILE_MAP_READ or FILE_MAP_WRITE via ACLs, audited through functions such as SetSecurityInfo.[29] To mitigate these portability issues, libraries provide abstraction layers. Boost.Interprocess, for instance, emulates portable shared memory using memory-mapped files, unifying POSIX mmap and Windows CreateFileMapping/MapViewOfFile interfaces to enable cross-platform interprocess communication without direct API exposure.[30] This approach ensures consistent behavior for mappings, handling flag equivalents and error conditions transparently across Unix-like and Windows systems.

Applications

File Handling and I/O

Memory-mapped files facilitate efficient sequential access to files by mapping the file content into virtual memory, allowing applications to use pointer arithmetic or array-like indexing instead of explicit seek operations in traditional I/O APIs. This is particularly beneficial for processing log files or streaming data, where data is accessed in a linear fashion, minimizing the overhead of repeated positioning calls. In R, for instance, the mmap package enables sequential subsetting of mapped files as native vectors, achieving high throughput through OS-managed paging and reducing garbage collection compared to loading data fully into memory.[31] For random access patterns, memory-mapped files provide direct byte-level addressing, treating the file as a contiguous memory block for non-sequential reads and writes, which is ideal for binary files requiring scattered access. This approach avoids the latency of seek and read system calls for each operation, as the OS handles paging transparently. In .NET, random access views created via CreateViewAccessor support this by allowing byte-level modifications to persisted files without buffering the entire content.[22] Handling large files represents a key strength of memory-mapped I/O, enabling operations on terabyte-scale datasets without loading them entirely into RAM, as only accessed pages are brought into physical memory by the OS. Sparse mappings further optimize this for files with irregular access, where untouched regions consume no resources; for example, Java implementations can map 8 TiB virtual files using just megabytes of RAM and disk for sparse writes. On 64-bit systems, such mappings can extend to 256 TB, supporting applications that process massive binary data streams efficiently.[32][4] In image processing, memory-mapped files allow direct manipulation of pixel data in binary image formats, such as BMP, by mapping the file and accessing RGB values at arbitrary offsets for operations like color adjustment, avoiding the need to allocate full in-memory buffers for high-resolution images. This technique leverages random access views to brighten or modify specific regions, with the OS ensuring data integrity during writes. Memory mapping enhances performance in these scenarios by eliminating explicit I/O calls after initial setup, relying on virtual memory mechanisms for efficient caching.[22]

Inter-Process Communication

Memory-mapped files enable inter-process communication (IPC) by allowing multiple processes to map the same region of memory, facilitating efficient data exchange without explicit copying. In POSIX systems, processes can create a shared memory object using shm_open(), which returns a file descriptor to a named object that serves as a handle for mapping the same memory region via mmap() with the MAP_SHARED flag. This setup permits unrelated processes to access the shared segment concurrently, where modifications by one process are visible to others, provided proper synchronization is employed.[33][34] For pure IPC without a persistent backing file, anonymous mappings can be used, particularly in scenarios where data does not need to survive process termination. In POSIX-compliant systems, mmap() with the MAP_ANONYMOUS and MAP_SHARED flags allocates a shared memory region directly, bypassing file descriptors entirely; this is suitable for related processes (e.g., parent-child after fork()) but requires naming mechanisms like shm_open() for unrelated processes to join the segment. Such mappings avoid disk I/O overhead, making them ideal for high-speed data transfer in temporary collaborations.[35][2] Synchronization is essential in shared mappings to prevent race conditions, as concurrent access can lead to data corruption without coordination. POSIX provides process-shared mutexes via pthread_mutex_init() with the PTHREAD_PROCESS_SHARED attribute, placing the mutex in the shared memory region to enforce mutual exclusion across processes. Alternatively, named semaphores created with sem_open() can signal availability and control access, ensuring atomic operations on shared data. These primitives must be explicitly managed, as the operating system does not provide automatic barriers for memory-mapped regions.[36] A common application is the producer-consumer pattern in client-server architectures, where a producer process writes data to the shared mapping while a consumer reads it, using semaphores to manage buffer fullness and emptiness. For instance, in a bounded buffer implementation, semaphores mutex (initialized to 1 for exclusion), full (0 for empty slots), and empty (buffer size for available slots) coordinate access: the producer waits on empty and mutex before adding an item and signals full, while the consumer reverses the order to avoid deadlock. This pattern leverages shared memory for low-latency exchange, as seen in systems handling real-time data streams.[37][38]

Database and Large Data Processing

In database engines such as SQLite, memory-mapped files enable memory-like access to on-disk indexes by directly mapping database pages into the process's virtual address space, allowing queries to fetch data without explicit read system calls and kernel-user space copies.[39] This approach is particularly beneficial for I/O-intensive read operations, where the operating system's page cache handles paging transparently, reducing overhead for index lookups and scans.[39] Similarly, LevelDB utilizes memory-mapped files for its immutable sorted string tables (SSTables), mapping these files to improve random read performance by leveraging the OS page cache for frequent key-value lookups without loading entire files into RAM.[40] In big data processing frameworks like Apache Spark, memory-mapped files facilitate efficient handling of large datasets by mapping input blocks from disk into memory during reads, avoiding unnecessary copies and enabling columnar storage formats such as Parquet to process petabyte-scale data without fully loading it into heap memory.[41] This integration supports distributed query execution on clusters, where mapped files allow executors to access data lazily, minimizing memory pressure for transformations and aggregations on massive datasets.[41] Modern NoSQL systems extend memory-mapped techniques through engines like RocksDB, which supports memory-mapped indexes by mmapping entire SSTables for reads via the allow_mmap_reads option, enabling efficient access to on-disk data structures without full in-memory loading even at petabyte scales.[42] This is crucial for read-heavy workloads in distributed databases, where mapped indexes reduce the need to cache all metadata in RAM while maintaining low-latency point lookups and range scans.[42]

Platform Support

Unix-like Operating Systems

In Unix-like operating systems, memory-mapped files are primarily supported through the POSIX standard, which defines the core system calls for mapping files into a process's virtual address space. The mmap() function establishes a mapping between a process's address space and a file or device, allowing direct memory access to file contents without explicit read or write system calls.[5] This mapping is specified by parameters including the starting address (addr), length (len), protection flags such as PROT_READ for read-only access or PROT_WRITE for read-write access, mapping flags like MAP_SHARED for shared changes or MAP_PRIVATE for private copies, and an offset into the file.[5] The munmap() function unmaps the region previously established by mmap(), releasing the virtual address space, while msync() synchronizes the mapped memory with the underlying file, ensuring changes are written to disk if the MS_SYNC flag is used.[5] These APIs are implemented consistently across Linux, macOS, and BSD variants, with mmap() returning a pointer to the mapped area or MAP_FAILED on error.[2] Linux extends these POSIX interfaces with advanced kernel features for optimizing memory mappings. Support for huge pages, also known as Huge TLB pages, allows mappings larger than the standard 4 KiB page size—typically 2 MiB or 1 GiB—to reduce translation lookaside buffer (TLB) overhead and improve performance for large files.[43] Applications can request huge page mappings by specifying the MAP_HUGETLB flag in mmap(), provided the kernel is configured with huge page support via boot parameters like hugepagesz=2M.[2] Additionally, the madvise() system call enables processes to provide hints to the kernel about expected access patterns for mapped regions, such as MADV_WILLNEED to prefetch pages or MADV_SEQUENTIAL for linear access, which can enhance caching and paging efficiency.[44] Despite these capabilities, memory mappings in Unix-like systems have limitations tied to file system support. For instance, mappings on Network File System (NFS) mounts may fail or behave inconsistently if the mount lacks proper options like noac or if the NFS version does not fully support coherent caching, as the kernel cannot guarantee atomic updates across network shares.[2] Regular files must support seeking to arbitrary offsets, and the offset parameter for the mapping must be a multiple of the page size.[5] Enhancements in Linux kernel versions 5.x and later, including the 6.x series as of 2025, have improved transparent huge pages (THP) for memory mappings, building on their introduction in kernel 2.6.38. THP automatically promotes contiguous base pages to 2 MiB huge pages during allocation or faulting, with optimizations in 5.x series including better collapse heuristics via khugepaged and support for file-backed mappings through madvise hints like MADV_HUGEPAGE.[45] These updates, such as refined scanning in kernel 5.0 and later, reduce allocation latency and memory fragmentation for mmap-based workloads without requiring explicit huge page configuration.[45] In macOS and BSD systems, while POSIX compliance ensures core functionality, huge page support is more limited, often relying on standard page sizes without the automatic THP mechanisms found in Linux.[46]

Windows Operating Systems

In Windows operating systems, memory-mapped files are implemented through the Windows API, which provides functions to create, map, and manage file mapping objects, also known as section objects, for associating file contents with virtual memory.[1] These section objects maintain the association between a file and a view of its data in process address space, enabling efficient sharing and access across processes.[1] The primary API for creating a file mapping object is CreateFileMapping, which takes a handle to a file (or INVALID_HANDLE_VALUE for pagefile-backed mappings) and specifies the mapping size, protection attributes, and an optional name for sharing.[47] This function returns a handle to the section object, which can be used by multiple processes for inter-process communication if named.[48] To access the mapped data, MapViewOfFile is called with the section handle, defining the offset and length of the view to map into the calling process's virtual address space.[25] Views can be mapped with various access protections, such as read-only or read-write, and the system handles paging automatically.[25] When finished, UnmapViewOfFile releases the view from the address space, and the section handle is closed with CloseHandle to decrement its reference count.[49] Security for file mapping objects is managed through security descriptors specified in the SECURITY_ATTRIBUTES structure passed to CreateFileMapping.[29] These descriptors define access control lists (ACLs) that control permissions, such as read, write, or execute, for processes attempting to open or map the section; by default, ACLs derive from the creator's token.[50] Additional functions like SetNamedSecurityInfo allow modifying these descriptors post-creation to enforce granular access rights, including standard rights like DELETE or WRITE_DAC.[29] Windows supports memory-mapped executables by loading Portable Executable (PE) files into process address space using section objects, where the loader creates mappings for code, data, and resource sections without requiring explicit file opens.[51] This mechanism ensures that executable images are efficiently paged and shared, with sections aligned to page boundaries for optimal performance.[52] Full support for these APIs has been available since Windows NT 3.1, with enhancements and stability in features like large-page support introduced in Windows 2000 and later.[50] In modern Windows 10 and later, Universal Windows Platform (UWP) applications face restrictions: file mappings are by default limited to processes within the same package, requiring full-trust capabilities for broader inter-process sharing.[53] For scenarios not involving files, such as reserving large virtual address spaces without immediate physical or pagefile backing, VirtualAlloc with the MEM_RESERVE flag can allocate regions up to the process's virtual limit, allowing subsequent commits or mappings as needed.[54]

Other Platforms and Libraries

In embedded systems and real-time operating systems (RTOS), memory-mapped files are supported through specialized mechanisms rather than standard POSIX interfaces. For instance, FreeRTOS provides memory mapping capabilities via hardware-specific ports and linker configurations that allow direct access to memory regions, though it lacks native mmap support due to its lightweight design for microcontrollers without full virtual memory management.[55][56] On Android, which serves as an embedded Linux variant, the ashmem (Android Shared Memory) allocator enables anonymous shared memory regions that can be mapped into process address spaces using mmap, facilitating efficient inter-process communication and data sharing while allowing the kernel to reclaim memory under pressure.[57][58][59] Cross-platform libraries abstract memory-mapped file operations to ensure portability across operating systems. Python's mmap module provides a high-level interface for mapping files into memory, treating them as mutable byte arrays or file-like objects, which leverages the underlying OS's mmap system call for efficient I/O on supported platforms.[60] Similarly, Java's New I/O (NIO) API includes MappedByteBuffer, a direct byte buffer that maps a file region into memory via FileChannel.map, enabling random access and modifications that are automatically synchronized back to the file.[61] These libraries promote conceptual uniformity, allowing developers to handle large files without loading them entirely into RAM, though they inherit platform-specific behaviors for mapping modes like read-only or read-write.[62] On other operating systems, memory-mapped files integrate with unique protocols or face security-imposed limitations. Plan 9 from Bell Labs uses the 9P protocol for distributed file access, where clients can map remote files into local memory spaces, treating network resources as local files for seamless mapping operations.[63][64] In iOS, memory-mapped files are restricted for security reasons, with the operating system enforcing protections against executable or writable mappings outside designated regions to prevent exploits, and limiting virtual memory usage to conserve resources on mobile devices.[65][66] Recent developments in WebAssembly runtimes have explored memory-mapped interfaces through the WebAssembly System Interface (WASI). Since 2020, proposals like an MVP for mmap in WASI aim to enable file mappings within sandboxed WebAssembly modules, allowing portable access to host file systems without direct OS calls, though implementations remain experimental and focused on emulated behaviors for compatibility.[67]

References

  1. File Mapping - Win32 apps - Microsoft Learn
  2. mmap(2) - Linux manual page - man7.org
  3. Understanding memory mapping - IBM
  4. Overview of Memory-Mapping - MATLAB & Simulink - MathWorks
  5. mmap
  6. [PDF] Virtual Memory - Carnegie Mellon University
  7. mmap
  8. Origins and Development of TOPS-20 - OPOST.COM Home
  9. TOPS-20 - Computer History Wiki
  10. [PDF] TOPS-20 Monitor Calls Reference Manual - Bitsavers.org
  11. [PDF] The Process File System and Process Model in UNIX System V
  12. [PDF] Memory-Mapped Files in Windows NT Simplify File ... - Jacob Filipp
  13. Section Objects and Views - Windows drivers - Microsoft Learn
  14. Creating a File Mapping Object - Win32 apps | Microsoft Learn
  15. Sparse Files - Win32 apps - Microsoft Learn
  16. Windows NT and VMS: The Rest of the Story - ITPro Today
  17. [PDF] Efficient Memory Mapped File I/O for In-Memory File Systems | USENIX
  18. [PDF] lmbench: Portable Tools for Performance Analysis - USENIX
  19. [PDF] OSPP: Address Translation
  20. COS 318: Operating Systems Virtual Memory Design ... - cs.Princeton
  21. 3.4. Shared Memory With Memory-mapped Files
  22. Memory-Mapped Files - .NET - Microsoft Learn
  23. [PDF] Good Old-Fashioned Persistent Memory - USENIX
  24. https://man7.org/linux/man-pages/man2/msync.2.html
  25. MapViewOfFile function (memoryapi.h) - Win32 apps | Microsoft Learn
  26. The phaseout of the mmap() file operation - LWN.net
  27. Memory-mapped I/O (The GNU C Library)
  28. Taming the OOM killer - LWN.net
  29. File Mapping Security and Access Rights - Win32 apps
  30. None
  31. [PDF] mmap: Memory Mapped Files in R
  32. Efficient Memory Mapping for Terabyte Sparse Files in Java
  33. shm_open
  34. mmap
  35. https://pubs.opengroup.org/onlinepubs/9699919799/functions/mmap.html
  36. pthread_mutex_destroy
  37. 8.3. Producer-Consumer Problem - Computer Science - JMU
  38. [PDF] Principles of Operating Systems
  39. Memory-Mapped I/O - SQLite
  40. LevelDB Explained - Posix File Operation Details
  41. Configuration - Spark 4.0.1 Documentation - Apache Spark
  42. IO · facebook/rocksdb Wiki - GitHub
  43. SQLite performance tuning - phiresky's blog
  44. HugeTLB Pages — The Linux Kernel documentation
  45. madvise(2) - Linux manual page - man7.org
  46. Transparent Hugepage Support - The Linux Kernel documentation
  47. Mac OS X Manual Page For mmap(2) - Apple Developer
  48. CreateFileMappingW function (memoryapi.h) - Win32 apps
  49. Creating Named Shared Memory - Win32 apps - Microsoft Learn
  50. Closing a File Mapping Object - Win32 apps | Microsoft Learn
  51. CreateFileMappingA function (winbase.h) - Win32 apps
  52. Executable Images - Windows drivers - Microsoft Learn
  53. Inside Windows: Win32 Portable Executable File Format in Detail
  54. Interprocess communication (IPC) - UWP applications
  55. VirtualAlloc function (memoryapi.h) - Win32 apps - Microsoft Learn
  56. RM44L520: Memory mapping issue with FreeRTOS - TI E2E
  57. Maping a pointer at a particular address. - Kernel
  58. Memory - NDK - Android Developers
  59. BitUnmap: Attacking Android Ashmem - Google Project Zero
  60. Implementing Ashmem to share data between processes - Jinhan Hu
  61. mmap — Memory-mapped file support — Python 3.14.0 ...
  62. MappedByteBuffer (Java Platform SE 8 ) - Oracle Help Center
  63. Python mmap: Improved File I/O With Memory Mapping
  64. The 64-bit Standalone Plan 9 File Server
  65. v9fs: Plan 9 Resource Sharing for Linux
  66. Operating system integrity - Apple Support
  67. Memory mapping limitations on iOS - JUCE Forum
  68. Support for an MVP of mmap #304 - WebAssembly/WASI - GitHub
User Avatar
No comments yet.