List of file systems
List of file systems
Main page
L

List of file systems

logo
Community Hub0 subscribers
Read side by side
List of file systems
View on Wikipedia
from Wikipedia

The following lists identify, characterize, and link to more thorough information on file systems. Many older operating systems support only their one "native" file system, which does not bear any name apart from the name of the operating system itself.

Disk file systems

[edit]

Disk file systems are usually block-oriented. Files in a block-oriented file system are sequences of blocks, often featuring fully random-access read, write, and modify operations.

  • ADFS – Acorn's Advanced Disc filing system, successor to DFS.
  • AdvFS – Advanced File System, designed by Digital Equipment Corporation for their Digital UNIX (now Tru64 UNIX) operating system.
  • APFS – Apple File System is a file system for Apple products.
  • AthFS – AtheOS File System, a 64-bit journaled filesystem now used by Syllable. Also called AFS.
  • BFS – the Boot File System used on System V release 4.0 and UnixWare.
  • BFS – the Be File System used on BeOS, occasionally misnamed as BeFS. Open source implementation called OpenBFS is used by the Haiku operating system.
  • Byte File System (BFS) - file system used by z/VM for Unix applications
  • Btrfs – is a copy-on-write file system for Linux announced by Oracle in 2007 and published under the GNU General Public License (GPL).
  • CFS – The Cluster File System from Veritas, a Symantec company. It is the parallel access version of VxFS.
  • CP/M file system — Native filesystem used in the CP/M (Control Program for Microcomputers) operating system which was first released in 1974.
  • DFS – Acorn's Disc filing system.
  • DOS 3.x – Original floppy operating system and file system developed for the Apple II.
  • Extent File System (EFS) – an older block filing system under IRIX.
  • ext – Extended file system, designed for Linux systems.
  • ext2 – Second extended file system, designed for Linux systems.
  • ext3 – A journaled form of ext2.
  • ext4 – A follow-up for ext3 and also a journaled filesystem with support for extents.
  • ext3cow – A versioning file system form of ext3.
  • FAT – File Allocation Table, initially used on DOS and Microsoft Windows and now widely used for portable USB storage and some other devices; FAT12, FAT16 and FAT32 for 12-, 16- and 32-bit table depths.
  • FFS (Amiga) – Fast File System, used on Amiga systems. This FS has evolved over time. Now counts FFS1, FFS Intl, FFS DCache, FFS2.
  • FFS – Berkeley Fast File System, used on *BSD systems
  • Fossil – Plan 9 from Bell Labs snapshot archival file system.
  • Files-11 – OpenVMS file system; also used on some PDP-11 systems; supports record-oriented files
  • Flex machine file system
  • HAMMER — clustered DragonFly BSD filesystem, production-ready since DragonFly 2.2 (2009)[1][2]
  • HAMMER2 — recommended as the default root filesystem in DragonFly since 5.2 release in 2018[3][4][5]
  • HFS – Hierarchical File System in IBM's MVS from MVS/ESA OpenEdition through z/OS V2R4; not to be confused with Apple's HFS. IBM stated that z/OS users should migrate from HFS to zFS, and in z/OS V2R5 dropped support for HFS.
  • HFS – Hierarchical File System, in use until HFS+ was introduced on Mac OS 8.1. Also known as Mac OS Standard format. Successor to Macintosh File System (MFS) & predecessor to HFS+; not to be confused with IBM's HFS provided with z/OS
  • HFS+ – Updated version of Apple's HFS, Hierarchical File System, supported on Mac OS 8.1 & above, including macOS. Supports file system journaling, enabling recovery of data after a system crash. Also referred to as 'Mac OS Extended format or HFS Plus
  • HPFS – High Performance File System, used on OS/2
  • HTFS – High Throughput Filesystem, used on SCO OpenServer
  • ISO 9660 – Used on CD-ROM and DVD-ROM discs (Rock Ridge and Joliet are extensions to this)
  • JFS – IBM Journaling file system, provided in Linux, OS/2, and AIX. Supports extents.
  • LFS – 4.4BSD implementation of a log-structured file system
  • MFS – Macintosh File System, used on early Classic Mac OS systems. Succeeded by Hierarchical File System (HFS).
  • Next3 – A form of ext3 with snapshots support.[6]
  • MFS – TiVo's Media File System, a proprietary fault tolerant format used on TiVo hard drives for real time recording from live TV.
  • Minix file system – Used on Minix systems
  • NILFS – Linux implementation of a log-structured file system
  • NTFS – (New Technology File System) Used on Microsoft's Windows NT-based operating systems
  • NeXT - NeXTstation and NeXTcube file system
  • NetWare File System – The original NetWare 2.x–5.x file system, used optionally by later versions.
  • NSS – Novell Storage Services. This is a new 64-bit journaling file system using a balanced tree algorithm. Used in NetWare versions 5.0-up and recently ported to Linux.
  • OneFS – One File System. This is a fully journaled, distributed file system used by Isilon. OneFS uses FlexProtect and Reed–Solomon encodings to support up to four simultaneous disk failures.
  • OFS – Old File System, on Amiga. Good for floppies, but fairly useless on hard drives.
  • OS-9 file system
  • PFS – and PFS2, PFS3, etc. Technically interesting file system available for the Amiga, performs very well under a lot of circumstances.
  • ProDOS – Successor to DOS 3.x, for Apple II computers, including the IIgs
  • Qnx4fs – File system that is used in QNX version 4 and 6.
  • ReFS (Resilient File System) – File system by Microsoft with a particular focus on data resilience in server environments.
  • ReiserFS – File system that uses journaling
  • Reiser4 – File system that uses journaling, newest version of ReiserFS
  • Reliance – Datalight's transactional file system for high reliability applications
  • Reliance Nitro – Tree-based transactional, copy-on-write file system developed for high-performance embedded systems, from Datalight (Acquired by Tuxera in 2019)[7]
  • RFS – Native filesystem for RTEMS[8]
  • SkyFS – Developed for SkyOS to replace BFS as the operating system's main file system. It is based on BFS, but contains many new features.
  • SFS – Smart File System, journaling file system available for the Amiga platforms.
  • Soup (Apple) – the "file system" for Apple Newton Platform, structured as a shallow database
  • Tux3 – An experimental versioning file system intended as a replacement for ext3
  • UDF – Packet-based file system for WORM/RW media such as CD-RW and DVD, now supports hard drives and flash memory as well.
  • UFS – Unix File System, used on Solaris and older BSD systems
  • UFS2 – Unix File System, used on newer BSD systems
  • VaultFS – parallel distributed clusterable file system for Linux/Unix by Swiss Vault
  • VxFS Veritas file system, first commercial journaling file system[citation needed]; HP-UX, Solaris, Linux, AIX, UnixWare
  • VTOC (Volume Table Of Contents) - Data structure on IBM mainframe direct-access storage devices (DASD) such as disk drives that provides a way of locating the data sets that reside on the DASD volume.
  • XFS – Used on SGI IRIX and Linux systems
  • zFS – z/OS File System; not to be confused with other file systems named zFS or ZFS.
  • zFS - an IBM research project to develop a distributed, decentralized file system; not to be confused with other file systems named zFS or ZFS.
  • ZFS – a combined file system and logical volume manager designed by Sun Microsystems

File systems with built-in fault-tolerance

[edit]

These file systems have built-in checksumming and either mirroring or parity for extra redundancy on one or several block devices:

File systems optimized for flash memory, solid state media

[edit]
Main article: Flash file system

Solid state media, such as flash memory, are similar to disks in their interfaces, but have different problems. At low level, they require special handling such as wear leveling and different error detection and correction algorithms. Typically a device such as a solid-state drive handles such operations internally and therefore a regular file system can be used. However, for certain specialized installations (embedded systems, industrial applications) a file system optimized for plain flash memory is advantageous.

Many, but not all, file systems optimized for flash memory support TRIM commands to tell the storage device that certain blocks are no longer in use and can be reused.

  • 3FS – (Fire-Flyer File System) is a File System made by DeepSeek designed for AI Training and Inference workloads.[13]
  • APFS – Apple File System is a next-generation file system for Apple products.
  • CHFS – a NetBSD filesystem for embedded systems optimised for raw flash media.
  • exFAT – Microsoft file system intended for flash cards (see also XCFiles, an exFAT implementation for Wind River VxWorks and other embedded operating systems). Does not support TRIM on Windows. Supports TRIM on Linux.
  • ExtremeFFS – internal filesystem for SSDs.
  • F2FS – Flash-Friendly File System. An open source Linux file system introduced by Samsung in 2012.[14]
  • FFS2 (presumably preceded by FFS1), one of the earliest flash file systems. Developed and patented by Microsoft in the early 1990s.[15]
  • JFFS – original log structured Linux file system for NOR flash media.
  • JFFS2 – successor of JFFS, for NAND and NOR flash.
  • LSFS – a Log-structured file system with writable snapshots and inline data deduplication created by StarWind Software. Uses DRAM and flash to cache spinning disks.
  • LogFS – intended to replace JFFS2, better scalability. No longer under active development.[16]
  • NILFS – a log-structured file system for Linux with continuous snapshots.
  • Non-Volatile File System – the system for flash memory introduced by Palm, Inc.
  • NOVA – the "non-volatile memory accelerated" file system for persistent main memory.
  • OneFS – a filesystem utilized by Isilon. It supports selective placement of meta-data directly onto flash SSD.
  • Reliance Velocity - a proprietary flash file system by Tuxera with high resilience (fail-safe technology) and built-in data integrity. This file system is best suited for embedded applications requiring heavy data workloads over long-term operations. Reliance Velocity can used for all block based media like eMMC, UFS, eSD, SD card, CF card, and SSD. It is compatible for Linux, Android and QNX with portability to other embedded operating systems.
  • Reliance Edge - a proprietary file system by Tuxera for resource-constrained embedded systems. It has built-in data integrity with copy-on-write transactional technology and deterministic operations. This file system can be used for block based media and is configurable for Small POSIX, Full POSIX and can be ported to many RTOS environments. Tuxera has a certified version of this file system called Reliance Assure. The source code of Reliance Assure is complaint to MISRA C and developed following the ASPICE framework.
  • Segger Microcontroller Systems emFile – filesystem for deeply embedded applications which supports both NAND and NOR flash. Wear leveling, fast read and write, and very low RAM usage.
  • SPIFFS – SPI Flash File System, a wear-leveling filesystem intended for small NOR flash devices.
  • TFAT – a transactional version of the FAT filesystem.
  • TrueFFS – internal file system for SSDs, implementing error correction, bad block re-mapping and wear-leveling.
  • UBIFS – successor of JFFS2, optimized to utilize NAND and NOR flash.
  • Write Anywhere File Layout (WAFL) – an internal journaling file system utilized by NetApp within their DataONTAP OS, originally designed to use hard disk drives. WAFL uses RAID-DP to protect against multiple disk failures and non-volatile DRAM (NVRAM) for transaction logging of file system changes.
  • YAFFS – a log-structured file system designed for NAND flash, but also used with NOR flash.
  • LittleFS – a little fail-safe filesystem designed for microcontrollers.
  • JesFS – Jo's embedded serial FileSystem.[17] A very small footprint and robust filesystem, designed for very small microcontroller (16/32 bit). Open Source and licensed under GPL v3.

File systems not directly advertised as flash friendly but that support TRIM in major implementations

[edit]

Record-oriented file systems

[edit]

In record-oriented file systems files are stored as a collection of records. They are typically associated with mainframe and minicomputer operating systems. Programs read and write whole records, rather than bytes or arbitrary byte ranges, and can seek to a record boundary but not within records. The more sophisticated record-oriented file systems have more in common with simple databases than with other file systems.

Shared-disk file systems

[edit]

Shared-disk file systems (also called shared-storage file systems, SAN file system, Clustered file system or even cluster file systems) are primarily used in a storage area network where all nodes directly access the block storage where the file system is located. This makes it possible for nodes to fail without affecting access to the file system from the other nodes. Shared-disk file systems are normally used in a high-availability cluster together with storage on hardware RAID. Shared-disk file systems normally do not scale over 64 or 128 nodes.

Shared-disk file systems may be symmetric where metadata is distributed among the nodes or asymmetric with centralized metadata servers.

Distributed file systems

[edit]
See also: Comparison of distributed file systems

Distributed file systems are also called network file systems. Many implementations have been made, they are location dependent and they have access control lists (ACLs), unless otherwise stated below.

Distributed fault-tolerant file systems

[edit]

Distributed fault-tolerant replication of data between nodes (between servers or servers/clients) for high availability and offline (disconnected) operation.

  • Coda from Carnegie Mellon University focuses on bandwidth-adaptive operation (including disconnected operation) using a client-side cache for mobile computing. It is a descendant of AFS-2. It is available for Linux under the GPL.
  • Distributed File System (Dfs) from Microsoft focuses on location transparency and high availability. Available for Windows under a proprietary software license.
  • HAMMER and HAMMER2 – DragonFly BSD's filesystems for clustered storage, created by Matt Dillon.[1][5]
  • InterMezzo from Cluster File Systems uses synchronization over HTTP. Available for Linux under GPL but no longer in development since the developers are working on Lustre.
  • LizardFS a networking, distributed file system based on MooseFS[21]
  • Moose File System (MooseFS) is a networking, distributed file system. It spreads data over several physical locations (servers), which are visible to a user as one resource. Works on Linux, FreeBSD, NetBSD, OpenSolaris and macOS. Master server and chunkservers can also run on Solaris and Windows with Cygwin.
  • Scality is a distributed fault-tolerant filesystem.
  • Tahoe-LAFS is an open source secure, decentralized, fault-tolerant filesystem utilizing encryption as the basis for a least-authority replicated design.
  • A FAT12 and FAT16 (and FAT32) extension to support automatic file distribution across nodes with extra attributes like local, mirror on update, mirror on close, compound on update, compound on close in IBM 4680 OS and Toshiba 4690 OS. The distribution attributes are stored on a file-by-file basis in special entries in the directory table.[22][23]
  • OpenHarmony Distributed File System (HMDFS) used for Huawei's HarmonyOS with HarmonyOS NEXT base and OpenHarmony-based operating systems, alongside openEuler server OS that is a cross-device file access where devices can read and edit files on transparently when the two devices are connected to the same network with Access token manager. Multiple embedded devices connected to the network can automatically synchronise file data with the edge server.[24]
  • VaultFS – parallel distributed clusterable filesystem using dynamically configurable any*Data + any*Parity EC (erasure coding) and dynamically tolerates bitrot, media & server failures

Distributed parallel file systems

[edit]

Distributed parallel file systems stripe data over multiple servers for high performance. They are normally used in high-performance computing (HPC).

Some of the distributed parallel file systems use an object storage device (OSD) (in Lustre called OST) for chunks of data together with centralized metadata servers.

  • BeeGFS is a hardware-independent parallel file system that features distributed metadata and striping of files across multiple targets, such as NVMe devices or logical volumes.
  • Lustre is an open-source high-performance distributed parallel file system for Linux, used on many of the largest computers in the world.
  • Parallel Virtual File System (PVFS, PVFS2, OrangeFS). Developed to store virtual system images, with a focus on non-shared writing optimizations. Available for Linux under GPL.
  • VaultFS – configurable any*Data + any*Parity EC (erasure coding) chunks are widely distributed on D+P disks across the cluster

Distributed parallel fault-tolerant file systems

[edit]

Distributed file systems, which also are parallel and fault tolerant, stripe and replicate data over multiple servers for high performance and to maintain data integrity. Even if a server fails no data is lost. The file systems are used in both high-performance computing (HPC) and high-availability clusters.

All file systems listed here focus on high availability, scalability and high performance unless otherwise stated below.

Name By License OS Description
Alluxio UC Berkeley, Alluxio Apache License Cross-platform An open-source virtual distributed file system (VDFS).
BeeGFS (formerly FhGFS) Fraunhofer Society GNU GPL v2 for client, other components are proprietary Linux A free to use file system with optional professional support, designed for easy usage and high performance, used on some of the fastest computer clusters in the world. BeeGFS allows replication of storage volumes with automatic failover and self-healing.
CephFS Inktank Storage, a company acquired by Red Hat GNU LGPL Linux kernel, FreeBSD via FUSE[25] A massively scalable object store. CephFS was merged into the Linux kernel in 2010. Ceph's foundation is the reliable autonomic distributed object store (RADOS), which provides object storage via programmatic interface and S3 or Swift REST APIs, block storage to QEMU/KVM/Linux hosts, and POSIX filesystem storage which can be mounted by Linux kernel and FUSE clients.
Chiron FS GNU GPL v3 Linux A FUSE-based, transparent replication file system, layering on an existing file system and implementing at the file system level what RAID 1 does at the device level. A notably convenient consequence is the possibility of picking single target directories, without the need of replicating entire partitions. (The project has no visible activity after 2008; a status request in Oct. 2009 in the chironfs-forum is unanswered.)
CloudStore Kosmix Apache License Google File System workalike. Replaced by Quantcast File System (QFS)
dCache DESY and others Proprietary (free for non-commercial usage)[26] Linux A write once filesystem, accessible via various protocols.
General Parallel File System (GPFS) IBM Proprietary Linux, Windows and AIX A POSIX-compliant, high-performance, parallel filesystem. Support synchronous replication between attached block storage, and asynchronous replication to remote filesystems. Also support erasure coding on dual homed SAS attached storage, and distributed over multiple storage nodes.
Gfarm file system NPO Tsukuba OSS Technical Support Center X11 License Linux, macOS, FreeBSD, NetBSD and Solaris Uses PostgreSQL for metadata and FUSE for mounting.
GlusterFS Gluster, a company acquired by Red Hat GNU GPL v3 Linux, NetBSD, FreeBSD, OpenSolaris A general purpose distributed file system for scalable storage. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. GlusterFS is the main component in Red Hat Storage Server.
Google File System (GFS) Google Internal software Focus on fault tolerance, high throughput and scalability.
Hadoop Distributed File System Apache Software Foundation Apache License Cross-platform Open source GoogleFS clone.
IBRIX Fusion IBRIX Proprietary
JuiceFS Juicedata Apache License Cross-platform An open-source POSIX-compliant file system built on top of Redis and object storage (e.g. Amazon S3), designed and optimized for cloud native environment.
LizardFS Skytechnology GNU GPL v3 Cross-platform An open source, highly available POSIX-compliant file system that supports Windows clients.
Lustre Originally developed by Cluster File Systems and currently supported by OpenSFS GNU GPL v2 & LGPL Linux A POSIX-compliant, high-performance filesystem used on a majority of systems in the Top-500 list of HPC systems. Lustre has high availability via storage failover.
MapR FS MapR Proprietary Linux Highly scalable, POSIX compliant, fault tolerant, read/write filesystem with a distributed, fault tolerant metadata service. It provides an HDFS and NFS interface to clients as well as a noSQL table interface and Apache Kafka compatible messaging system.
MooseFS Core Technology GNU GPL v2 and proprietary[27] Cross-platform (Linux, NetBSD, FreeBSD, macOS, OpenSolaris) A fault tolerant, highly available and high performance scale-out network distributed file system. It spreads data over several physical commodity x86 servers, which are visible to the user as one namespace. For standard file operations MooseFS acts like any other Unix-like file systems.
ObjectiveFS Objective Security Corporation Proprietary Linux, macOS POSIX-compliant shared distributed filesystem. Uses object store as a backend. Runs on AWS S3, GCS and object store devices.
OneFS distributed file system Isilon Proprietary[28] FreeBSD BSD-based OS on dedicated Intel based hardware, serving NFS v3 and SMB/CIFS to Windows, macOS, Linux and other UNIX clients under a proprietary software.
OIO-FS OpenIO Proprietary Linux OIO-FS provides file-oriented access to OpenIO SDS object storage backend. It is based on FUSE technology and presents a POSIX file system to users. This access can be used locally, or over a network using NFS or SMB.[29]
PanFS Panasas Proprietary Linux, macOS, FreeBSD A POSIX-compliant, high-performance, parallel filesystem used by HPC clusters. It uses erasure coding and snapshots for data protection, is based upon a scale-out object store, and is focused on transparent failure recovery and ease of use.
Quobyte DCFS Quobyte Proprietary Linux, macOS, FreeBSD A fault-tolerant, parallel POSIX file system, with block (VMs) and object (S3) interfaces, and advanced enterprise features like multi-tenancy, strong authentication, encryption. Split-brain safe fault-tolerance is achieved through Paxos-based leader election and erasure coding.
RozoFS Rozo Systems GNU GPL v2 Linux A POSIX DFS focused on fault-tolerance and high-performance, based on the Mojette erasure code to reduce significantly the amount of redundancy (compared to plain replication).
Scality Scality ring Proprietary Linux A POSIX file system[citation needed] focused on high availability and performance. Also provides S3/REST/NFS interfaces.
Tahoe-LAFS Tahoe-LAFS Software Foundation GNU GPL v2+ and other[30] Linux, Windows, macOS A secure, decentralized, fault-tolerant, peer-to-peer distributed data store and distributed file system.
VaultFS Swiss Vault Proprietary Linux, Unix Peer-to-Peer dynamically configurable EC (any*Data + any*Parity) bitrot & HW fault-tolerant POSIX/S3 distributed file system using intermixable CMR & SMR shingled magnetic recording disks.
XtreemFS Contrail E.U. project, the German MoSGrid project and the German project "First We Take Berlin" BSD 3-Clause[31] Linux, Solaris, macOS, Windows A cross-platform file system for wide area networks. It replicates the data for fault tolerance and caches metadata and data to improve performance over high-latency links. SSL and X.509 certificates support makes XtreemFS usable over public networks. It also supports striping for usage in a cluster.

In development:

Peer-to-peer file systems

[edit]

Some of these may be called cooperative storage cloud.

  • IBM Cloud Object Storage uses Cauchy Reed–Solomon information dispersal algorithms to separate data into unrecognizable slices and distribute them, via secure Internet connections, to multiple storage locations.
  • Scality is a distributed filesystem using the Chord peer-to-peer protocol.
  • IPFS InterPlanetary File System is p2p, worldwide distributed content-addressable, file-system.
  • VaultFS – fully peer-to-peer with distributed data & metadata, without separate Master or Failover nodes

Special-purpose file systems

[edit]
  • aufs an enhanced version of UnionFS stackable unification file system
  • AXFS (small footprint compressed read-only, with XIP)
  • Barracuda WebDAV plug-in. Secure Network File Server for embedded devices.
  • Boot File System is used on UnixWare to store files necessary for its boot process.
  • CDfs - a Linux virtual file system that provides access to individual data and audio tracks on compact discs[32][33]
  • Compact Disc File System (reading and writing of CDs; experimental)
  • cfs (caching)
  • Cramfs (small footprint compressed read-only)
  • Davfs2 (WebDAV)
  • Freenet – Decentralized, censorship-resistant
  • FTPFS (FTP access)
  • GmailFS (Google Mail File System)
  • GridFS – GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16 MB for MongoDB.
  • lnfs (long names)
  • LTFS (Linear Tape File System for LTO and Enterprise tape)
  • MVFS – MultiVersion File System, proprietary, used by IBM DevOps Code ClearCase.
  • Nexfs Combines Block, File, Object and Cloud storage into a single pool of auto-tiering POSIX compatible storage.
  • OverlayFS – A union mount filesystem implementation for Linux. Used mainly by Docker for its image layers.
  • romfs
  • SquashFS (compressed read-only)
  • UMSDOS, UVFAT – FAT file systems extended to store permissions and metadata (and in the case of UVFAT, VFAT long file names), used for Linux
  • UnionFS – stackable unification file system, which can appear to merge the contents of several directories (branches), while keeping their physical content separate
  • VaultFS – can intermixably utilize SMR shingled magnetic recording disks to achieve the highest storage densities
  • Venti – Plan 9 de-duplicated storage used by Fossil.

Pseudo file systems

[edit]
  • devfs – a virtual file system in Unix-like operating systems for managing device nodes on-the-fly
  • procfs – a pseudo-file system, used to access kernel information about processes
  • tmpfs – in-memory temporary file system (on Unix-like platforms)
  • sysfs – a virtual file system in Linux holding information about buses, devices, firmware, filesystems, etc.
  • debugfs – a virtual file system in Linux for accessing and controlling kernel debugging
  • configfs – a writable file system used to configure various kernel components of Linux
  • sysctlfs – allow accessing sysctl nodes via a file system; available on NetBSD via PUFFS,[34] FreeBSD kernel via a 3rd-party module,[35][36][37] and Linux as a part of Linux procfs.[38]
  • kernfs – a file system found on some BSD systems (notably NetBSD) that provides access to some kernel state variables; similar to sysctlfs, Linux procfs and Linux sysfs.
  • WinFS - Uses a relational database to manage files
  • wikifs – a server application for Plan 9's virtual, wiki, file system

Compressed file systems

[edit]
  • VaultFS – auto background compression (writing) & decompression (reading) per file or directory: ten levels: 0 (uncompressed) ... 9 (maximal compression)

Encrypted file systems

[edit]
See also: Rubberhose (file system)

File system interfaces

[edit]

These are not really file systems; they allow access to file systems from an operating system standpoint.

  • FUSE (file system in userspace, like LUFS but better maintained)
  • LUFS (Linux userland file system – seems to be abandoned in favour of FUSE)
  • PUFFS (Userspace filesystem for NetBSD, including a compatibility layer called librefuse for porting existing FUSE-based applications)
  • Secure Shell File System (SSHFS) – locally mount a remote directory on a server using only a secure shell login.
  • VFS Virtual Filesystem

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

List of file systems

View on Grokipedia
from Grokipedia
A file system is a method used by operating systems to organize, store, and retrieve data on storage devices such as hard drives, solid-state drives, SSDs, and optical media.[1] These systems vary widely in design to accommodate different hardware, performance needs, and use cases, from local disk management to networked and virtual environments.[2] File systems are broadly categorized into several types, including disk-based systems for physical storage media, network-based systems for remote access, and virtual systems for in-memory or kernel-level operations.[2] Notable examples of disk-based file systems include FAT for cross-platform compatibility on removable media, NTFS for advanced features like encryption and compression in Windows environments, ext2/ext3/ext4 for Linux journaling and reliability, HFS+ or APFS for Apple macOS hierarchical storage, ISO 9660 for CD-ROMs, UDF for DVDs and Blu-ray discs, ODS-5 for OpenVMS, exFAT for modern removable media, and Btrfs for advanced Linux features like snapshots. Network-based file systems, such as NFS, enable sharing of files and directories across networked servers and clients, facilitating distributed computing.[2] Virtual file systems like PROCFS provide access to system processes and kernel data without physical storage, while CacheFS improves performance by caching remote data locally.[2] Additional categories encompass distributed parallel systems for cluster environments and specialized systems optimized for flash memory or fault tolerance, reflecting ongoing evolution to support modern workloads like cloud storage and big data.[3]

Disk file systems

Conventional disk file systems

Conventional disk file systems are traditional block-based structures designed primarily for hard disk drives (HDDs) on rotating magnetic media, employing allocation methods such as contiguous allocation for sequential file storage, linked allocation for non-contiguous blocks via pointers, or indexed allocation using inodes or similar metadata to map file blocks efficiently. These systems manage data organization without specialized hardware integration, focusing on basic reliability through optional journaling to log metadata changes and reduce recovery times after crashes, though many early variants lack this feature.[4] The evolution of conventional disk file systems began in the late 1970s with simple designs for personal computing, progressing to address growing storage capacities and performance needs. The File Allocation Table (FAT) system, developed by Microsoft in the 1980s for MS-DOS, used a table-based index for cluster mapping and became a standard for compatibility across operating systems.[5] FAT12 and FAT16 variants supported partitions up to 2 GB in MS-DOS environments but were limited by 16-bit addressing, capping individual file sizes at 2 GB in FAT16.[5] This prompted the introduction of FAT32 in 1996 with Windows 95 OEM Service Release 2, expanding to 32-bit entries for partitions up to 2 TB and files up to 4 GB, though it retained simplicity at the cost of lacking native security features like permissions.[5] FAT's lightweight design made it ideal for removable media such as floppy disks and flash drives, ensuring broad interoperability despite performance degradation on volumes exceeding 200 MB.[5] In parallel, Unix-like systems advanced with inode-based designs. The second extended file system (ext2), introduced in 1993 for Linux, improved upon earlier prototypes by using indexed allocation with block groups for better scalability, supporting partitions up to 4 TB and files up to 2 TB on 32-bit systems without journaling.[6] Its successor, ext3, added journaling in 2001 to enhance crash recovery while maintaining backward compatibility with ext2, though limited to 16 TB filesystems and 2 TB files.[6] By 2008, ext4 introduced extent-based allocation to reduce fragmentation for large files, extent trees for efficient mapping, and 64-bit addressing, enabling up to 1 EB filesystems and 16 TB files with 4 KB blocks, making it suitable for modern HDDs.[7][6] Microsoft's NT File System (NTFS), released in 1993 with Windows NT 3.1, marked a shift toward enterprise features in conventional designs, using a Master File Table (MFT) for centralized metadata storage where each file or directory occupies a fixed record describing attributes like size and location.[8] NTFS supports access permissions via Access Control Lists (ACLs), file-level encryption, and compression to optimize space on large volumes, with variable-sized MFT records for handling diverse file types efficiently.[8] Other notable systems include ReiserFS, developed by Hans Reiser and released in 2001 for Linux, which emphasized balanced tree structures for fast small-file handling and journaling, outperforming ext2 in metadata operations but later deprecated in the Linux kernel by 2025 due to halted development following the creator's legal issues.[9][10] IBM's Journaled File System (JFS), originally introduced in 1990 for AIX with extent-based allocation and B+ trees for directories, was ported to Linux in 2001 as JFS2, offering sub-second recovery and scalability for multi-processor environments without built-in redundancy.[11]

File systems with built-in fault-tolerance

File systems with built-in fault-tolerance integrate mechanisms for data integrity, redundancy, and recovery directly into their core design, distinguishing them from conventional systems that rely on external hardware like RAID controllers for protection. These file systems commonly employ copy-on-write (CoW) techniques to ensure atomic updates, preventing partial writes that could lead to corruption, while end-to-end checksums verify data integrity across the storage stack. Snapshots enable point-in-time copies for rapid recovery, and self-healing processes automatically detect and repair errors using redundant copies or parity information, all without depending on separate volume management layers. This contrasts with add-on RAID solutions, as fault-tolerance is natively managed at the file system level for seamless operation and scalability. Building on foundational journaling methods from earlier disk file systems, these advanced features provide proactive recovery from silent data corruption and hardware failures. ZFS, developed by Sun Microsystems and first released in November 2005 as part of Solaris 10, exemplifies integrated fault-tolerance through its pooled storage model that combines file system and volume management. It uses CoW to maintain data consistency, creating immutable blocks for new writes and enabling efficient snapshots that capture the entire pool state instantaneously. End-to-end checksums, computed for every block and stored in parent blocks, allow ZFS to detect silent corruption—even bit flips introduced by faulty RAM or disks—triggering self-healing by reconstructing data from mirrors or RAID-Z parity. RAID-Z variants (Z1 for single-disk tolerance, Z2 for double, and Z3 for triple) provide software-based redundancy without the write-hole issues of traditional RAID-5/6, supporting pool sizes up to 256 zettabytes theoretically. Deduplication further enhances efficiency by identifying and storing unique blocks only. Originally licensed under the Common Development and Distribution License (CDDL), ZFS faced compatibility challenges after Oracle's 2010 acquisition of Sun, leading to the OpenZFS fork in 2013 to continue open-source development across platforms like Linux and FreeBSD.[12][13] Btrfs, initiated by Oracle in 2007 and merged into the Linux kernel in 2009, serves as the Linux community's response to ZFS, emphasizing CoW for fault-tolerant operations on conventional disks. Its subvolumes act as lightweight, mountable containers within a single file system, facilitating hierarchical organization and isolation for snapshots that share unchanged data blocks via CoW to minimize storage overhead. Built-in RAID support includes levels 0 (striping), 1 (mirroring), 5/6 (single/double parity), and 10 (mirrored striping), with checksums on both data and metadata enabling self-healing during scrubs to repair corruptions automatically from redundant copies. The send/receive functionality allows incremental backups by streaming snapshot differences over networks, aiding efficient data migration and recovery. While Btrfs has evolved to address early stability concerns, its design prioritizes ease of administration for large-scale storage.[14][15] Bcachefs, begun in 2015 by Kent Overstreet as an evolution of the bcache caching layer, achieved stable integration into the Linux kernel with version 6.7 in December 2023, focusing on superior reliability through a copy-on-write structure modeled after database principles. It employs CoW for snapshots and versioning, ensuring crash consistency and enabling rollback to prior states, while extensive checksumming and logging detect anomalies for online repair via a comprehensive fsck tool that operates without downtime. Erasure coding, inspired by scalable parity schemes, provides flexible redundancy beyond fixed RAID levels, tolerating multiple device failures in large pools. Integration with bcache allows hybrid setups where fast SSDs cache slower HDDs, optimizing performance without compromising fault-tolerance. However, due to ongoing development disputes, bcachefs was marked externally maintained in kernel 6.17 and fully removed from the mainline in 6.18 (September 2025), now available as a DKMS module. This shift underscores its emphasis on robustness over Btrfs's historical complexity issues, positioning it for high-reliability environments.[16][17]

File systems optimized for flash memory and solid-state media

File systems optimized for flash memory and solid-state media address the unique constraints of non-volatile storage devices like NAND flash and SSDs, which differ significantly from traditional magnetic disks. These devices feature limited program/erase (P/E) cycles, typically ranging from 3,000 to 100,000 per block, necessitating techniques such as wear leveling to distribute writes evenly across cells and prevent premature failure of individual blocks.[18] Flash memory requires block-level erases before reprogramming, leading to out-of-place writes where updates are appended to new locations rather than overwritten in place, which helps mitigate write amplification but introduces the need for garbage collection to reclaim obsolete data.[18] Additionally, support for the TRIM command allows file systems to inform the underlying storage controller of freed blocks, enabling efficient garbage collection and reducing unnecessary internal writes.[19] Log-structured designs are common, organizing data in sequential logs aligned to flash page sizes (often 4KB or multiples) to minimize random writes and align with the sequential nature of flash operations.[18] The evolution of these file systems began in the early 2000s with efforts to support embedded and resource-constrained environments. YAFFS (Yet Another Flash File System), introduced in 2002, was one of the first file systems specifically designed for NAND flash in embedded systems, employing a log-structured approach with direct interfacing to flash hardware to handle bad block management and wear leveling without relying on a flash translation layer (FTL).[20] It uses out-of-place updates and checksums for data integrity, making it suitable for NOR and NAND flash in devices like mobile phones and IoT gadgets.[21] Subsequent developments built on these foundations, incorporating advanced compression and zoning to further optimize for modern flash hierarchies. F2FS (Flash-Friendly File System), developed by Samsung and integrated into the Linux kernel in 2012, exemplifies a log-structured file system tailored for NAND flash.[22] It divides storage into zones for hot and cold data to reduce write amplification, employing multi-head logging where multiple log streams allow concurrent writes to different sections, improving throughput on SSDs by up to 50% in random write benchmarks compared to ext4.[18] F2FS aligns allocations to flash page boundaries and supports TRIM, enabling efficient garbage collection and extending device lifespan in mobile and embedded Linux environments.[18] APFS (Apple File System), released by Apple in 2017 for macOS High Sierra and iOS 10.3, is optimized for SSDs across Apple's ecosystem, using a copy-on-write (CoW) mechanism for metadata and snapshots to ensure crash protection without journaling overhead.[23] Its container-based architecture allows multiple volumes to share free space dynamically within a single partition, facilitating efficient use of flash storage while supporting native encryption at the volume level.[23] Snapshots enable instantaneous backups and clones, reducing write operations and aligning with flash's sequential write preferences.[24] exFAT (Extended File Allocation Table), introduced by Microsoft in 2006, serves as a lightweight file system for removable flash media such as SD cards and USB drives, prioritizing simplicity and cross-platform compatibility over advanced features like journaling.[25] It employs a cluster bitmap for allocation, supporting file sizes up to 128 petabytes and volumes up to 128 petabytes, which suits high-capacity flash cards without the fragmentation issues of FAT32.[25] Lacking built-in journaling, exFAT relies on the host device's error handling but achieves fast formatting and mounting times, making it ideal for consumer flash storage.[25] EROFS (Enhanced Read-Only File System), developed by Huawei and upstreamed to the Linux kernel in version 4.19 in 2018, targets read-heavy workloads on flash devices, particularly for Android over-the-air (OTA) updates and system images.[26] It uses fixed-sized output compression (e.g., LZ4 or Zstd) to reduce storage footprint by up to 50% while enabling inline decompression for low-latency reads, without the overhead of full read-write file systems. EROFS supports deduplication and slab-based allocation for metadata, optimizing for mobile flash where boot times and memory usage are critical; further enhancements, including on-demand loading via fscache, were added in Linux 5.19 in 2022.[27][28]

Record-oriented file systems

Record-oriented file systems organize data into collections of fixed- or variable-length records, often with associated metadata such as record length and type, providing a structured abstraction that contrasts with the byte-stream model of conventional disk file systems. This design supports efficient sequential access and is well-suited for environments emphasizing batch processing or structured data retrieval, such as mainframes and embedded applications.[29] A key example is IBM's Virtual Storage Access Method (VSAM), introduced in the early 1970s as part of OS/VS for z/OS mainframes. VSAM structures data within control intervals—logical units typically sized at 4 KB—that group one or more records, enabling efficient I/O transfers. It supports multiple record organization types, including key-sequenced data sets (KSDS) for indexed random access via keys, entry-sequenced data sets (ESDS) for append-only sequential storage, and relative record data sets (RRDS) for fixed-position addressing. VSAM's hierarchical indexes facilitate rapid record location and retrieval, making it ideal for high-volume transaction processing.[30][31][32] The Indexed Sequential Access Method (ISAM), developed by IBM in the 1960s, served as an early precursor to database indexing and exemplified record-oriented design for combining sequential and direct access. ISAM stores records in sorted order within a primary data file, paired with a separate index file that maps keys to record addresses for quick lookups. However, frequent insertions could fill primary blocks, forcing records into overflow areas that required periodic reorganization to maintain performance; these overflow issues highlighted limitations in dynamic growth, paving the way for more balanced structures like B-trees in subsequent systems.[33][34] Record-oriented file systems dominated mainframe environments in the 1970s and 1980s, where they optimized sequential access for business applications akin to tape-based processing. Their influence waned in the late 1980s and 1990s with the ascendancy of relational databases, which provided superior ad-hoc querying and normalization for complex data relationships. Despite this, they endure in legacy z/OS systems for critical workloads and have adapted to modern contexts. Additionally, the Distributed Computing Environment's File Service (DCE DFS) in the 1990s incorporated record-oriented handling in components like the System File Service (SFS) to support transactional access beyond standard Unix semantics.[35][36]

Shared-disk file systems

Shared-disk file systems enable multiple hosts to concurrently access the same underlying block storage, typically connected via a Storage Area Network (SAN) or the iSCSI protocol, providing low-latency shared access within local clusters without requiring data replication across nodes. These systems rely on sophisticated concurrency controls, such as distributed lock managers (DLM), to coordinate access and prevent data corruption by enforcing exclusive locks on files, metadata, or blocks during writes.[37] This architecture contrasts with fully distributed systems by centralizing storage while distributing compute, making it suitable for high-availability clustering in environments like databases or virtualization.[38] The origins of shared-disk file systems trace back to the 1970s with early mainframe implementations, such as IBM's systems that allowed multiple processors to share physical disks through channel connections, evolving into more robust clusters like DEC's VAXcluster in the early 1980s, which used distributed lock managers for file sharing across VMS nodes. This foundational approach influenced modern designs, adapting to virtualization and cloud-era demands by integrating with protocols like iSCSI for cost-effective shared storage over Ethernet.[39] Today, these file systems emphasize scalability for multi-host environments, often incorporating fencing mechanisms to isolate faulty nodes and maintain data integrity during failures.[40] A prominent example is GFS2 (Global File System 2), introduced by Red Hat in 2005 as a journaling file system for Linux clusters, which uses DLM for distributed locking to ensure coherent access across nodes.[41] GFS2 supports standard Linux quotas for resource management and integrates multi-host fencing via cluster infrastructure to handle node failures gracefully, enabling seamless failover in enterprise setups.[42] Another key implementation is OCFS2 (Oracle Cluster File System 2), released by Oracle in 2006 and optimized for Oracle Real Application Clusters (RAC), featuring extent-based allocation for efficient large-file handling and integration with Heartbeat for high-availability monitoring via disk and network heartbeats.[43] OCFS2's design allows concurrent read/write access while maintaining POSIX compliance, making it ideal for shared Oracle environments. VMFS (Virtual Machine File System), developed by VMware starting in 1999 for ESXi hypervisors, employs SCSI-3 reservations and file-level locking to manage concurrent access to virtual machine files, supporting features like live VM migration and snapshots through atomic locking protocols.[44] In its VMFS7 version from 2017, it gained enhancements for integration with vSAN hyperconverged infrastructure, improving scalability to support files up to 64 TB and better handling of dense virtualization workloads. These systems collectively demonstrate the evolution toward reliable, cluster-optimized storage for virtualization and enterprise applications.

Distributed file systems

Client-server network file systems

Client-server network file systems operate on a model where clients mount and access remote file systems hosted on dedicated servers over a network, providing transparent file sharing as if the files were local. This architecture relies on protocols that handle remote procedure calls (RPCs) for operations like reading, writing, and directory traversal, often incorporating caching mechanisms on the client side to improve performance and reduce network latency. Early implementations emphasized stateless designs to simplify recovery from failures, while later versions introduced stateful elements for better consistency and security.[45][46] The Network File System (NFS), developed by Sun Microsystems in 1984, became a foundational protocol for Unix-like environments, standardizing remote file access through RPC-based communication. NFS version 2, specified in RFC 1094 (1989), supported stateless operations and file handles for identification, enabling clients to mount server exports transparently. NFS version 3 (RFC 1813, 1995) enhanced performance with larger file support and asynchronous writes, while NFS version 4 (RFC 3530, 2003) shifted toward statefulness, integrating security via Kerberos and adding features like compound operations to batch requests. Further evolution in NFS version 4.1 (RFC 5661, 2010) introduced pNFS for parallel data access, and NFS version 4.2 (RFC 7862, 2016) added server-side clone operations, allowing efficient duplication of files without client-server data transfer.[45] Server Message Block (SMB), originating in the mid-1980s as part of IBM's LAN Manager and adopted by Microsoft, facilitates file and printer sharing primarily in Windows ecosystems through a request-response protocol. SMB version 1.0, refined as the Common Internet File System (CIFS) in 1996, introduced opportunistic locking to manage concurrent access and support long file names. SMB version 2 (2006, with Windows Vista and Server 2008) improved efficiency by reducing command overhead and adding leasing for caching. SMB version 3 (2012, with Windows 8 and Server 2012) incorporated end-to-end encryption using AES-CMAC, multichannel support for aggregating multiple network paths to enhance throughput and fault tolerance, and persistent handles for resilient sessions. In 2020, Microsoft introduced SMB over QUIC, leveraging the QUIC transport protocol (RFC 9000) to enable secure, low-latency access over the internet without requiring a VPN, as implemented in Windows Server 2022.[47][48][49][50] Apple Filing Protocol (AFP), introduced by Apple in 1988 as part of AppleShare for Mac OS System 6, provided network file services over AppleTalk and later TCP/IP, emphasizing integration with Macintosh hierarchies like resource forks. AFP version 3.0 (2002, with Mac OS X Server) added Unicode support and better security, but it has been progressively deprecated in favor of SMB; as of macOS Sequoia 15.5 (2025), the AFP client is marked for removal in future updates.[51][52] The 9P protocol, designed for the Plan 9 operating system at Bell Labs in 1989, offers a lightweight, RPC-based interface for file system access, treating resources uniformly as files accessible via simple messages like T-messages (requests) and R-messages (replies). Its minimalism supports transparent mounting and has influenced implementations in Linux (via v9fs) for virtualized environments.[53][54]

Distributed fault-tolerant file systems

Distributed fault-tolerant file systems are engineered for large-scale environments where data must remain accessible and intact across distributed nodes, even in the face of hardware failures, network partitions, or node outages. Central to their design are mechanisms like data replication, which maintains multiple identical copies of data across nodes to enable quick recovery, and erasure coding, a technique that divides data into fragments with added parity blocks, allowing reconstruction of lost portions using fewer resources than full replication while tolerating a configurable number of failures. Automatic failover further ensures continuity by detecting failures through heartbeats or monitoring and redirecting client requests to surviving replicas or rebuilt data without manual intervention, often achieving near-zero downtime in production clusters. These systems prioritize durability and availability over strict consistency in some cases, handling petabyte-scale datasets in big data ecosystems.[55][56] The Google File System (GFS), released in 2003 by Google, pioneered many of these principles with a master-worker model tailored for data-intensive applications. In GFS, files are split into fixed-size 64 MB chunks, each replicated typically three times across commodity chunkservers to guard against disk or server failures, with the single master coordinating metadata and placement decisions for load balancing and recovery. Upon detecting a chunkserver failure, the master initiates re-replication from healthy copies to maintain the redundancy level, ensuring sustained availability for streaming workloads like MapReduce jobs. This architecture influenced subsequent systems by demonstrating scalable fault tolerance on inexpensive hardware.[57] Inspired by GFS, the Hadoop Distributed File System (HDFS), which began development in 2006 as part of the Apache Hadoop project, adopts a similar block-based approach with a NameNode managing metadata and multiple DataNodes storing data blocks of 128 MB or larger, defaulting to three replicas per block. HDFS incorporates rack awareness to enhance fault tolerance, placing the first replica on the same node, the second on another node in the same rack, and the third on a node in a different rack, thereby mitigating risks from rack-wide failures like power outages while optimizing network throughput. Automatic failover is supported via secondary NameNodes or high-availability configurations that promote standby nodes during outages, making HDFS a cornerstone for reliable big data storage in open-source environments.[58][56] For specialized workloads like media storage, Facebook's Haystack, introduced in 2008 and detailed in a 2010 publication, serves as an object storage system optimized for billions of immutable photos, aggregating up to 200 million objects per haystack file to minimize metadata overhead. Haystack employs multi-level replication across storage hosts and machines, with automatic failover handled by directory services that redirect reads to healthy replicas upon host failures, achieving high read throughput for write-once-read-many access patterns without a traditional file system hierarchy. This design underscores fault tolerance in object-centric distributed systems for social media-scale data.[59] GlusterFS, launched in 2005 and integrated into Red Hat's portfolio after its 2011 acquisition, offers a scale-out architecture without a centralized metadata server, using modular "bricks" on storage servers to form volumes through stackable translators for replication or erasure coding. It supports geo-replication for asynchronous mirroring across data centers, enabling disaster recovery and fault tolerance against regional outages, with automatic healing via self-checksumming and rebalancing during node additions or failures. Ceph, which gained prominence around 2010 through Inktank's commercialization efforts (acquired by Red Hat in 2014), provides unified object, block, and file storage via the RADOS layer, leveraging the CRUSH algorithm for pseudo-random data placement and supporting both replication and erasure coding profiles—such as 4+2 configurations tolerating two OSD failures—for efficient, self-managing fault tolerance.[60][61][62] Apache Ozone, originating in 2016 within the Hadoop ecosystem, extends HDFS fault tolerance to S3-compatible object storage, using containerized blocks with replication or erasure coding for billions of objects across clusters. In the 2020s, Ozone has advanced scalability through features like improved multi-tenancy in version 1.4 (2022) for handling exabyte-scale deployments and Ozone 2.0 (2025), which introduces enhanced Recon monitoring for real-time fault detection and a modernized UI for managing large-scale recoveries. These evolutions, driven by big data and cloud-native demands, have solidified distributed fault-tolerant file systems as essential for resilient, petabyte-to-exabyte storage in analytics and AI workloads.[63][64]

Distributed parallel file systems

Distributed parallel file systems are designed to enable high-throughput input/output operations across multiple clients and storage nodes simultaneously, making them particularly suitable for high-performance computing (HPC) environments where large-scale data access is required. These systems typically employ data striping to distribute file contents across multiple storage nodes, allowing parallel reads and writes that scale bandwidth with the number of clients. Collective operations, such as coordinated I/O from multiple processes, further optimize performance by reducing contention and enabling efficient data aggregation.[65][66] One of the earliest examples is PVFS (Parallel Virtual File System), initially developed in the late 1990s by researchers at Argonne National Laboratory, Clemson University, and the Ohio Supercomputer Center to support parallel I/O on Linux clusters. PVFS2, released in the early 2000s, introduced advanced features like integration with ROMIO, an implementation of the MPI-IO standard, which facilitates high-performance parallel I/O for Message Passing Interface (MPI) applications commonly used in scientific computing. This integration allows PVFS to handle collective I/O operations efficiently, such as two-phase I/O patterns that minimize network overhead. Evolving into OrangeFS in the 2010s, PVFS laid foundational work for cluster-based storage and was widely used in early HPC deployments.[66][67][68] Lustre, originating in 2001 from Cluster File Systems (now part of Whamcloud under Intel), emerged as a leading solution for supercomputing with its object-based storage architecture, separating metadata handling via dedicated metadata servers (MDS) from data storage on object storage servers (OSS). Data striping in Lustre distributes file extents across OSS nodes, enabling scalable bandwidth that grows linearly with added storage. A key advancement is the Distributed Namespace Environment (DNE), introduced in Lustre 2.x, which supports multiple metadata targets (MDTs) to distribute namespace operations and enhance scalability for millions of files. Lustre has powered over 50% of the top supercomputers on the TOP500 list throughout the 2020s, including systems like Frontier and El Capitan.[69][70][71][72] BeeGFS, developed by Fraunhofer ITWM and first released in 2014 (initially as FhGFS), offers a modular design with separate services for metadata, storage, and management, allowing flexible scaling across multiple metadata servers for high concurrency. Its data striping and self-healing capabilities support parallel access, with bandwidth scaling through additional nodes. BeeGFS incorporates buddy-mirroring for metadata, pairing servers to replicate directory structures and enable rapid failover without interrupting client operations. This feature enhances availability in dynamic HPC workloads.[73][74][75] More recently, DAOS (Distributed Asynchronous Object Storage), initiated by Intel in 2012, represents a next-generation approach using a key-value store model optimized for non-volatile memory (NVM) like Intel Optane, shifting from traditional POSIX semantics to asynchronous, fine-grained access for HPC and AI workloads. DAOS supports parallel I/O through object-based distribution and collective operations via integrations like MPI-IO and HDF5. POSIX access is provided via the user-space libdfs library rather than mainline kernel integration. DAOS emphasizes low-latency access with features like inline compression and encryption as optional extensions.[76][77][78][79][80]

Distributed parallel fault-tolerant file systems

Distributed parallel fault-tolerant file systems integrate parallel data striping across multiple nodes with built-in redundancy mechanisms such as replication or erasure coding, enabling high-throughput access while ensuring data availability and automatic recovery from failures through self-healing processes like scrubbing and rebalancing. These systems are designed for high-performance computing (HPC) environments, where large-scale data operations require both scalability and resilience against node or disk failures. Unlike purely parallel systems, they incorporate native fault-tolerance to maintain operations during hardware issues, often using decentralized algorithms for data placement to avoid single points of failure.[81][82][83] Ceph, originally developed as an open-source project starting in 2004 and first released in 2010 by Inktank (later acquired by Red Hat), exemplifies this integration through its Reliable Autonomic Distributed Object Store (RADOS) layer, which supports parallel client access to striped objects. It employs the CRUSH (Controlled Replication Under Scalable Hashing) algorithm for pseudo-random data placement and load balancing across object storage daemons (OSDs), with fault-tolerance achieved via configurable replication (e.g., three replicas) or erasure coding profiles that distribute data chunks with parity for recovery. OSDs monitor cluster health using periodic heartbeats to detect failures, triggering automatic rebalancing and self-healing via deep scrubbing to repair inconsistencies. Ceph's evolution includes the BlueStore backend, introduced in the Luminous release (2017) and default by 2018, which stores objects directly on block devices for improved performance and reduced overhead compared to the older FileStore. The Pacific release (v16.2.0, March 2021) enhanced multi-site replication for RADOS Gateway (RGW) with better sync policies and data log handling, while the Quincy release (v17.2.0, April 2022) optimized OSD scheduling with the mClock scheduler for balanced I/O prioritization and improved RBD (RADOS Block Device) diff-iterate performance.[84][81][85][86][87] IBM Spectrum Scale, formerly known as General Parallel File System (GPFS) and introduced in 1998, provides parallel I/O striping across network-shared disks with fault-tolerance through policy-based replication and, since version 4.1 (2014), integrated erasure coding for efficient data protection in large clusters. It supports non-stop availability (NSA) via dynamic node failover and quorum mechanisms that ensure continuous access even during server outages, with self-healing accomplished through automatic disk lease renewal and metadata replication. In the 2020s, Spectrum Scale extended its capabilities to cloud environments, including deployments on Google Cloud Marketplace (announced 2023) and AWS, allowing hybrid on-premises and cloud configurations with seamless data tiering.[82][88][89][90] Panasas PanFS, developed in the early 2010s as part of the company's ActiveStor appliances (building on foundations from 1999), combines parallel file striping with N+2 erasure coding for fault tolerance, where files are distributed across blade servers with direct-attached storage for low-latency access. It features active-active Director Nodes in a replicated set for metadata management, ensuring high availability through automatic failover and load balancing, while Storage Nodes in BladeSets handle data with self-healing via periodic scrubbing and parallel reconstruction from parity data. This architecture supports scaling to thousands of nodes without performance degradation, targeting HPC workloads with resilient, object-based storage.[83][91]

Peer-to-peer file systems

Peer-to-peer file systems operate without a central authority, relying instead on decentralized networks where participating nodes function as both clients and servers to store, retrieve, and distribute data. These systems typically employ distributed hash tables (DHTs) for efficient lookup and routing of resources, enabling scalable discovery of content across the network without single points of failure. Content addressing forms a core principle, where files are identified and located by cryptographic hashes of their contents rather than location-based identifiers, promoting immutability, deduplication, and resistance to tampering. This architecture contrasts with client-server models by decentralizing control and storage, fostering voluntary participation among peers, unlike managed cluster-based distributed systems.[92][93] The roots of peer-to-peer file systems trace back to early file-sharing applications like Napster, launched in June 1999 by Shawn Fanning, which popularized decentralized music distribution despite its hybrid centralized indexing. Subsequent advancements shifted toward fully decentralized designs, emphasizing fault tolerance and anonymity. A prominent example is the InterPlanetary File System (IPFS), developed by Protocol Labs and first specified in 2014, which uses content-addressed blocks organized in Merkle directed acyclic graphs (DAGs) for versioning and efficient sharing. IPFS employs S/Kademlia DHT for peer coordination and introduces content identifiers (CIDs) derived from multihashes to ensure data immutability and verifiable integrity. Building on IPFS, Filecoin emerged in 2017 as a decentralized storage economy, incentivizing miners with tokens to provide persistent storage via proof-of-replication and proof-of-spacetime mechanisms, with its mainnet launching in October 2020 to support Web3 applications.[94][93][95] Tahoe-LAFS, initiated in 2007, exemplifies secure peer-to-peer storage through erasure coding, which divides files into verifiable shares distributed across nodes for fault tolerance and confidentiality. It enforces the principle of least authority using capabilities for access control and convergent encryption to prevent unauthorized reconstruction, ensuring data integrity even if some shares are compromised. Freenet, introduced in 2000 by Ian Clarke and colleagues, prioritizes censorship resistance in its peer-to-peer design, utilizing adaptive routing over a distributed data store to anonymize requests and insertions. In darknet mode, Freenet restricts connections to trusted peers for enhanced privacy, routing data through chains that obscure origins and destinations while maintaining availability.

Special-purpose file systems

Pseudo file systems

Pseudo file systems are virtual file systems that provide an interface to kernel data structures and system abstractions by presenting information as files, without relying on physical storage devices. These file systems generate content dynamically in memory or on-the-fly, ensuring no data persistence across reboots, and are primarily used for system monitoring, debugging, configuration, and exposing runtime kernel state to user-space applications. They integrate with the virtual file system (VFS) layer in Unix-like operating systems to allow standard file operations like reading and writing for accessing or modifying kernel parameters. In Linux, one of the earliest and most prominent examples is procfs, introduced in kernel version 0.97.3 in September 1992, which mounts at /proc and exposes process-specific information along with global system details. For instance, the /proc/[pid]/status file for a given process ID (pid) provides runtime statistics such as memory usage, CPU times, and signal handling details, enabling tools like ps and top to query live kernel data without direct system calls. Procfs originated from earlier Unix implementations like SVR4 and 4.4BSD but was adapted for Linux to facilitate debugging and administration.[96] Another key example is sysfs, added in the Linux 2.6.0 kernel release in December 2003 during the 2.5 development series, mounted at /sys to represent the device hierarchy and kernel subsystems in a tree structure. Sysfs utilizes kobjects—kernel objects that manage device lifecycles and relationships—to export attributes and enable event notifications via uevents, which user-space tools like udev use to handle hotplug events. This design replaced parts of procfs for device-related data, promoting a more organized exposure of hardware and driver information.[97] Devfs, introduced in Linux kernel 2.3.46 in February 2000 and mounted at /dev, dynamically populates device nodes based on driver registrations, eliminating the need for static /dev entries and supporting on-demand creation of special files for devices. It aimed to simplify device management in large systems but was later deprecated in favor of udev due to policy and permission limitations.[98] Tmpfs, a RAM-based temporary file system introduced in Linux 2.4 in January 2001 as a successor to ramfs, allows files to reside in virtual memory and supports swapping to disk if needed, making it suitable for short-lived data like /tmp contents. Unlike pure in-memory systems, tmpfs enforces size limits to prevent memory exhaustion.[99] For debugging purposes, debugfs was added in Linux kernel 2.6.10-rc3 in December 2004, providing a simple, unstructured way for developers to export arbitrary kernel data to /sys/kernel/debug without the constraints of procfs or sysfs. It is commonly used with tracing tools like ftrace, introduced in 2008, where debugfs files under /sys/kernel/debug/tracing control function tracing, event recording, and latency analysis to diagnose kernel performance issues.[100][101] In the Plan 9 operating system, the /srv directory serves a similar role by binding network services and file servers as file system resources, allowing abstractions like remote storage or protocol handlers to appear as local files, emphasizing Plan 9's "everything is a file" philosophy for service integration.[102]

Compressed file systems

Compressed file systems apply data compression techniques transparently to files or blocks during storage, reducing disk space usage at the expense of increased CPU overhead for compression and decompression operations. These systems typically employ inline compression, where data is compressed on write and decompressed on read, often using algorithms such as LZ4, LZO, Zlib, or Zstandard (Zstd), making the process invisible to applications and users. Compression occurs at the file or block level, with metadata stored to track compressed extents, enabling efficient random access without full decompression.[103][104][105] The concept of compressed file systems emerged in the late 1990s and early 2000s to address storage constraints in embedded and resource-limited environments. One of the earliest examples is cramfs, introduced around 2000 for Linux kernels, designed as a simple, read-only compressed file system using Zlib for page-level compression to support small, static images like boot partitions. It prioritized minimal memory footprint and fast mounting but lacked support for larger block sizes or advanced features, achieving typical compression ratios of 2:1 to 3:1 for mixed data. Cramfs paved the way for more sophisticated implementations by demonstrating the viability of transparent compression in kernel space. SquashFS, released in October 2002 as a highly compressed read-only file system for Linux, built on cramfs by supporting larger block sizes up to 64 KB and multiple compression algorithms including Zlib, LZ4, LZO, XZ, and Zstd. It compresses not only files but also inodes and directories, enabling up to 4x compression ratios for image-heavy data sets like those in live CDs or embedded systems. SquashFS is widely used for read-only partitions, including system images in some Android releases starting with Nougat (2016), where it helps minimize flash storage needs. Its design allows mounting as a standard file system, with decompression handled on-demand for efficiency.[103][106] NTFS, Microsoft's file system introduced in 1993 with Windows NT 3.1 and enhanced with compression in NTFS 1.1 (1994 via Windows NT 3.51), supports per-file and per-folder compression using the LZNT1 algorithm, a variant of Lempel-Ziv suitable for cluster sizes up to 4 KB. Compression is applied transparently to qualifying files (those smaller than 16 clusters or explicitly enabled), integrating with sparse file support to avoid allocating space for zero-filled regions, which further optimizes storage for partially empty files. This feature reduces disk usage for text and compressible data without requiring user intervention, though it incurs CPU costs during I/O.[105][107] Btrfs, a modern copy-on-write file system for Linux first proposed in 2007 and stabilized by 2013, added transparent compression in kernel 2.6.38 (March 2011), initially supporting Zlib and LZO algorithms that compress 128 KiB extents in parallel. Zstd integration arrived in kernel 4.14 (November 2017), offering better speed-ratio trade-offs and levels from 1 to 15, with auto-detection heuristics that skip compression for low-entropy data to avoid overhead. Btrfs compression extends file system lifespan on SSDs by reducing write amplification, as smaller compressed blocks mean fewer physical writes. It can be enabled at mount time, per file, or via defragmentation for existing data.[104]

Encrypted file systems

Encrypted file systems integrate cryptographic mechanisms to protect data at rest, ensuring confidentiality against unauthorized access. These systems typically employ symmetric encryption algorithms such as AES in XTS mode, which provides strong security for block-level data by treating the entire storage volume or individual files as encrypted containers. They can be implemented as full-volume encryption, securing an entire disk or partition, or as per-file encryption, applying protection selectively to specific data objects. This distinction allows for flexible deployment in environments ranging from personal devices to enterprise storage, with full-volume approaches often used for full-disk encryption (FDE) to simplify management. One foundational example is LUKS (Linux Unified Key Setup), introduced in 2004 as part of the dm-crypt device mapper in the Linux kernel, which stores an encrypted header containing multiple key slots for passphrase-based access. LUKS supports AES-XTS and other ciphers, enabling seamless integration with underlying file systems like ext4 without modifying their structure. It evolved from earlier tools and has become a standard for Linux distributions. Stacked encrypted file systems, which layer encryption atop existing file systems, include eCryptfs, released in 2006 for Linux and built on the ext4 file system to provide per-file and per-directory encryption. eCryptfs uniquely encrypts filenames to obscure directory structures, using AES with a derived key from user passphrases, and is designed for enterprise use cases like email and home directory protection. Similarly, EncFS, developed in the early 2000s using the FUSE (Filesystem in Userspace) framework, offers user-space encryption for directories on Unix-like systems, employing AES in CBC mode for file contents while allowing transparent mounting. A native kernel-level approach is provided by fscrypt, introduced in Linux kernel 4.1 in 2015, which enables filesystem-level encryption for supported filesystems including ext4, f2fs, and ubifs. fscrypt allows transparent encryption of files and directories using AES-256-XTS or AES-256-CTS, with keys managed via the kernel keyring, offering better performance than user-space solutions by avoiding context switches. It is commonly used in Android for file-based encryption since version 9 (2018).[108] Microsoft's BitLocker, launched in 2007 with Windows Vista, provides full-volume encryption for NTFS volumes using AES in CBC or XTS modes, integrated with the Trusted Platform Module (TPM) for hardware-bound key protection. It supports both fixed and removable drives, enforcing policy-based encryption in enterprise settings via Active Directory. On the cross-platform front, VeraCrypt, forked from TrueCrypt in 2013, enables container-based or full-disk encryption with AES, Serpent, or Twofish ciphers, featuring hidden volumes that embed deniable encryption layers within larger containers. The evolution of encrypted file systems traces back to the 1990s with tools like PGPdisk, part of the Pretty Good Privacy suite, which introduced container-based encryption for Windows and Macintosh using IDEA and later AES algorithms. This paved the way for modern FDE standards, such as those outlined in NIST SP 800-88 for media sanitization and protection.

File system interfaces

File system interfaces provide standardized protocols and application programming interfaces (APIs) that enable applications to interact with underlying storage systems in an abstract manner, decoupling software from specific file system implementations for enhanced portability and flexibility. The conceptual foundation traces back to the Multics operating system in the 1960s, where early designs introduced hierarchical file structures and access controls as part of a unified resource management model, influencing subsequent abstractions in time-sharing systems.[109] By the 1990s, operating systems like Linux adopted the Virtual File System (VFS) layer, a kernel-level abstraction that translates generic file operations into implementation-specific calls, allowing multiple file systems (e.g., ext4, NTFS) to coexist transparently and promoting cross-platform compatibility.[110] POSIX, ratified in 1988 as IEEE Std 1003.1, established a foundational API for Unix-like systems, defining core semantics for file operations such as open(), read(), and write(), which ensure atomicity and sequential consistency in data access—meaning that reads following writes must reflect the updated content without intermediate buffering ambiguities.[111][112] In contrast, the Windows API uses functions like CreateFile() to open or create files and devices, returning handles for subsequent I/O operations via ReadFile() and WriteFile(), with built-in support for asynchronous modes and security descriptors to manage access in a Windows NT kernel environment.[113] The Plan 9 operating system from Bell Labs extended this paradigm with the 9P protocol, treating all resources (files, devices, networks) as file-like entities in a unified namespace, where pathnames enable seamless local and remote access without distinct protocols for different resource types.[114] Filesystem in Userspace (FUSE), introduced in 2005 for Linux, allows non-privileged users to implement custom file systems in user space, bridging kernel VFS calls to user-level daemons via a protocol that handles operations like mounting NTFS-3G for read-write support on Linux; however, this incurs overhead from context switches and data copying between kernel and user spaces, potentially reducing performance by up to 83% for I/O-intensive workloads compared to kernel-native implementations.[115][116] Modern evolutions include io_uring, added to the Linux kernel in 2019, which provides an efficient asynchronous I/O interface using shared ring buffers to minimize syscalls and enable batched submissions/completions for file operations, significantly improving scalability for high-throughput applications.[117] In the 2020s, eBPF (extended Berkeley Packet Filter) hooks have emerged for file system monitoring, attaching lightweight programs to kernel tracepoints like VFS entry/exit to observe events such as file opens or modifications in real-time with minimal overhead, aiding in security auditing and performance analysis.[118]

References

  1. Understanding File Systems - Kingston Technology
  2. Types of File Systems - Oracle Help Center
  3. Chapter 1. Overview of available file systems | 8
  4. [PDF] A Comparison of FFS Disk Allocation Policies - USENIX
  5. Overview of FAT, HPFS, and NTFS File Systems - Windows Client
  6. [PDF] The new ext4 filesystem: current status and future plans
  7. ext4 General Information - The Linux Kernel documentation
  8. NTFS overview - Microsoft Learn
  9. [PDF] The structure of the Reiser file system
  10. Reiserfs going away in 2025 - LWN.net
  11. [PDF] Anatomy of Linux journaling file systems - IBM
  12. Introduction to ZFS — openzfs latest documentation
  13. History - - OpenZFS
  14. Introduction — BTRFS documentation - Read the Docs
  15. A short history of btrfs - LWN.net
  16. bcachefs
  17. Bcachefs removed from the mainline kernel - LWN.net
  18. [PDF] F2FS: A New File System for Flash Storage - USENIX
  19. [PDF] Extending the Lifetime of Flash-based Storage through Reducing ...
  20. YAFFS: the NAND-specific flash file system - Introductory Article
  21. How Yaffs works | Yaffs - A Flash File System for embedded use
  22. Samsung's F2FS filesystem - LWN.net
  23. Apple File System Guide
  24. Role of Apple File System
  25. exFAT File System Specification - Win32 apps - Microsoft Learn
  26. EROFS: A Compression-friendly Readonly File System for ... - USENIX
  27. EROFS - Enhanced Read-Only File System
  28. EROFS, exFAT & EXT4 File-System Updates Arrive For Linux 5.19
  29. Record-based File I/O
  30. What is VSAM? - IBM
  31. VSAM data sets: KSDS, ESDS, and RRDS - IBM
  32. [PDF] VSAM Demystified - IBM Redbooks
  33. ISAM in Database - GeeksforGeeks
  34. ISAM and B+ trees | Yanbin's Blog
  35. The Coda Distributed File System
  36. [PDF] Disconnected Operation in the Coda File System - People @EECS
  37. History of Databases: Relational, NoSQL, Distributed SQL
  38. Distributed transaction processing with Encina
  39. [PDF] Shared (Disk) File Systems - MSST
  40. Distributed Lock Manager for Distributed File System in Shared-Disk ...
  41. Chapter 24. Using shared storage with virtual disk images
  42. Cluster logical volume manager (Cluster LVM) | SLE HA 15 SP7
  43. [PDF] The GFS2 Filesystem
  44. Global File System 2 | Red Hat Enterprise Linux | 7
  45. History of OCFS2 - IBM
  46. Investigating Virtual Machine file locks on ESXi Host(s)
  47. RFC 1094 - NFS: Network File System Protocol specification
  48. Network file systems and Linux - IBM Developer
  49. Microsoft SMB Protocol and CIFS Protocol Overview - Win32 apps
  50. History of SMB - Samba.org
  51. Overview of file sharing using the SMB 3 protocol in Windows Server
  52. SMB over QUIC | Microsoft Learn
  53. Check your network backups and shares, as AFP is being removed
  54. Apple Filing Protocol client deprecated as of macOS Sequoia 15.5.0
  55. [PDF] Plan 9 File Protocol, 9P
  56. v9fs: Plan 9 Resource Sharing for Linux
  57. Erasure code - Ceph Documentation
  58. Rack Awareness - Apache Hadoop 3.4.2
  59. [PDF] The Google File System
  60. [PDF] The Hadoop Distributed File System - cs.wisc.edu
  61. [PDF] Finding a needle in Haystack: Facebook's photo storage - USENIX
  62. Architecture - Gluster Docs
  63. Ceph.io — Technology
  64. Architecture - Ceph Documentation
  65. Apache Ozone
  66. The Apache Software Foundation Announces Apache Ozone™ 2.0.0
  67. About the Lustre® File System
  68. PVFS: Parallel Virtual File System | Argonne National Laboratory
  69. [PDF] PVFS: A Parallel File System for Linux Clusters
  70. PVFS: A Parallel File System for Linux Clusters - USENIX
  71. [PDF] Lustre Technical Project Summary – version 2, July 29, 2001
  72. Lustre* Software Release 2.x - Documentation
  73. Understanding Lustre Internals
  74. Lustre Stronger-than-Ever - TOP500
  75. Fraunhofer Parallel File System – BeeGFS
  76. New Release of FhGFS is Now Available - HPCwire
  77. Mirroring — BeeGFS Documentation 8.2
  78. DAOS
  79. [PDF] DAOS: Data Access-aware Operating System - Amazon Science
  80. DAOS Foundation Launches to Broaden Governance and ...
  81. Architecture — Ceph Documentation
  82. https://www.ibm.com/docs/en/storage-scale/5.1.0?topic=overview-ibm-spectrum-scale
  83. [PDF] PanFS 9: Architectural Overview
  84. [PDF] Ceph: A Scalable, High-Performance Distributed File System
  85. [PDF] IBM Storage Ceph Concepts and Architecture Guide
  86. Quincy — Ceph Documentation
  87. Pacific — Ceph Documentation
  88. [PDF] IBM Spectrum Scale Erasure Code Edition - IBM Redbooks
  89. [PDF] Deploying IBM Spectrum Scale with NetApp E-Series Storage
  90. Announcing IBM Spectrum Scale is now available in Google Cloud
  91. Panasas New PanFS on the ActiveStor Ultra Storage System Now in ...
  92. [PDF] Peer-to-Peer Systems and Distributed Hash Tables - cs.Princeton
  93. [1407.3561] IPFS - Content Addressed, Versioned, P2P File System
  94. File Sharing and Copyright: Innovation Policy and the Economy
  95. None
  96. The /proc Filesystem - The Linux Kernel documentation
  97. [PDF] The sysfs Filesystem - The Linux Kernel Archives
  98. 9.3. Overview of Device and Module Handling - Linux From Scratch!
  99. tmpfs(5) - Linux manual page - man7.org
  100. DebugFS - The Linux Kernel documentation
  101. ftrace - Function Tracer - The Linux Kernel documentation
  102. Russ Cox: Plan 9 - PDOS-MIT
  103. Squashfs 4.0 Filesystem - The Linux Kernel documentation
  104. Compression — BTRFS documentation - Read the Docs
  105. File Compression and Decompression - Win32 apps | Microsoft Learn
  106. A/B (seamless) system updates - Android Open Source Project
  107. NTFS Compressed Files
  108. [PDF] Multics—The first seven - Reliable Computer Systems
  109. Overview of the Linux Virtual File System
  110. standards(7) - Linux manual page - man7.org
  111. read
  112. CreateFileA function (fileapi.h) - Win32 apps | Microsoft Learn
  113. The Use of Name Spaces in Plan 9
  114. FUSE — The Linux Kernel documentation
  115. [PDF] To FUSE or Not to FUSE: Performance of User-Space File Systems
  116. An Introduction to the io_uring Asynchronous I/O Framework | linux
  117. File Monitoring with eBPF and Tetragon (Part 1) - Isovalent
  118. DAOS Overview
  119. DAOS File System Documentation
User Avatar
No comments yet.