Hubbry Logo
Data deduplicationData deduplicationMain
Open search
Data deduplication
Community hub
Data deduplication
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Data deduplication
Data deduplication
Data deduplication
View on Wikipedia
from Wikipedia

In computing, data deduplication is a technique for eliminating duplicate copies of repeating data. Successful implementation of the technique can improve storage utilization, which may in turn lower capital expenditure by reducing the overall amount of storage media required to meet storage capacity needs. It can also be applied to network data transfers to reduce the number of bytes that must be sent.

The deduplication process requires comparison of data 'chunks' (also known as 'byte patterns') which are unique, contiguous blocks of data. These chunks are identified and stored during a process of analysis, and compared to other chunks within existing data. Whenever a match occurs, the redundant chunk is replaced with a small reference that points to the stored chunk. Given that the same byte pattern may occur dozens, hundreds, or even thousands of times (the match frequency is dependent on the chunk size), the amount of data that must be stored or transferred can be greatly reduced.[1][2]

A related technique is single-instance (data) storage, which replaces multiple copies of content at the whole-file level with a single shared copy. While possible to combine this with other forms of data compression and deduplication, it is distinct from newer approaches to data deduplication (which can operate at the segment or sub-block level).

Deduplication is different from data compression algorithms, such as LZ77 and LZ78. Whereas compression algorithms identify redundant data inside individual files and encodes this redundant data more efficiently, the intent of deduplication is to inspect large volumes of data and identify large sections – such as entire files or large sections of files – that are identical, and replace them with a shared copy.

Functioning principle

[edit]

For example, a typical email system might contain 100 instances of the same 1 MB (megabyte) file attachment. Each time the email platform is backed up, all 100 instances of the attachment are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; the subsequent instances are referenced back to the saved copy for deduplication ratio of roughly 100 to 1. Deduplication is often paired with data compression for additional storage saving: Deduplication is first used to eliminate large chunks of repetitive data, and compression is then used to efficiently encode each of the stored chunks.[3]

In computer code, deduplication is done by, for example, storing information in variables so that they don't have to be written out individually but can be changed all at once at a central referenced location. Examples are CSS classes and named references in MediaWiki.

Benefits

[edit]

Storage-based data deduplication reduces the amount of storage needed for a given set of files. It is most effective in applications where many copies of very similar or even identical data are stored on a single disk. In the case of data backups, which routinely are performed to protect against data loss, most data in a given backup remain unchanged from the previous backup. Common backup systems try to exploit this by omitting (or hard linking) files that haven't changed or storing differences between files. Neither approach captures all redundancies, however. Hard-linking does not help with large files that have only changed in small ways, such as an email database; differences only find redundancies in adjacent versions of a single file (consider a section that was deleted and later added in again, or a logo image included in many documents).

In-line network data deduplication is used to reduce the number of bytes that must be transferred between endpoints, which can reduce the amount of bandwidth required. See WAN optimization for more information.

Virtual servers and virtual desktops benefit from deduplication because it allows nominally separate system files for each virtual machine to be coalesced into a single storage space. At the same time, if a given virtual machine customizes a file, deduplication will not change the files on the other virtual machines—something that alternatives like hard links or shared disks do not offer. Backing up or making duplicate copies of virtual environments is similarly improved.

Classification

[edit]

Post-process versus in-line deduplication

[edit]

Deduplication may occur "in-line", as data is flowing, or "post-process" after it has been written.

With post-process deduplication, new data is first stored on the storage device and then a process at a later time will analyze the data looking for duplication. The benefit is that there is no need to wait for the hash calculations and lookup to be completed before storing the data, thereby ensuring that store performance is not degraded. Implementations offering policy-based operation can give users the ability to defer optimization on "active" files, or to process files based on type and location. One potential drawback is that duplicate data may be unnecessarily stored for a short time, which can be problematic if the system is nearing full capacity.

Alternatively, deduplication hash calculations can be done in-line: synchronized as data enters the target device. If the storage system identifies a block which it has already stored, only a reference to the existing block is stored, rather than the whole new block.

The advantage of in-line deduplication over post-process deduplication is that it requires less storage and network traffic, since duplicate data is never stored or transferred. On the negative side, hash calculations may be computationally expensive, thereby reducing the storage throughput. However, certain vendors with in-line deduplication have demonstrated equipment which performs in-line deduplication at high rates.

Post-process and in-line deduplication methods are often heavily debated.[4][5]

Data formats

[edit]

The SNIA Dictionary identifies two methods:[2]

  • content-agnostic data deduplication - a data deduplication method that does not require awareness of specific application data formats.
  • content-aware data deduplication - a data deduplication method that leverages knowledge of specific application data formats.

Source versus target deduplication

[edit]

Another way to classify data deduplication methods is according to where they occur. Deduplication occurring close to where data is created, is referred to as "source deduplication". When it occurs near where the data is stored, it is called "target deduplication".

Source deduplication ensures that data on the data source is deduplicated. This generally takes place directly within a file system. The file system will periodically scan new files creating hashes and compare them to hashes of existing files. When files with same hashes are found then the file copy is removed and the new file points to the old file. Unlike hard links however, duplicated files are considered to be separate entities and if one of the duplicated files is later modified, then using a system called copy-on-write a copy of that changed file or block is created. The deduplication process is transparent to the users and backup applications. Backing up a deduplicated file system will often cause duplication to occur resulting in the backups being bigger than the source data.[6][7]

Source deduplication can be declared explicitly for copying operations, as no calculation is needed to know that the copied data is in need of deduplication. This leads to a new form of "linking" on file systems called the reflink (Linux) or clonefile (MacOS), where one or more inodes (file information entries) are made to share some or all of their data. It is named analogously to hard links, which work at the inode level, and symbolic links that work at the filename level.[8] The individual entries have a copy-on-write behavior that is non-aliasing, i.e. changing one copy afterwards will not affect other copies.[9] Microsoft's ReFS also supports this operation.[10]

Target deduplication is the process of removing duplicates when the data was not generated at that location. Example of this would be a server connected to a SAN/NAS, The SAN/NAS would be a target for the server (target deduplication). The server is not aware of any deduplication, the server is also the point of data generation. A second example would be backup. Generally this will be a backup store such as a data repository or a virtual tape library.

Deduplication methods

[edit]

One of the most common forms of data deduplication implementations works by comparing chunks of data to detect duplicates. For that to happen, each chunk of data is assigned an identification, calculated by the software, typically using cryptographic hash functions. In many implementations, the assumption is made that if the identification is identical, the data is identical, even though this cannot be true in all cases due to the pigeonhole principle; other implementations do not assume that two blocks of data with the same identifier are identical, but actually verify that data with the same identification is identical.[11] If the software either assumes that a given identification already exists in the deduplication namespace or actually verifies the identity of the two blocks of data, depending on the implementation, then it will replace that duplicate chunk with a link.

Once the data has been deduplicated, upon read back of the file, wherever a link is found, the system simply replaces that link with the referenced data chunk. The deduplication process is intended to be transparent to end users and applications.

Commercial deduplication implementations differ by their chunking methods and architectures.

  • Chunking: In some systems, chunks are defined by physical layer constraints (e.g. 4 KB block size in WAFL). In some systems only complete files are compared, which is called single-instance storage or SIS. The most intelligent (but CPU intensive) method to chunking is generally considered to be sliding-block, also called Content-Defined Chunking. In sliding block, a window is passed along the file stream to seek out more naturally occurring internal file boundaries.
  • Client backup deduplication: This is the process where the deduplication hash calculations are initially created on the source (client) machines. Files that have identical hashes to files already in the target device are not sent, the target device just creates appropriate internal links to reference the duplicated data. The benefit of this is that it avoids data being unnecessarily sent across the network thereby reducing traffic load.
  • Primary storage and secondary storage: By definition, primary storage systems are designed for optimal performance, rather than lowest possible cost. The design criteria for these systems is to increase performance, at the expense of other considerations. Moreover, primary storage systems are much less tolerant of any operation that can negatively impact performance. Also by definition, secondary storage systems contain primarily duplicate, or secondary copies of data. These copies of data are typically not used for actual production operations and as a result are more tolerant of some performance degradation, in exchange for increased efficiency.

To date, data deduplication has predominantly been used with secondary storage systems. The reasons for this are two-fold: First, data deduplication requires overhead to discover and remove the duplicate data. In primary storage systems, this overhead may impact performance. The second reason why deduplication is applied to secondary data, is that secondary data tends to have more duplicate data. Backup application in particular commonly generate significant portions of duplicate data over time.

Data deduplication has been deployed successfully with primary storage in some cases where the system design does not require significant overhead, or impact performance.

Single instance storage

[edit]

Single-instance storage (SIS) is a system's ability to take multiple copies of content objects and replace them by a single shared copy. It is a means to eliminate data duplication and to increase efficiency. SIS is frequently implemented in file systems, email server software, data backup, and other storage-related computer software. Single-instance storage is a simple variant of data deduplication. While data deduplication may work at a segment or sub-block level, single instance storage works at the object level, eliminating redundant copies of objects such as entire files or email messages.[12]

Single-instance storage can be used alongside (or layered upon) other data duplication or data compression methods to improve performance in exchange for an increase in complexity and for (in some cases) a minor increase in storage space requirements.

Drawbacks and concerns

[edit]

One method for deduplicating data relies on the use of cryptographic hash functions to identify duplicate segments of data. If two different pieces of information generate the same hash value, this is known as a collision. The probability of a collision depends mainly on the hash length (see birthday attack). Thus, the concern arises that data corruption can occur if a hash collision occurs, and additional means of verification are not used to verify whether there is a difference in data, or not. Both in-line and post-process architectures may offer bit-for-bit validation of original data for guaranteed data integrity. The hash functions used include standards such as SHA-1, SHA-256, and others.

The computational resource intensity of the process can be a drawback of data deduplication. To improve performance, some systems utilize both weak and strong hashes. Weak hashes are much faster to calculate but there is a greater risk of a hash collision. Systems that utilize weak hashes will subsequently calculate a strong hash and will use it as the determining factor to whether it is actually the same data or not. Note that the system overhead associated with calculating and looking up hash values is primarily a function of the deduplication workflow. The reconstitution of files does not require this processing and any incremental performance penalty associated with re-assembly of data chunks is unlikely to impact application performance.

Another concern is the interaction of compression and encryption. The goal of encryption is to eliminate any discernible patterns in the data. Thus encrypted data cannot be deduplicated, even though the underlying data may be redundant.

Although not a shortcoming of data deduplication, there have been data breaches when insufficient security and access validation procedures are used with large repositories of deduplicated data. In some systems, as typical with cloud storage,[citation needed] an attacker can retrieve data owned by others by knowing or guessing the hash value of the desired data.[13]

Implementations

[edit]

Deduplication is implemented in some filesystems such as in ZFS or Write Anywhere File Layout and in different disk arrays models.[citation needed] It is a service available on both NTFS and ReFS on Windows servers.

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Data deduplication

View on Grokipedia
from Grokipedia
Data deduplication is a specialized data reduction technique that systematically identifies and eliminates redundant copies of across storage systems, ensuring that only unique instances are retained while maintaining full and application functionality. By replacing duplicates with pointers or references to the original , it significantly optimizes storage capacity without compromising . Data deduplication emerged in the early 2000s as storage needs grew with the shift from tape to disk-based backups. Commercial implementations began around 2003, with companies like Data Domain pioneering inline and post-process methods for backup systems. The process typically involves dividing data into fixed or variable-sized chunks, generating cryptographic hashes or fingerprints for each chunk to detect duplicates, and storing only unique chunks in a repository while updating metadata to reference them for repeated occurrences. Key methods include file-level deduplication, which targets entire identical files; block-level deduplication, which operates on smaller data segments for finer granularity; and variable-length chunking to better handle fragmented data. Deduplication can occur inline (real-time during data ingestion to prevent writing duplicates) or post-process (after data is stored, allowing for initial write speed but requiring additional analysis). Additionally, it may be performed at the source (client-side before transmission) or target (server-side after receipt), influencing network efficiency and computational overhead. This technique delivers notable benefits, such as reducing storage needs by 30-95% depending on data redundancy—higher ratios in virtual machine images or backups—and lowering costs for hardware, energy, and maintenance. It enhances backup and recovery speeds by minimizing data volumes transferred over networks and improves overall system performance in resource-constrained environments. Common applications span virtualized infrastructures like VDI deployments, file servers for user shares, backup targets, cloud archives, and databases, where redundant data from emails, documents, or logs is prevalent. Despite these advantages, implementation requires balancing computational demands, as hashing and indexing can introduce latency, and safeguards against rare hash collisions to preserve data fidelity.

Introduction

Definition and Purpose

Data deduplication is a reduction technique that eliminates exact duplicate copies of repeating , optimizing storage by retaining only a single unique instance of the while referencing it multiple times for subsequent occurrences. This process transparently identifies and removes redundancy without altering the original 's fidelity or integrity, enabling efficient management of large-scale datasets. The primary purpose of data deduplication is to reduce storage requirements and associated costs in environments with high , such as backups and archives, where exponential data growth—for example, reaching approximately 64 zettabytes in 2020 and projected to reach 181 zettabytes by —demands scalable solutions. It also improves speeds, enhances data transfer efficiency over networks, and supports overall resource optimization by minimizing physical storage footprints. In the context of , deduplication facilitates the capture, storage, and retention phases by replacing multiple data copies with a single instance, thereby streamlining data handling across its lifecycle stages. Unlike data compression, which reduces within unique data segments by encoding them more efficiently, deduplication entirely removes duplicate instances across datasets, often achieving complementary savings when combined. This distinction positions deduplication as a targeted approach for inter-file or inter-dataset duplicates rather than intra-file optimization. Single-instance storage serves as a key enabler, ensuring that only one copy occupies space while pointers maintain accessibility. In practice, data deduplication is commonly applied in enterprise storage environments, where identical files—such as user documents or virtual machine images—or data blocks frequently appear across multiple datasets, yielding space savings of 20% to 30% on average and up to 70% in highly redundant scenarios like high-performance computing systems.

Historical Development

The origins of data deduplication trace back to the 1990s, when it emerged as a technique in backup and archiving applications to address rising storage demands by eliminating redundant copies of entire files, known as file-level deduplication. Early implementations were primarily seen in backup software for tape libraries, where the focus was on reducing physical media usage during data replication and storage, driven by the limitations of tape-based systems in handling growing volumes of information. A pivotal milestone occurred in 2004 with Permabit Technology Corporation's introduction of its first commercial deduplication product, which included block-level deduplication capabilities by dividing files into smaller chunks to detect and eliminate duplicates at a sub-file , significantly improving efficiency over file-level methods. This innovation gained widespread adoption throughout the 2000s through key vendors, including , whose disk-based deduplication appliances revolutionized backup processes and led to its acquisition by EMC Corporation in 2009 for approximately $2.1 billion. Similarly, Diligent Technologies' ProtecTIER system, emphasizing high-performance deduplication for enterprise backups, was acquired by in 2008, further accelerating commercial integration. The technology evolved in the mid-2000s with a clear shift from file-level to block-level approaches, enabling more precise redundancy elimination and higher compression ratios in diverse environments. By the 2010s, deduplication integrated deeply with virtualization platforms, such as and , optimizing storage for virtual machine images and reducing overhead in and deployments. In the , advancements in inline processing—performing deduplication in real-time during —enhanced efficiency for high-velocity workloads, supported by improved algorithms and to minimize latency. In recent years, integration of and has further optimized deduplication algorithms for complex environments. This progression was propelled by exponential data growth following the post-2000 digital explosion, which multiplied storage needs, alongside persistent pressures from escalating hardware costs and the need for scalable solutions. Deduplication's ability to achieve substantial storage savings motivated its adoption, transforming it from a niche backup tool into a foundational element of modern data management.

Operational Principles

Core Functioning Mechanism

Data deduplication functions by systematically identifying and eliminating redundant within a storage system through a series of computational steps that ensure only unique instances are retained. The process begins with the of incoming streams, such as files or backups, which are then processed to detect and remove duplicates at the block level. This mechanism relies on breaking down the into manageable segments and using cryptographic techniques to verify , ultimately leading to significant storage optimization. The core steps involve several distinct phases. First, the ingested data is segmented into smaller units called chunks or blocks, which serve as the basic elements for detection. Next, each chunk undergoes fingerprinting, where a —typically a cryptographic hash such as or —is computed to represent its content precisely. These fingerprints are then compared against a repository of existing unique identifiers maintained in the system. If a fingerprint matches an entry in the repository, indicating a duplicate, the chunk is not stored anew; instead, it is replaced with a lightweight reference or pointer to the original unique instance. Conversely, if no match is found, the chunk is stored as a new unique entry, and its fingerprint is added to the repository. This ensures that redundant data is efficiently eliminated without altering the logical view of the data. Chunking methods form a critical part of this segmentation phase, with two primary approaches: fixed-size and variable-size blocking. Fixed-size chunking divides the data stream into uniform blocks of predetermined length, such as 4 KiB, which simplifies processing but may lead to lower deduplication due to boundary misalignments from insertions or deletions. Variable-size chunking, in contrast, determines chunk boundaries dynamically based on the data content, often targeting an average size like 8 KiB, which better preserves redundancies across similar datasets by avoiding shifts in block edges. To manage the mapping of duplicates to unique instances, deduplication systems employ reference structures such as metadata tables or indexes. These structures store fingerprints alongside reference counts that track how many times a unique chunk is referenced, enabling quick lookups and efficient garbage collection of unreferenced data. For instance, fingerprint indexes allow rapid duplicate detection, while metadata entries link file offsets to the corresponding unique chunks in storage. The effectiveness of this mechanism is quantified by the deduplication ratio, a key metric of storage efficiency. This ratio is calculated as follows: Deduplication Ratio=Total Data SizeUnique Data Size\text{Deduplication Ratio} = \frac{\text{Total Data Size}}{\text{Unique Data Size}} Here, the total data size represents the original volume before processing, and the unique data size is the reduced volume after eliminating redundancies (approximating the space for unique chunks plus minimal metadata). A ratio of 10:1, for example, signifies that only 10% of the original space is needed, achieving 90% savings. This process culminates in single-instance storage, where each unique chunk exists only once, referenced as needed by multiple data objects.

Single-Instance Storage Technique

Single-instance storage is a fundamental technique in data deduplication where only one physical copy of each unique —whether at the file or block level—is retained on storage media, while all duplicate instances are replaced by lightweight pointers or stubs that reference the single stored copy. This approach ensures that identical data does not consume redundant space, with uniqueness typically determined based on cryptographic hashes from the deduplication process. In implementation, unique data segments are organized into container files or dedicated deduplicated volumes that hold the sole instances, while a separate metadata structure maintains mappings from pointers to these containers. Metadata includes reference counts for each unique segment, which track the number of pointers referencing it; when a reference count reaches zero—such as after all duplicates are deleted—the segment becomes eligible for garbage collection to reclaim storage space. This reference counting mechanism enables efficient space management without immediate data loss risks. During read operations, the rehydration process reconstructs the original data by traversing the pointers or stubs in the metadata to fetch and assemble the corresponding segments from the container, ensuring transparent access as if the full data were stored conventionally. For instance, in NTFS with deduplication enabled, reparse points serve as these stubs, directing the file system to the chunk store where unique segments reside, allowing seamless file reconstruction. This technique integrates into file systems by embedding deduplication logic at the storage layer, such as in where block-level reference counts are updated in the deduplication table to manage shared data blocks across files. Similarly, in architectures supporting deduplication, the file system filter driver handles pointer resolution and metadata updates, fitting into the volume structure without altering the user-facing file organization.

Classifications

Processing Approaches: In-Line vs. Post-Process

Data deduplication can be performed using two primary processing approaches: in-line and post-process, which differ in the timing of duplicate detection and elimination relative to ingestion. In-line deduplication processes in real-time as it enters the storage system, identifying and removing duplicates before writing to disk. This approach typically involves hashes or fingerprints for incoming chunks and comparing them against an existing index to avoid storing redundant blocks immediately. In contrast, post-process deduplication writes all incoming to storage first and then scans it in batches to detect and eliminate duplicates afterward, reclaiming space through techniques like pointer updates or block rewriting. Both methods often rely on similar underlying algorithms, such as hash-based chunking, but apply them at different stages of the pipeline. In-line deduplication offers immediate space savings by reducing the initial storage , which is particularly beneficial in bandwidth-constrained environments like remote backups where only unique is written and replicated. For instance, systems like those described in early implementations achieve high throughput, such as 250 MB/s on single-drive setups, by leveraging sampling and locality to maintain efficient indexing without excessive RAM usage. However, this real-time processing imposes computational overhead during ingestion, potentially increasing latency and slowing write speeds due to on-the-fly hash computations and lookups. It demands robust hardware to avoid bottlenecks, making it suitable for with high , such as incremental backups where duplicate rates can exceed 90%. Post-process deduplication, by deferring the deduplication task, allows for faster initial writes since data is stored without immediate , minimizing impact on . This approach is implemented in systems that run scans during idle periods, updating metadata to consolidate duplicates and free space, which can be scheduled to optimize resource utilization. A key drawback is the temporary requirement for additional storage to hold full copies of data until processing completes, which can lead to higher I/O demands during the batch phase and risks of capacity exhaustion in space-limited setups. It is commonly used in general file storage scenarios where write speed is prioritized over instant efficiency, such as archival systems with variable workloads. The choice between in-line and post-process approaches involves trade-offs centered on , use, and workload characteristics. In-line methods save and bandwidth upfront but add latency to writes, ideal for high-redundancy streams to accelerate recovery times. Post-process techniques reduce write delays at the cost of extra temporary and later I/O spikes, fitting better for environments with ample initial capacity and less emphasis on real-time optimization. For example, in appliances, in-line processing can yield predictable and faster restores, while post-process may extend overall windows by up to 50% in some configurations due to deferred operations.

Deduplication Locations: Source vs. Target

Source deduplication, also known as client-side or source-side deduplication, involves performing the deduplication process on the originating device or client before transmission to the storage target. This approach identifies and eliminates redundant chunks locally, transmitting only unique along with metadata references to the target, thereby significantly reducing network traffic in scenarios such as remote s. For instance, in backup clients, source deduplication minimizes the volume of sent over wide-area networks, which is particularly advantageous for environments with limited bandwidth. In contrast, target deduplication, or server-side deduplication, applies the process at the destination storage system after the full dataset has been received and written. Here, the client transmits the entire original , and the target storage appliance or server then scans for duplicates to optimize long-term storage efficiency by storing only unique instances. This method simplifies the client-side workload, as no computational overhead for deduplication is imposed on the source device, but it requires transferring redundant data across the network, leading to higher bandwidth consumption. Hybrid approaches combine elements of both source and target deduplication, often implementing partial deduplication at the source to filter obvious redundancies before transmission, followed by more comprehensive deduplication at the target for finer-grained optimization. These methods balance network savings with storage efficiency, such as by exchanging metadata between source and target to avoid transmitting known duplicates while leveraging the target's resources for unresolved cases. Network implications favor source deduplication for bandwidth-constrained links, as it can achieve deduplication ratios of 10:1 or higher in highly redundant workloads, reducing data transfer by 90% or more. In-line processing is often paired with source deduplication to achieve immediate efficiency gains without buffering full datasets.

Data Handling Formats

Data deduplication operates at various granularities depending on the data handling format, which determines how redundancy is identified and eliminated across different structures. These formats include file-level, block-level, and byte-level approaches, each balancing efficiency, computational demands, and effectiveness in detecting duplicates. File-level deduplication identifies and stores only a single instance of identical whole files, replacing duplicates with references or pointers to the unique copy. This method is straightforward and incurs minimal processing overhead, as it relies on file metadata such as names, sizes, and hashes for comparison, but it cannot detect redundancy within a single file or across partially similar files. Block-level deduplication divides files into smaller fixed or variable-sized chunks, typically ranging from a few kilobytes, and eliminates duplicates at this sub-file level by storing unique blocks and reconstructing files via metadata. This approach captures intra-file redundancies and partial overlaps, such as in versioned documents where only portions change, offering greater space savings than file-level methods at the cost of increased . Byte-level deduplication provides the finest by scanning for exact duplicate byte sequences across the entire , without fixed boundaries, enabling detection of redundancies at the most precise scale. While this yields the highest potential deduplication ratios for highly similar , it demands significant computational resources due to the exhaustive byte-by-byte . The choice of format depends on the data environment: file-level deduplication suits archival systems with mostly static, identical files where simplicity is prioritized, while block-level deduplication is more applicable to scenarios involving versioned or incrementally modified files to handle partial changes effectively. Variable block sizing can further improve adaptability to diverse data formats by aligning chunks with content boundaries.

Deduplication Algorithms and Methods

Data deduplication relies on hash-based methods to generate unique s for data chunks, enabling the detection of exact duplicates. In these approaches, incoming data streams are segmented into chunks—either of fixed size or variable length—and each chunk is processed through a , such as SHA-256, to produce a fixed-size digest that serves as its identifier. This allows systems to compare chunks efficiently via hash tables, storing only one copy of identical content while referencing duplicates through metadata pointers. SHA-256, part of the SHA-2 family, provides 256-bit outputs that are computationally infeasible to reverse or collide under current attack capabilities, making it suitable for high-integrity environments like backup storage. However, these methods remain vulnerable to targeted collision attacks, where adversaries craft distinct inputs yielding the same hash, potentially leading to false positives or issues in shared storage systems; practical collisions for SHA-256 remain infeasible as of 2025. To address limitations of fixed chunking, such as sensitivity to data shifts from insertions or deletions, similarity detection techniques employ content-defined chunking. These methods divide data into variable-sized blocks by identifying boundaries based on intrinsic content patterns, rather than arbitrary offsets, which enhances deduplication effectiveness for similar-but-not-identical files. A prominent example is the use of , which compute fingerprints over a to locate breakpoints where the hash meets a predefined criterion, such as matching a specific mask. This approach, introduced in low-bandwidth file systems, typically uses a 48-byte and aims for average chunk sizes around 8 KB to balance overhead and redundancy reduction. The is calculated as a rolling hash modulo a large prime, formally expressed as: h=(sk⋅pw−1+sk−1⋅pw−2+⋯+sk−w+1⋅p0)mod  qh = \left( s_{k} \cdot p^{w-1} + s_{k-1} \cdot p^{w-2} + \cdots + s_{k-w+1} \cdot p^{0} \right) \mod q where sis_i are the bytes in the ww-byte window starting at position kk, pp is the base (often a primitive polynomial), and qq is a large prime modulus; boundaries are selected when the low-order bits of hh equal zero or a chosen value, allowing efficient incremental updates as the window slides. Such techniques improve similarity detection by aligning chunks across modified documents, achieving up to an order of magnitude better bandwidth savings in networked storage compared to fixed chunking. Advanced methods extend deduplication beyond exact matches to handle near-duplicates. , for instance, identifies similar chunks and stores only the differential changes (deltas) relative to a base version, using algorithms like or to compute minimal patches. This is particularly effective for versioned data, such as incremental backups, where small edits produce near-identical blocks; by combining it with chunk-level deduplication, systems can coalesce deltas across multiple similar items, yielding higher compression ratios—often 20-50% additional savings on mutable datasets—while maintaining fast reconstruction via patching. Emerging post-2020 techniques incorporate for pattern-based deduplication in , such as text or in storage systems, where models learn semantic similarities to cluster and eliminate redundancies that traditional hashes miss. These approaches have shown improvements over traditional methods in handling noisy datasets.

Advantages

Storage and Cost Efficiency

Data deduplication significantly enhances storage efficiency by eliminating redundant copies, leading to substantial space savings that vary based on types and levels. Typical deduplication ratios range from 5:1 to 50:1, meaning organizations can store up to 50 times more logical than the physical capacity required. For instance, virtual machine images often achieve ratios around 20:1 due to shared operating system and application components across instances. These savings are enabled by techniques like single-instance storage, which stores only one copy of identical blocks or files. The distinction between logical and physical capacity is central to understanding these benefits: logical capacity represents the total data volume as perceived by users, while physical capacity reflects the actual disk space consumed after deduplication. A 20:1 , for example, implies that 20 TB of logical data occupies just 1 TB of physical storage, directly lowering the need for hardware acquisitions. This reduction in physical translates to cost efficiencies, including decreased expenditures on storage devices and simplified data center expansion. Additionally, lower physical storage demands contribute to reduced and power costs, as fewer drives and cooling resources are required. Return on investment (ROI) calculations for deduplication often hinge on these ratios, with payback periods shortening as savings compound over time. For a achieving a 15:1 to 20:1 ratio in environments, ROI can be realized within months through avoided hardware purchases and operational savings. Over the long term, deduplication facilitates extended periods without linearly increasing storage growth, allowing organizations to maintain compliance and archival needs at a fraction of the original cost. This is particularly valuable in environments with accumulating historical data, where retention policies can extend from years to decades without proportional investments.

Performance and Bandwidth Benefits

Data deduplication accelerates ingestion and restore processes by substantially reducing the volume of transferred and processed, enabling operations to complete within tighter windows. Source-side deduplication, in particular, eliminates redundant before transmission, which can reduce transfer times by up to 90% in environments with high , such as virtualized systems or repeated full backups. This efficiency is evident in systems like ProtecTIER, where inline processing ensures is deduplicated in a single pass, avoiding additional post- overhead and supporting multi-stream throughputs exceeding 200 MB/s. In distributed setups involving remote replication, deduplication minimizes bandwidth consumption over wide area networks (WANs) by sending only unique chunks rather than full copies, which is critical for off-site disaster recovery. For example, replication traffic can be reduced by factors aligning with deduplication ratios of 13:1 to 38:1, making feasible the synchronization of terabyte-scale datasets across sites without saturating network links. Inline deduplication further contributes to immediate bandwidth savings during these transfers by applying reductions in real-time. Deduplication optimizes I/O performance by limiting the number of unique blocks written to and read from storage, thereby decreasing overall disk access demands. Techniques such as locality-preserved caching and summary vectors can cut disk I/Os by 98-99%, preserving responsiveness even under heavy workloads. This reduction in physical I/O translates to improved endurance for storage media and faster query times in verification or restore scenarios. The approach also bolsters for handling expansive datasets, as deduplicated storage maintains consistent metrics without linear growth in resource demands. Systems employing partitioned indexing and variable-size chunking achieve throughputs of 26-30 MB/s while scaling to petabyte volumes, ensuring that primary workloads remain unaffected by expanding footprints.

Disadvantages

Computational and Latency Overhead

Data deduplication imposes significant computational overhead due to the intensive operations involved in identifying and eliminating redundancies, particularly during hash computations for chunk fingerprinting and subsequent lookups in indexing structures. In primary storage environments, these processes can consume 30-40% of single-core CPU utilization across various workloads, with optimizations like efficient indexing slightly increasing this to around 40.8% to balance deduplication effectiveness and resource use. Such overhead arises primarily from algorithms like cryptographic hashing (e.g., or ), which require substantial cycles for processing incoming data streams. Latency impacts are particularly pronounced in inline deduplication, where real-time processing integrates into the critical write path, potentially delaying ingestion. Optimized inline approaches achieve 60-70% of maximum deduplication ratios with less than 5% additional CPU overhead and an average 2-4% increase in request latency, as demonstrated in evaluations using real-world traces on enterprise storage systems. In contrast, post-process deduplication defers these computations to background operations, avoiding immediate write delays but introducing periodic throughput spikes—typically 26-30 MB/s during scans—that must align with off-peak periods to prevent interference with foreground I/O. Memory demands further exacerbate the overhead, as deduplication relies on large in-memory hash tables to store fingerprints of unique chunks for rapid duplicate detection. Efficient designs allocate approximately 6 bytes of RAM per unique chunk, enabling significant reductions (up to 24x) through techniques like partitioning and prefetch caching; however, scaling to billions of entries—for instance, in petabyte-scale systems—can necessitate tens of gigabytes to terabytes of , often supplemented by SSD-based caching to handle overflow without compromising lookup speeds. To mitigate these resource burdens, contemporary storage systems incorporate via application-specific integrated circuits (), which offload CPU-intensive tasks like hashing directly into silicon. Introduced around 2015, solutions such as the HPE Gen5 Thin Express ASIC enable inline deduplication for block and file workloads without performance degradation, reducing operational cycles and supporting mixed workloads at scale. These advancements prioritize low-latency processing while maintaining high deduplication ratios, making them suitable for modern data centers.

Security and Reliability Concerns

Data deduplication relies on cryptographic hash functions to identify and eliminate duplicate data chunks, but hash collisions—where distinct data blocks produce the same hash value—pose a significant risk by potentially leading to or unauthorized substitution. Although modern deduplication systems typically employ robust algorithms like SHA-256, which offer high , vulnerabilities in weaker hashes such as have been exploited in practice. For instance, the 2017 SHAttered attack demonstrated a practical collision on , enabling attackers to craft two different PDF files with identical hashes; in deduplication contexts like repositories, committing such colliding files results in repository corruption, as the system treats them as identical and overwrites or merges content erroneously. This can amplify risks in storage systems, where a collision might silently corrupt multiple referenced instances of the shared chunk, underscoring the need for transitioning to collision-resistant hashes to mitigate deliberate attacks or accidental integrity failures. Privacy concerns arise prominently in multi-tenant environments like , where cross-user deduplication shares unique instances among users to optimize space, potentially exposing sensitive information if access controls or references are mismanaged. An attacker with knowledge of another user's could infer its presence in the by attempting to it and observing deduplication outcomes, creating side-channel leaks that reveal file existence or content patterns without direct access. Such shared instances exacerbate risks for confidential , as a breach in one user's reference chain might indirectly compromise others' through unintended or exposure during operations. To address this, techniques like message-locked (e.g., convergent encryption) are employed, but they introduce trade-offs in deduplication efficiency while aiming to preserve . Reliability issues in deduplication stem from the single-instance storage model, where errors in —tracking how many files point to a shared chunk—can cascade into widespread during system failures. If a chunk with a high reference count is corrupted by media errors, latent sector faults, or unrecoverable reads, multiple files become inaccessible, amplifying the impact compared to non-deduplicated systems; conversely, undercounting references might lead to premature chunk deletion, erasing still in use. Reference count inaccuracies often arise from concurrent updates, crashes, or software bugs, necessitating robust mechanisms like periodic scrubbing—systematic scanning and validation of chunks against hashes—and schemes such as per-file parity to detect and repair errors before they propagate. Without these safeguards, deduplication can reduce overall system , particularly in high-deduplication-ratio datasets where popular chunks are shared extensively. Deduplication's data co-mingling, where chunks from diverse sources share physical storage, presents compliance challenges under regulations like the EU's (GDPR), which mandates strict separation, minimization, and protection of to prevent unauthorized access or . Shared instances complicate proving data isolation and consent tracking, as co-mingled storage may hinder the ability to delete or isolate specific users' information upon request (right to erasure), potentially violating GDPR's data minimization and purpose limitation principles. Additionally, integrating —required by GDPR for securing at rest and in transit—conflicts with deduplication, as client-side encryption prevents servers from identifying duplicates without exposing keys, forcing a choice between privacy-preserving encryption and storage efficiency. Solutions like hybrid convergent encryption schemes aim to balance these, but they require careful implementation to ensure auditability and compliance without undermining deduplication benefits.

Applications and Implementations

Backup and Archival Systems

Data deduplication has become a core feature in modern solutions, particularly those developed or enhanced in the post-2000s era, enabling efficient handling of incremental backups where redundancy is high due to minor changes between sessions. Tools such as integrate source-side deduplication to identify and transmit only unique data blocks, significantly reducing network traffic and storage needs for incremental jobs across multiple virtual machines. Similarly, Commvault's deduplication engine eliminates duplicate blocks during backup operations, achieving substantial space savings in incremental cycles by referencing a deduplication database that tracks unique data across jobs. These integrations typically yield deduplication ratios of 10:1 to 30:1 or higher for incremental backups, depending on data patterns and retention periods. Key features of deduplication in backup systems include the creation of synthetic full backups, which leverage deduplicated increments to construct complete backups without rescanning the production source, thereby minimizing backup windows and resource impact. In , this process combines active fulls with deduplicated incrementals to generate synthetic fulls, optimizing for environments with frequent small changes. employs DASH fulls for deduplicated data, accelerating synthetic full creation by referencing only changed blocks in the deduplication database, which avoids full data reads from primary storage. Additionally, deduplication optimizes tape-based archival by reducing the volume of data written to media; for instance, virtual tape libraries with inline deduplication compress backup streams before to physical tapes, enabling longer retention on limited tape capacity while maintaining fast access for restores. Prominent vendors specializing in deduplication for and archival include and ExaGrid, both of which provide target-side appliances tailored for long-term storage efficiency. systems, originally developed by EMC and now under , apply inline deduplication to data, routinely achieving ratios of 20:1 on average for weekly full and daily incremental cycles, with potential up to 50:1 for archival datasets featuring high over extended retention. ExaGrid's adaptive deduplication performs post-process analysis on ingested data, delivering average ratios of 20:1 for workloads and up to 50:1 or higher for long-term archival with multi-year retention, as the system identifies duplicates across versions without impacting initial write performance. Best practices for implementing deduplication in and archival emphasize post-process deduplication to prioritize low-latency initial backups, where is first written to disk without hashing overhead, followed by offline deduplication to reclaim space and maintain high throughput during active jobs. For handling versioned files common in backups—such as evolving documents or —variable block deduplication is recommended, as it dynamically adjusts chunk boundaries to capture changes efficiently, reducing fragmentation and improving ratios compared to fixed blocks in scenarios with partial file updates. Source deduplication can also be briefly applied for remote backups to further minimize WAN bandwidth by filtering duplicates at the client before transmission.

Cloud and Distributed Storage

In cloud storage environments, data deduplication is often implemented implicitly through architectures that allow users to manage and eliminate redundant objects. For instance, supports deduplication by enabling the identification of potential duplicate objects using S3 Inventory reports and subsequent operations via S3 Batch to delete redundancies, thereby optimizing storage costs without native block-level elimination. This approach is particularly useful in multi-tenant scenarios where object versioning and lifecycle policies further reduce duplication overhead. Similarly, Azure Blob Storage lacks built-in automatic deduplication but provides explicit tools post-2018, such as integration with Azure Data Factory for processing duplicates during data ingestion and Azure Data Explorer's materialized views for handling redundant records in blob-sourced datasets, achieving efficient space reclamation in large-scale unstructured data stores. Distributed storage systems like Hadoop Distributed File System (HDFS) and Ceph incorporate block-level deduplication to manage petabyte-scale across clusters, supporting multi-tenant environments by minimizing replication of identical blocks. In HDFS, dynamic deduplication decisions evaluate chunk similarity before storage, reducing overhead while maintaining through configurable block sizes, as demonstrated in extensions that achieve up to 50% space savings on redundant workloads without altering core replication. Ceph, designed for exabyte scalability, uses RADOS (Reliable Autonomic Distributed Object Store) machinery for deduplication, where cluster-wide algorithms like TiDedup perform online block-level detection and elimination, handling multi-tenant sharing by isolating metadata per namespace and scaling to thousands of nodes for petabyte deployments. Advancements in the have extended deduplication to serverless and paradigms, enabling efficient processing in distributed setups. Serverless frameworks incorporate memory deduplication techniques, such as , which exploit redundancy in warm container states to reduce latency by up to 40% in function executions, integrating seamlessly with edge nodes for low-latency data handling. Deduplication appliances like Pure Storage's Cloud Block Store further enhance integration, providing inline deduplication for hybrid environments with AWS, where data is thin-provisioned and reduced at the block level before cloud replication, supporting seamless workload mobility. These techniques address key challenges in global replication, such as bandwidth constraints, by compressing data transfers through deduplication, yielding significant savings in VM workloads. For example, in environments, ratios up to 30:1 are achievable with global deduplication, reducing replicated data volume and network usage while preserving performance across distributed clouds.

File Systems and Virtualization

Data deduplication in file systems integrates directly into the storage layer of operating systems to eliminate redundant data blocks or files, enhancing efficiency at the OS level. , originally developed for Solaris and now maintained as , includes built-in deduplication capabilities that identify and store unique data chunks while referencing duplicates via metadata pointers. This feature was introduced in build 128 in December 2009, allowing administrators to enable it per dataset with properties like dedup=on for inline processing during writes. , a for kernels since version 2.6.29 in 2009, supports deduplication through kernel syscalls that enable tools to clone extents sharing identical data blocks without full copies. , the default in editions starting from 2012, incorporates Data Deduplication as a server role that performs post-process optimization on volumes, chunking files into 32-128 KB segments and storing unique ones while maintaining file integrity. However, built-in data deduplication is not available on NTFS volumes in Windows 11 client editions, such as Home and Pro. The full Data Deduplication feature is exclusive to Windows Server editions and supports NTFS there. In Windows 11 client editions, deduplication is limited to ReFS volumes via built-in ReFS features, such as refsutil dedup commands or block cloning for Dev Drives. NTFS volumes in client editions have no native deduplication support, relying only on compression or third-party tools. Unofficial hacks, such as DISM injection, exist but are unsupported and risky. At the operating system level, kernel modules facilitate real-time deduplication by intercepting I/O operations and applying hashing to incoming data before committing to disk. For instance, the kvdo (Kernel Virtual Data Optimizer) module in integrates with the Device Mapper layer to provide inline deduplication, compression, and for block devices, reducing in real-time workflows. However, this metadata-intensive process—tracking hashes and mappings—can accelerate SSD wear due to frequent small writes for deduplication tables, potentially reducing drive lifespan in high-churn environments unless mitigated by special allocation classes or wear-leveling optimizations. In virtualization environments, deduplication optimizes storage for (VM) images, which often contain redundant operating system and application data across instances. , since version 4 in 2009, supports for virtual disks that dynamically allocates space, combined with deduplication in storage features to share common blocks among VMs, particularly effective for virtual desktop infrastructure (VDI). Microsoft Hyper-V integrates deduplication with its storage stack, allowing optimized cloning of VMs on volumes where block cloning accelerates snapshot merges and reduces duplication in checkpoint operations. For identical VMs, such as multiple instances of the same OS image, deduplication ratios commonly exceed 10:1, as a single unique base image is referenced multiple times, significantly lowering storage footprint. VM images typically employ block-level formats like VMDK, enabling fine-grained deduplication at the sector level across virtual disks. VMware vSAN extends these capabilities with inline deduplication at the cluster level, processing data as it moves from cache to capacity tiers using fixed-size block hashing to eliminate duplicates before persistence. This approach minimizes latency overhead in virtualized setups while complementing , though it requires sufficient RAM for hash tables to maintain in dense VM deployments. Overall, these implementations balance space savings with operational trade-offs, prioritizing workloads with high redundancy like VM sprawl.

References

  1. https://www.netapp.com/data-services/what-is-data-deduplication/
  2. https://www.oracle.com/data-deduplication/
  3. https://www.cohesity.com/glossary/data-deduplication/
  4. https://www.hpe.com/us/en/what-is/data-deduplication.html
  5. https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/overview
  6. https://www.usenix.org/system/files/conference/atc12/atc12-final293.pdf
  7. https://mycodelesswebsite.com/big-data-statistics/
  8. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-209.pdf
  9. https://dl.acm.org/doi/pdf/10.5555/2388996.2389006
  10. https://www.komprise.com/glossary_terms/deduplication/
  11. https://www.theregister.com/2016/06/29/permabit_offering_dedupe_to_linux_masses_almost/
  12. https://www.dell.com/en-us/dt/corporate/newsroom/announcements/2009/07/20090708-02.htm
  13. https://www.networkworld.com/article/803984/data-center-ibm-acquires-diligent-technologies.html
  14. https://www.crn.com/news/storage/222400378/symantec-boosts-backup-exec-netbackup-with-dedupe-virtualization
  15. https://www.techtarget.com/searchdatabackup/tutorial/Inline-deduplication-vs-post-processing-Data-dedupe-best-practices
  16. https://www.archivemarketresearch.com/reports/data-deduplication-software-561123
  17. https://www.dataversity.net/articles/scale-out-zfs-scalable-storage-for-exponential-data-growth-and-powerful-data-protection/
  18. https://www.enterprisestorageforum.com/management/data-deduplication-trends/
  19. https://static.usenix.org/events/fast12/poster_descriptions/Xiadescription.pdf
  20. https://www.sciencedirect.com/science/article/pii/S1319157821000914
  21. https://www.cs.williams.edu/~jannen/teaching/s21/cs333/meetings/dedup.html
  22. https://ceph.com/assets/pdfs/ICDCS_2018_mwoh.pdf
  23. https://cdn.allbound.com/iq-ab/2020/02/Data-Deduplication-Background_-A-Technical-White-Paper-WP00126A.pdf
  24. https://cs.brown.edu/courses/cs227/archives/2016/papers/DataDeduplicationTechniques.pdf
  25. https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/understand
  26. http://www.mckusick.com/bookrefs/zfs_dedup.html
  27. https://www.ibm.com/think/topics/data-deduplication-types
  28. https://www.usenix.org/legacy/event/fast09/tech/full_papers/lillibridge/lillibridge.pdf
  29. https://dl.acm.org/doi/fullHtml/10.5555/2015976.2015983
  30. https://learning.dell.com/content/dam/dell-emc/documents/en-us/2010KS_Yellin-The_Illusion_of_Space_and_the_Science_of_Data_Compression_and_Deduplication.pdf
  31. https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/uk/demystifying-deduplication.pdf
  32. https://conservancy.umn.edu/bitstreams/15d8ca76-0f90-454e-bea6-a0feb846569f/download
  33. https://www.networkcomputing.com/data-center-networking/source-vs-target-deduplication-scale-matters
  34. https://www.usenix.org/system/files/atc21_slides_ren-yanjing.pdf
  35. https://ranger.uta.edu/~jiang/publication/Journals/2013/2013-J%2520Sign%2520Proc%2520Syst-SAFE-%2520A%2520Source%2520Deduplication%2520Framework%2520for%2520Efficient%2520Cloud%2520Backup%2520Services.pdf
  36. https://storageswiss.com/2013/12/23/source-vs-target-based-data-deduplication/
  37. https://backupcentral.com/dedupe-ratios/
  38. https://www.purestorage.com/knowledge/what-is-data-deduplication.html
  39. https://www.usenix.org/legacy/event/fast11/tech/full_papers/Meyer.pdf
  40. https://www.techtarget.com/searchdatabackup/tip/Data-deduplication-methods-Block-level-versus-byte-level-dedupe
  41. https://www.netapp.com/media/22774-Variable_length-dedup_Giridhar.pdf
  42. https://ieeexplore.ieee.org/document/7087116
  43. https://specopssoft.com/blog/sha256-hashing-password-cracking/
  44. https://pdos.csail.mit.edu/papers/lbfs:sosp01/lbfs.pdf
  45. https://dl.acm.org/doi/10.1145/2611778
  46. https://cswxia.github.io/pub/dcc-wen-delta-2014.pdf
  47. https://arxiv.org/abs/2308.00721
  48. https://dev.snia.org/sites/default/files/DMF_DeDupe.PDF
  49. https://www.eweek.com/storage/how-to-choose-the-right-deduplication-technology/
  50. https://www.techtarget.com/searchdatabackup/tip/Understanding-data-deduplication-ratios-in-backup-systems
  51. https://www.oracle.com/middleeast/data-deduplication/
  52. https://www.ibm.com/think/topics/data-deduplication
  53. https://i.dell.com/sites/csdocuments/Business_smb_sb360_Documents/en/us/ps2q10-20100364-edemc.pdf
  54. https://www.exagrid.com/exagrid-products/data-deduplication-impact/
  55. https://documentation.arcserve.com/Arcserve-Backup/Available/19.0/ENU/Bookshelf_Files/HTML/admingde/Content/dedup_how_data_deduplication_works.htm
  56. https://www.networkworld.com/article/966188/what-is-data-deduplication-and-how-is-it-implemented.html
  57. https://www.arcserve.com/blog/understanding-importance-deduplication-your-backup-strategy
  58. https://www.usenix.org/event/fast08/tech/full_papers/zhu/zhu.pdf
  59. https://www.ibm.com/docs/en/ts7650g-protectier/3.4.0?topic=introduction-software
  60. https://www.dell.com/support/manuals/en-us/powerstore-9000/pwrstr-monitoring/data-reduction?guid=guid-db85dda3-fe0d-40f1-bd9a-703696fda302&lang=en-us
  61. https://dl.acm.org/doi/full/10.1145/3750447
  62. https://www.usenix.org/system/files/conference/fast12/srinivasan.pdf
  63. https://www.storagereview.com/news/hp-introduces-the-worlds-most-affordable-automated-all-flash-array
  64. https://go.vlcmtech.com/hubfs/Thin_Deduplication_Brochure.pdf
  65. https://shattered.io/
  66. https://www.pinkas.net/PAPERS/hps.pdf
  67. https://www.sciencedirect.com/science/article/pii/S1319157820305140
  68. https://ieeexplore.ieee.org/document/8640256
  69. https://ranger.uta.edu/~jiang/publication/Conferences/2018/2018-SRDS-Improving%2520Reliability%2520of%2520Deduplication-based%2520Storage%2520Systems%2520with%2520Per-File%2520Parity.pdf
  70. https://blog.purestorage.com/perspectives/gdpr-data-encryption-dilemma/
  71. https://gdpr-info.eu/issues/encryption/
  72. https://helpcenter.veeam.com/docs/backup/vsphere/compression_deduplication.html
  73. https://documentation.commvault.com/v11/commcell-console/optimize_storage_space_using_deduplication.html
  74. https://www.delltechnologies.com/asset/en-us/products/data-protection/technical-support/docu85190.pdf
  75. https://documentation.commvault.com/v11/commcell-console/synthetic_full_backups.html
  76. https://www.nakivo.com/blog/dellemc-data-domain-boost-support-nakivo-backup-replication/
  77. https://www.techtarget.com/searchdatabackup/tip/The-ultimate-guide-to-backup-deduplication
  78. https://aws.amazon.com/blogs/storage/managing-duplicate-objects-in-amazon-s3/
  79. https://learn.microsoft.com/en-us/azure/data-explorer/dealing-with-duplicates
  80. https://serverfault.com/questions/822312/deduplication-and-cost-savings-on-iaas-object-stores-s3-azure-blobs
  81. https://journals.sagepub.com/doi/10.1155/2014/630380
  82. https://docs.ceph.com/en/latest/dev/deduplication/
  83. https://www.usenix.org/system/files/atc23-oh.pdf
  84. https://www.researchgate.net/publication/359529123_Memory_deduplication_for_serverless_computing_with_Medes
  85. https://www.purestorage.com/content/dam/pdf/en/datasheets/ds-pure-storage-cloud-block-store-for-aws.pdf
  86. https://www.cohesity.com/blogs/global-deduplication-matters/
  87. https://community.hpe.com/t5/array-performance-and-data/nimble-storage-deduplication/td-p/6981835
  88. https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gjhbo/index.html
  89. https://btrfs.readthedocs.io/en/latest/Deduplication.html
  90. https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/interop
  91. https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview
  92. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html-single/deduplicating_and_compressing_storage/index
  93. https://queue.acm.org/detail.cfm?id=1985003
  94. https://www.vmware.com/docs/vsp_4_thinprov_perf
  95. https://www.oracle.com/technetwork/articles/servers-storage-admin/zfs-storage-deduplication-335298.html
  96. https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration/increasing-space-efficiency-in-a-vsan-cluster/using-deduplication-and-compression-in-vsan-cluster.html
Add your contribution
Related Hubs
User Avatar
No comments yet.