Hubbry Logo
Apache HadoopApache HadoopMain
Open search
Apache Hadoop
Community hub
Apache Hadoop
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Apache Hadoop
Apache Hadoop
Apache Hadoop
View on Wikipedia
from Wikipedia
Apache Hadoop
Original authorsDoug Cutting, Mike Cafarella
DeveloperApache Software Foundation
Initial releaseApril 1, 2006; 19 years ago (2006-04-01)[1]
Stable release
2.10.x2.10.2 / May 31, 2022; 3 years ago (2022-05-31)[2]
3.4.x3.4.0 / March 17, 2024; 19 months ago (2024-03-17)[2]
Repositorygithub.com/apache/hadoop
Written inJava
Operating systemCross-platform
TypeDistributed file system
LicenseApache License 2.0
Websitehadoop.apache.org Edit this at Wikidata

Apache Hadoop (/həˈduːp/) is a collection of open-source software utilities for reliable, scalable, distributed computing. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model. Hadoop was originally designed for computer clusters built from commodity hardware, which is still the common use.[3] It has since also found use on clusters of higher-end hardware.[4][5] All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common occurrences and should be automatically handled by the framework.[6]

Overview

[edit]

The core of Apache Hadoop consists of a storage part, known as Hadoop Distributed File System (HDFS), and a processing part which is a MapReduce programming model. Hadoop splits files into large blocks and distributes them across nodes in a cluster. It then transfers packaged code into nodes to process the data in parallel. This approach takes advantage of data locality,[7] where nodes manipulate the data they have access to. This allows the dataset to be processed faster and more efficiently than it would be in a more conventional supercomputer architecture that relies on a parallel file system where computation and data are distributed via high-speed networking.[8][9]

The base Apache Hadoop framework is composed of the following modules:

  • Hadoop Common - contains libraries and utilities needed by other Hadoop modules;
  • Hadoop Distributed File System (HDFS) - a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster;
  • Hadoop YARN - (introduced in 2012) is a platform responsible for managing computing resources in clusters and using them for scheduling users' applications;[10][11]
  • Hadoop MapReduce - an implementation of the MapReduce programming model for large-scale data processing.
  • Hadoop Ozone - (introduced in 2020) An object store for Hadoop

The term Hadoop is often used for both base modules and sub-modules and also the ecosystem,[12] or collection of additional software packages that can be installed on top of or alongside Hadoop, such as Apache Pig, Apache Hive, Apache HBase, Apache Phoenix, Apache Spark, Apache ZooKeeper, Apache Impala, Apache Flume, Apache Sqoop, Apache Oozie, and Apache Storm.[13]

Apache Hadoop's MapReduce and HDFS components were inspired by Google papers on MapReduce and Google File System.[14]

The Hadoop framework itself is mostly written in the Java programming language, with some native code in C and command line utilities written as shell scripts. Perl language can be easily used with Hadoop Streaming to implement the map and reduce parts of the user's program.[15]

History

[edit]

According to its co-founders, Doug Cutting and Mike Cafarella, the idea of Hadoop was conceived in the Google File System paper that was published in October 2003.[16][17] The concept was extended in the Google paper "MapReduce: Simplified Data Processing on Large Clusters".[18] Development started on the Apache Nutch project, but was moved to the new Hadoop subproject in January 2006.[19] Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant.[20] The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce.

In March 2006, Owen O'Malley was the first committer to add to the Hadoop project;[21] Hadoop 0.1.0 was released in April 2006.[22] It continues to evolve through contributions that are being made to the project.[23] The first design document for the Hadoop Distributed File System was written by Dhruba Borthakur in 2007.[24]

Version Original release date Latest version Release date
Unsupported: 0.10 0.10.1 2007-01-11
Unsupported: 0.11 0.11.2 2007-02-16
Unsupported: 0.12 2007-03-02 0.12.3 2007-04-06
Unsupported: 0.13 2007-06-04 0.13.1 2007-07-23
Unsupported: 0.14 2007-09-04 0.14.4 2007-11-26
Unsupported: 0.15 2007-10-29 0.15.3 2008-01-18
Unsupported: 0.16 2008-02-07 0.16.4 2008-05-05
Unsupported: 0.17 2008-05-20 0.17.2 2008-08-19
Unsupported: 0.18 2008-08-22 0.18.3 2009-01-29
Unsupported: 0.19 2008-11-21 0.19.2 2009-07-23
Unsupported: 0.20 2009-04-22 0.20.205.0 2011-10-17
Unsupported: 0.21 2011-05-11 0.21.0
Unsupported: 0.22 2011-12-10 0.22.0
Unsupported: 0.23 2011-11-11 0.23.11 2014-06-27
Unsupported: 1.0 2011-12-27 1.0.4 2012-10-12
Unsupported: 1.1 2012-10-13 1.1.2 2013-02-15
Unsupported: 1.2 2013-05-13 1.2.1 2013-08-01
Unsupported: 2.0 2012-05-23 2.0.6-alpha 2013-08-23
Unsupported: 2.1 2013-08-25 2.1.1-beta 2013-09-23
Unsupported: 2.2 2013-12-11 2.2.0
Unsupported: 2.3 2014-02-20 2.3.0
Unsupported: 2.4 2014-04-07 2.4.1 2014-06-30
Unsupported: 2.5 2014-08-11 2.5.2 2014-11-19
Unsupported: 2.6 2014-11-18 2.6.5 2016-10-08
Unsupported: 2.7 2015-04-21 2.7.7 2018-05-31
Unsupported: 2.8 2017-03-22 2.8.5 2018-09-15
Unsupported: 2.9 2017-12-17 2.9.2 2018-11-19
Supported: 2.10 2019-10-29 2.10.2 2022-05-31[25]
Unsupported: 3.0 2017-12-13[26] 3.0.3 2018-05-31[27]
Unsupported: 3.1 2018-04-06 3.1.4 2020-08-03[28]
Latest version: 3.2 2019-01-16 3.2.4 2022-07-22[29]
Latest version: 3.3 2020-07-14 3.3.6 2023-06-23[30]
Latest version: 3.4 2024-03-17 3.4.0 2024-07-17[31]
Legend:
Unsupported
Supported
Latest version
Preview version
Future version

Architecture

[edit]
See also: Hadoop Distributed File System, Apache HBase, and MapReduce

Hadoop consists of the Hadoop Common package, which provides file system and operating system level abstractions, a MapReduce engine (either MapReduce/MR1 or YARN/MR2)[32] and the Hadoop Distributed File System (HDFS). The Hadoop Common package contains the Java Archive (JAR) files and scripts needed to start Hadoop.

For effective scheduling of work, every Hadoop-compatible file system should provide location awareness, which is the name of the rack, specifically the network switch where a worker node is. Hadoop applications can use this information to execute code on the node where the data is, and, failing that, on the same rack/switch to reduce backbone traffic. HDFS uses this method when replicating data for data redundancy across multiple racks. This approach reduces the impact of a rack power outage or switch failure; if any of these hardware failures occurs, the data will remain available.[33]

Hadoop cluster
A multi-node Hadoop cluster

A small Hadoop cluster includes a single master and multiple worker nodes. The master node consists of a Job Tracker, Task Tracker, NameNode, and DataNode. A slave or worker node acts as both a DataNode and TaskTracker, though it is possible to have data-only and compute-only worker nodes. These are normally used only in nonstandard applications.[34]

Hadoop requires the Java Runtime Environment (JRE) 1.6 or higher. The standard startup and shutdown scripts require that Secure Shell (SSH) be set up between nodes in the cluster.[35]

In a larger cluster, HDFS nodes are managed through a dedicated NameNode server to host the file system index, and a secondary NameNode that can generate snapshots of the namenode's memory structures, thereby preventing file-system corruption and loss of data. Similarly, a standalone JobTracker server can manage job scheduling across nodes. When Hadoop MapReduce is used with an alternate file system, the NameNode, secondary NameNode, and DataNode architecture of HDFS are replaced by the file-system-specific equivalents.

File systems

[edit]

Hadoop distributed file system

[edit]

The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system written in Java for the Hadoop framework. A Hadoop instance is divided into HDFS and MapReduce. HDFS is used for storing the data and MapReduce is used for processing data. HDFS has five services as follows:

  1. Name Node
  2. Secondary Name Node
  3. Job tracker
  4. Data Node
  5. Task Tracker

Top three are Master Services/Daemons/Nodes and bottom two are Slave Services. Master Services can communicate with each other and in the same way Slave services can communicate with each other. Name Node is a master node and Data node is its corresponding Slave node and can talk with each other.

Name Node: HDFS consists of only one Name Node that is called the Master Node. The master node can track files, manage the file system and has the metadata of all of the stored data within it. In particular, the name node contains the details of the number of blocks, locations of the data node that the data is stored in, where the replications are stored, and other details. The name node has direct contact with the client.

Data Node: A Data Node stores data in it as blocks. This is also known as the slave node and it stores the actual data into HDFS which is responsible for the client to read and write. These are slave daemons. Every Data node sends a Heartbeat message to the Name node every 3 seconds and conveys that it is alive. In this way when Name Node does not receive a heartbeat from a data node for 2 minutes, it will take that data node as dead and starts the process of block replications on some other Data node.

Secondary Name Node: This is only to take care of the checkpoints of the file system metadata which is in the Name Node. This is also known as the checkpoint Node. It is the helper Node for the Name Node. The secondary name node instructs the name node to create & send fsimage & editlog file, upon which the compacted fsimage file is created by the secondary name node.[36]

Job Tracker: Job Tracker receives the requests for Map Reduce execution from the client. Job tracker talks to the Name Node to know about the location of the data that will be used in processing. The Name Node responds with the metadata of the required processing data.

Task Tracker: It is the Slave Node for the Job Tracker and it will take the task from the Job Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on the file. The process of applying that code on the file is known as Mapper.[37]

Hadoop cluster has nominally a single namenode plus a cluster of datanodes, although redundancy options are available for the namenode due to its criticality. Each datanode serves up blocks of data over the network using a block protocol specific to HDFS. The file system uses TCP/IP sockets for communication. Clients use remote procedure calls (RPC) to communicate with each other.

HDFS stores large files (typically in the range of gigabytes to terabytes[38]) across multiple machines. It achieves reliability by replicating the data across multiple hosts, and hence theoretically does not require redundant array of independent disks (RAID) storage on hosts (but to increase input-output (I/O) performance some RAID configurations are still useful). With the default replication value, 3, data is stored on three nodes: two on the same rack, and one on a different rack. Data nodes can talk to each other to rebalance data, to move copies around, and to keep the replication of data high. HDFS is not fully POSIX-compliant, because the requirements for a POSIX file-system differ from the target goals of a Hadoop application. The trade-off of not having a fully POSIX-compliant file-system is increased performance for data throughput and support for non-POSIX operations such as Append.[39]

In May 2012, high-availability capabilities were added to HDFS,[40] letting the main metadata server called the NameNode manually fail-over onto a backup. The project has also started developing automatic fail-overs.

The HDFS file system includes a so-called secondary namenode, a misleading term that some might incorrectly interpret as a backup namenode when the primary namenode goes offline. In fact, the secondary namenode regularly connects with the primary namenode and builds snapshots of the primary namenode's directory information, which the system then saves to local or remote directories. These checkpointed images can be used to restart a failed primary namenode without having to replay the entire journal of file-system actions, then to edit the log to create an up-to-date directory structure. Because the namenode is the single point for storage and management of metadata, it can become a bottleneck for supporting a huge number of files, especially a large number of small files. HDFS Federation, a new addition, aims to tackle this problem to a certain extent by allowing multiple namespaces served by separate namenodes. Moreover, there are some issues in HDFS such as small file issues, scalability problems, Single Point of Failure (SPoF), and bottlenecks in huge metadata requests. One advantage of using HDFS is data awareness between the job tracker and task tracker. The job tracker schedules map or reduce jobs to task trackers with an awareness of the data location. For example: if node A contains data (a, b, c) and node X contains data (x, y, z), the job tracker schedules node A to perform map or reduce tasks on (a, b, c) and node X would be scheduled to perform map or reduce tasks on (x, y, z). This reduces the amount of traffic that goes over the network and prevents unnecessary data transfer. When Hadoop is used with other file systems, this advantage is not always available. This can have a significant impact on job-completion times as demonstrated with data-intensive jobs.[41]

HDFS was designed for mostly immutable files and may not be suitable for systems requiring concurrent write operations.[39]

HDFS can be mounted directly with a Filesystem in Userspace (FUSE) virtual file system on Linux and some other Unix systems.

File access can be achieved through the native Java API, the Thrift API (generates a client in a number of languages e.g. C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml), the command-line interface, the HDFS-UI web application over HTTP, or via 3rd-party network client libraries.[42]

HDFS is designed for portability across various hardware platforms and for compatibility with a variety of underlying operating systems. The HDFS design introduces portability limitations that result in some performance bottlenecks, since the Java implementation cannot use features that are exclusive to the platform on which HDFS is running.[43] Due to its widespread integration into enterprise-level infrastructure, monitoring HDFS performance at scale has become an increasingly important issue. Monitoring end-to-end performance requires tracking metrics from datanodes, namenodes, and the underlying operating system.[44] There are currently several monitoring platforms to track HDFS performance, including Hortonworks, Cloudera, and Datadog.

Other file systems

[edit]

Hadoop works directly with any distributed file system that can be mounted by the underlying operating system by simply using a file:// URL; however, this comes at a price – the loss of locality. To reduce network traffic, Hadoop needs to know which servers are closest to the data, information that Hadoop-specific file system bridges can provide.

In May 2011, the list of supported file systems bundled with Apache Hadoop were:

  • HDFS: Hadoop's own rack-aware file system.[45] This is designed to scale to tens of petabytes of storage and runs on top of the file systems of the underlying operating systems.
  • Apache Hadoop Ozone: HDFS-compatible object store targeting optimized for billions of small files.
  • FTP file system: This stores all its data on remotely accessible FTP servers.
  • Amazon S3 (Amazon Simple Storage Service) object storage: This is targeted at clusters hosted on the Amazon Elastic Compute Cloud server-on-demand infrastructure. There is no rack-awareness in this file system, as it is all remote.
  • Windows Azure Storage Blobs (WASB) file system: This is an extension of HDFS that allows distributions of Hadoop to access data in Azure blob stores without moving the data permanently into the cluster.

A number of third-party file system bridges have also been written, none of which are currently in Hadoop distributions. However, some commercial distributions of Hadoop ship with an alternative file system as the default – specifically IBM and MapR.

  • In 2009, IBM discussed running Hadoop over the IBM General Parallel File System.[46] The source code was published in October 2009.[47]
  • In April 2010, Parascale published the source code to run Hadoop against the Parascale file system.[48]
  • In April 2010, Appistry released a Hadoop file system driver for use with its own CloudIQ Storage product.[49]
  • In June 2010, HP discussed a location-aware IBRIX Fusion file system driver.[50]
  • In May 2011, MapR Technologies Inc. announced the availability of an alternative file system for Hadoop, MapR FS, which replaced the HDFS file system with a full random-access read/write file system.

JobTracker and TaskTracker: the MapReduce engine

[edit]
Main article: MapReduce

Atop the file systems comes the MapReduce Engine, which consists of one JobTracker, to which client applications submit MapReduce jobs. The JobTracker pushes work to available TaskTracker nodes in the cluster, striving to keep the work as close to the data as possible. With a rack-aware file system, the JobTracker knows which node contains the data, and which other machines are nearby. If the work cannot be hosted on the actual node where the data resides, priority is given to nodes in the same rack. This reduces network traffic on the main backbone network. If a TaskTracker fails or times out, that part of the job is rescheduled. The TaskTracker on each node spawns a separate Java virtual machine (JVM) process to prevent the TaskTracker itself from failing if the running job crashes its JVM. A heartbeat is sent from the TaskTracker to the JobTracker every few minutes to check its status. The Job Tracker and TaskTracker status and information is exposed by Jetty and can be viewed from a web browser.

Known limitations of this approach are:

  1. The allocation of work to TaskTrackers is very simple. Every TaskTracker has a number of available slots (such as "4 slots"). Every active map or reduce task takes up one slot. The Job Tracker allocates work to the tracker nearest to the data with an available slot. There is no consideration of the current system load of the allocated machine, and hence its actual availability.
  2. If one TaskTracker is very slow, it can delay the entire MapReduce job – especially towards the end, when everything can end up waiting for the slowest task. With speculative execution enabled, however, a single task can be executed on multiple slave nodes.

Scheduling

[edit]

By default Hadoop uses FIFO scheduling, and optionally 5 scheduling priorities to schedule jobs from a work queue.[51] In version 0.19 the job scheduler was refactored out of the JobTracker, while adding the ability to use an alternate scheduler (such as the Fair scheduler or the Capacity scheduler, described next).[52]

Fair scheduler
[edit]

The fair scheduler was developed by Facebook.[53] The goal of the fair scheduler is to provide fast response times for small jobs and Quality of service (QoS) for production jobs. The fair scheduler has three basic concepts.[54]

  1. Jobs are grouped into pools.
  2. Each pool is assigned a guaranteed minimum share.
  3. Excess capacity is split between jobs.

By default, jobs that are uncategorized go into a default pool. Pools have to specify the minimum number of map slots, reduce slots, as well as a limit on the number of running jobs.

Capacity scheduler
[edit]

The capacity scheduler was developed by Yahoo. The capacity scheduler supports several features that are similar to those of the fair scheduler.[55]

  1. Queues are allocated a fraction of the total resource capacity.
  2. Free resources are allocated to queues beyond their total capacity.
  3. Within a queue, a job with a high level of priority has access to the queue's resources.

There is no preemption once a job is running.

Difference between Hadoop 1 and Hadoop 2 (YARN)

[edit]

The biggest difference between Hadoop 1 and Hadoop 2 is the addition of YARN (Yet Another Resource Negotiator), which replaced the MapReduce engine in the first version of Hadoop. YARN strives to allocate resources to various applications effectively. It runs two daemons, which take care of two different tasks: the resource manager, which does job tracking and resource allocation to applications, the application master, which monitors progress of the execution.

Difference between Hadoop 2 and Hadoop 3

[edit]

There are important features provided by Hadoop 3. For example, while there is one single namenode in Hadoop 2, Hadoop 3, enables having multiple name nodes, which solves the single point of failure problem.

In Hadoop 3, there are containers working in principle of Docker, which reduces time spent on application development.

One of the biggest changes is that Hadoop 3 decreases storage overhead with erasure coding.

Also, Hadoop 3 permits usage of GPU hardware within the cluster, which is a very substantial benefit to execute deep learning algorithms on a Hadoop cluster.[56]

Other applications

[edit]

The HDFS is not restricted to MapReduce jobs. It can be used for other applications, many of which are under development at Apache. The list includes the HBase database, the Apache Mahout machine learning system, and the Apache Hive data warehouse. Theoretically, Hadoop could be used for any workload that is batch-oriented rather than real-time, is very data-intensive, and benefits from parallel processing. It can also be used to complement a real-time system, such as lambda architecture, Apache Storm, Flink, and Spark Streaming.[57]

Commercial applications of Hadoop include:[58]

  • Log or clickstream analysis
  • Marketing analytics
  • Machine learning and data mining
  • Image processing
  • XML message processing
  • Web crawling
  • Archival work for compliance, including of relational and tabular data

Prominent use cases

[edit]

On 19 February 2008, Yahoo! Inc. launched what they claimed was the world's largest Hadoop production application. The Yahoo! Search Webmap is a Hadoop application that runs on a Linux cluster with more than 10,000 cores and produced data that was used in every Yahoo! web search query.[59] There are multiple Hadoop clusters at Yahoo! and no HDFS file systems or MapReduce jobs are split across multiple data centers. Every Hadoop cluster node bootstraps the Linux image, including the Hadoop distribution. Work that the clusters perform is known to include the index calculations for the Yahoo! search engine. In June 2009, Yahoo! made the source code of its Hadoop version available to the open-source community.[60]

In 2010, Facebook claimed that they had the largest Hadoop cluster in the world with 21 PB of storage.[61] In June 2012, they announced the data had grown to 100 PB[62] and later that year they announced that the data was growing by roughly half a PB per day.[63]

As of 2013, Hadoop adoption had become widespread: more than half of the Fortune 50 companies used Hadoop.[64]

Papers

[edit]

Influential papers on the birth, growth, and curation of Hadoop and big data processing include: Jeffrey Dean, Sanjay Ghemawat (2004) MapReduce: Simplified Data Processing on Large Clusters, Google. This paper inspired Doug Cutting to develop an open-source implementation of the Map-Reduce framework. He named it Hadoop, after his son's toy elephant.

See also

[edit]

References

[edit]

Bibliography

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Apache Hadoop

View on Grokipedia
from Grokipedia
Apache Hadoop is an framework designed for the distributed storage and processing of large-scale data sets across clusters of commodity hardware, enabling reliable, scalable, and fault-tolerant analytics. It provides a simple programming model based on for parallel processing and the Hadoop Distributed File System (HDFS) for high-throughput access to data, allowing applications to handle petabyte-scale datasets efficiently. Introduced in 2006 by developers and as part of the project, Hadoop was inspired by Google's 2003 Google File System paper and 2004 paper, addressing the need for scalable . Originally developed at Yahoo to support its search infrastructure, it evolved into a top-level project in 2008 and has since become a foundational technology for the ecosystem. The framework's architecture centers on three primary components: HDFS, which distributes data across nodes for redundancy and high availability; , a batch-processing engine that breaks tasks into map and reduce phases for parallel execution; and (Yet Another Resource Negotiator), introduced in Hadoop 2.0 in to decouple resource management from processing, enabling multi-tenancy and support for diverse workloads beyond , such as . Licensed under the 2.0, Hadoop is written primarily in and supports integration with a rich ecosystem of tools including Hive for SQL-like querying, for data transformation scripting, and HBase for NoSQL storage. As of November 2025, the latest stable release is version 3.4.2, which includes enhancements for security, performance, and compatibility with modern cloud environments. Widely adopted by organizations like Yahoo, , and for handling massive data volumes, Hadoop has democratized processing but faces competition from cloud-native alternatives like AWS EMR and Cloud Dataproc.

Introduction

Overview

Apache Hadoop is an framework that supports the distributed storage and processing of large datasets across clusters of computers, providing a reliable, scalable, and fault-tolerant platform for applications. It was inspired by two seminal publications: the programming model for simplifying large-scale data processing, as described in the 2004 paper "MapReduce: Simplified Data Processing on Large Clusters" by Jeffrey Dean and Sanjay Ghemawat, and the (GFS) for scalable distributed storage, outlined in the 2003 paper "The Google File System" by Sanjay Ghemawat et al. At its core, Hadoop embodies design principles focused on through data replication and automatic failure recovery, enabling it to operate reliably even with hardware failures; to clusters of thousands of nodes for handling massive workloads; and cost-effectiveness by leveraging inexpensive commodity hardware rather than specialized equipment. These principles allow Hadoop to manage challenges, such as data locality and parallel execution, without requiring custom infrastructure. The framework's primary components include the Hadoop Distributed File System (HDFS) for reliable distributed storage, Yet Another Resource Negotiator () for cluster resource management and job scheduling, and for batch processing of large datasets in parallel. As of November 2025, Hadoop is an active top-level project under , with version 3.4.2 serving as the latest stable release, issued on August 29, 2025. Hadoop forms the foundational layer for big data ecosystems, enabling the of petabyte-scale data volumes through integration with tools like and Hive, and remains widely adopted for , , and data warehousing in enterprise environments.

History

The development of Apache Hadoop originated in 2003 when and initiated the Nutch project, an open-source web search engine aimed at crawling and indexing large-scale web data, but encountered significant scalability challenges with web-scale datasets. To address these issues, the project drew inspiration from two seminal Google research papers: the (GFS) published in 2003 by Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, which described a distributed for large-scale data storage, and the paper from 2004 by Jeffrey Dean and Sanjay Ghemawat, outlining a for parallel processing of massive datasets. These influences shaped the core ideas behind Hadoop's distributed storage and processing capabilities. In 2006, joined Yahoo, where he integrated Hadoop as a sub-project of Nutch to support Yahoo's growing need for handling petabyte-scale data across thousands of machines; the first Hadoop cluster was deployed at Yahoo on January 28, 2006. The initial release, version 0.1.0, followed in April 2006, marking the project's entry into as an incubator project. By January 2008, Hadoop had matured sufficiently to graduate as an independent Apache top-level project, fostering broader community involvement beyond Yahoo. Key milestones in Hadoop's evolution include the release of version 1.0 on December 27, 2011, which stabilized the framework and added support for security features like Kerberos authentication, enabling enterprise adoption. Version 2.2.0, released on October 16, 2013, introduced (Yet Another Resource Negotiator), transforming Hadoop into a multi-tenancy platform that supported diverse data processing engines beyond . Hadoop 3.0 arrived on December 8, 2017, incorporating erasure coding for more efficient storage, reducing replication overhead while maintaining data reliability, alongside enhancements like Timeline Service v2 for improved job monitoring. More recent advancements reflect ongoing refinements for cloud integration and . Hadoop 3.4.0 was released on March 17, 2024, featuring full support for AWS SDK for v2 in the S3A connector to boost performance with object stores. The subsequent 3.4.2 release on August 29, 2025, introduced a leaner distribution tarball, further S3A optimizations for conditional writes, and fixes for multiple CVEs, enhancing usability in production environments. As of November 2025, development on 3.5.0-SNAPSHOT continues, focusing on bolstered protocols and tuning for hybrid cloud deployments. Hadoop's growth from a Yahoo-internal tool to a global standard was propelled by influential contributors like Cutting and Cafarella, alongside Google's foundational research, and expanded through active participation from companies such as and Hortonworks (merged into Cloudera in ), which drove widespread adoption across industries.

Architecture

Hadoop Distributed File System (HDFS)

The Hadoop Distributed File System (HDFS) serves as the primary storage layer in Apache Hadoop, designed to handle large-scale across clusters of commodity hardware while providing high-throughput access to application . It abstracts the underlying storage into a distributed filesystem that supports reliable, scalable storage for massive datasets, typically in the range of petabytes. HDFS achieves this by distributing across multiple nodes, ensuring through , and optimizing for batch-oriented workloads that involve sequential reads and writes of large files. HDFS employs a master-slave architecture, consisting of a single NameNode as the master server that manages the filesystem and metadata, such as file directories, permissions, and block locations, while regulating client access to files. The slave nodes, known as DataNodes, handle the actual storage of data blocks on local disks attached to the nodes where they run, typically one DataNode per cluster node. The NameNode maintains an in-memory image of the filesystem and a persistent edit log for transactions, periodically checkpointed to ensure durability. DataNodes register with the NameNode upon startup and periodically send block reports detailing the blocks they store. Files in HDFS are divided into fixed-size blocks for storage, with a default block size of 128 MB, though this can be configured per file or cluster-wide; larger blocks, such as 256 MB, are often used for very large files to reduce metadata overhead. Each block is replicated across multiple DataNodes to ensure , with the default replication factor set to 3, meaning three copies of each block are maintained. Block placement follows rack-awareness policies, where the NameNode prefers to store replicas on different racks to minimize risk from rack failures and optimize network locality for reads; for instance, the first replica is placed on the same node as the client, the second on a different node in the same rack, and the third on a node in a different rack. This rack-aware placement minimizes inter-rack traffic and optimizes bandwidth usage. This approach balances reliability and performance in multi-rack clusters. Fault tolerance in HDFS relies on automatic replication and monitoring mechanisms. DataNodes send heartbeat signals to the NameNode every few seconds; failure to receive a heartbeat for a configurable interval (default 10 minutes) causes the NameNode to mark the DataNode as dead and initiate replication of its blocks to other available nodes to maintain the desired replication factor. For NameNode failures, a Secondary NameNode periodically merges the edit log with the filesystem image to create checkpoints, reducing recovery time from hours to seconds in non-HA setups, though it does not provide automatic . High Availability (HA) configurations address this by deploying multiple NameNodes, with one active and others in standby mode, sharing edit logs via a shared storage system like NFS or a journal service, enabling automatic within seconds upon detecting the active NameNode's failure through mechanisms like ZooKeeper-based . Read and write operations in HDFS are optimized for access, particularly large sequential reads, which achieve high bandwidth by pipelining through multiple replicas and leveraging local reads when possible, aided by data locality to reduce bandwidth usage. During a write, the client contacts the NameNode for block locations, then streams to the first DataNode, which replicates it to subsequent nodes in a fashion; once written and closed, files follow a write-once-read-many semantic, where appends are supported but random writes or modifications to existing blocks are not permitted to simplify consistency. For reads, the client retrieves block locations from the NameNode and connects directly to the nearest DataNode for data transfer, bypassing the NameNode for the actual I/O to avoid bottlenecks. This design prioritizes high throughput over low latency for batch processing. Recent benchmarks show that Hadoop often exhibits higher latency than Spark, for example, 4.05 × 10^{-5} seconds per row in certain workloads. Despite its strengths, HDFS has notable limitations that make it unsuitable for certain workloads. It incurs high metadata overhead for small files, as each file requires an entry in the NameNode's memory regardless of size, potentially exhausting resources with millions of tiny files (e.g., under 128 MB), leading to NameNode scalability issues; techniques like HAR (Hadoop Archive) files are sometimes used to mitigate this. HDFS is also not designed for low-latency access, prioritizing over real-time queries, with operations like random seeks being inefficient due to its streaming focus. Furthermore, HDFS is not fully -compliant, relaxing requirements such as atomic file modifications and hard links to enable its distributed, model, which can complicate porting traditional POSIX applications without adaptations. Key configuration parameters in HDFS include dfs.blocksize, which sets the default block size (e.g., 134217728 bytes for 128 MB), dfs.replication for the default replication factor (default 3), and support for federation to scale beyond a single namespace by allowing multiple independent NameNodes, each managing its own namespace and block pool, with clients mounting them via a ViewFS client-side mount table. Federation enables horizontal scaling of metadata operations without a single point of failure in namespace management, though each NameNode still requires its own HA setup for failover. In Hadoop 3, HDFS introduced erasure coding as an alternative to traditional replication for improved space efficiency, using techniques like Reed-Solomon codes (e.g., RS-6-3 policy with 6 data cells and 3 parity cells) to provide equivalent while reducing storage overhead; this can achieve up to 1.5x space savings compared to 3x replication (from 200% overhead to 50%), particularly beneficial for cold data storage, though it trades some read/write performance for the gains. Erasure-coded files are striped across blocks in groups, with the NameNode tracking parity for reconstruction upon failures. Recent research has proposed advanced replication strategies for HDFS, such as clustering-aware multi-feature replication approaches that consider file access patterns and cluster characteristics to reduce latency and improve bandwidth utilization beyond the default rack-aware strategy.

Resource Management and Job Scheduling

In the initial Hadoop 1.x architecture, resource management and job scheduling were tightly coupled to the MapReduce framework through a centralized master-slave model. The JobTracker acted as the master daemon, responsible for accepting jobs from clients, scheduling tasks on available nodes, monitoring task progress, and handling failures by reassigning tasks. On each slave node, a TaskTracker executed the assigned map or reduce tasks, reporting status back to the JobTracker and utilizing local resources for computation. To address multi-user scenarios and prevent resource monopolization, Hadoop 1.x introduced pluggable schedulers: the Capacity Scheduler, which organized jobs into queues with guaranteed resource capacities to support multi-tenancy, and the Fair Scheduler, which dynamically allocated resources equally across active jobs over time to ensure fairness. However, this design created a in the JobTracker and limited cluster scalability to approximately 4,000 nodes due to its centralized oversight of both resources and job-specific logic. Hadoop 2.x introduced Yet Another Resource Negotiator () to decouple from job execution, enabling scalable, generalized cluster utilization. The ResourceManager serves as the global master, consisting of two primary components: the Scheduler, which allocates resources based on application requests without monitoring individual tasks, and the ApplicationsManager, which handles application submissions, lifecycle management, and negotiation of initial resources. On slave nodes, the NodeManager acts as a local agent, managing lifecycles—isolated units of resources specified by virtual cores (vCores) and —and enforcing usage limits while reporting node health and resource availability to the ResourceManager. For each submitted application, launches a lightweight, per-application ApplicationMaster that requests additional resources from the ResourceManager and coordinates task execution directly with NodeManagers, decentralizing job-specific scheduling. This model supports data locality by preferring allocations near HDFS data blocks to minimize network overhead. YARN's pluggable scheduler interface allows customization for diverse environments, with two primary implementations: the Capacity Scheduler and the Fair Scheduler. The Capacity Scheduler uses a hierarchical queue structure to enforce resource guarantees, where queues receive fixed capacities (e.g., as percentages of total cluster or vCores) and can elastically utilize excess resources from underutilized queues, supporting multi-tenant isolation through configurable limits and priorities. In contrast, the Fair Scheduler dynamically apportions resources among active applications via weighted shares, ensuring proportional allocation over time; it features hierarchical queues with policies like fair sharing or dominant resource fairness for CPU- balanced distribution, and allows idle resources to be reclaimed by higher-priority or new applications. Both schedulers handle resource requests by matching application demands to available node capacities, prioritizing based on queue configurations or fairness criteria. Fault tolerance in is enhanced through decentralized mechanisms and . Each ApplicationMaster manages its own task failures by requesting replacement containers from the ResourceManager, isolating issues to the application level without cluster-wide disruption. For the ResourceManager itself, (HA) configurations deploy an active-standby pair coordinated by , enabling automatic and state recovery to maintain scheduling continuity during failures. Cluster resource utilization and job performance are tracked using Hadoop's Metrics2 framework, which exposes counters, gauges, and histograms for components like the ResourceManager and NodeManagers. This integrates with Ganglia via a dedicated for real-time, distributed metric collection and visualization across large clusters, and with Ambari for centralized monitoring, alerting on resource thresholds, and dashboard-based analysis of utilization patterns. Compared to Hadoop 1.x, significantly improves scalability by distributing responsibilities, supporting clusters beyond 4,000 nodes through features like federation for sub-cluster management, and enabling non- workloads such as graph processing or interactive queries on the same infrastructure.

Data Processing Frameworks

Apache Hadoop's primary data processing framework is , a designed for distributed processing of large datasets across clusters of commodity hardware. In the paradigm, the input data is divided into independent chunks processed in parallel by map tasks, where each map function transforms input key-value pairs into intermediate key-value pairs, enabling scalable data transformation. The reduce phase then aggregates these intermediate outputs by grouping values associated with the same key, producing the final results. This model ensures through mechanisms like task retries and , allowing the framework to recover from node failures without restarting the entire job. The programming model for is primarily exposed through a , where developers implement subclasses of Mapper and Reducer interfaces to define the logic for . Mappers process input records, emitting intermediate key-value pairs, while reducers perform aggregation on those pairs. Input and output formats, such as TextInputFormat for line-based text files and SequenceFile for binary key-value storage, handle data and deserialization to optimize storage and transmission. Combiners, which act as mini-reducers during the map phase, enable partial aggregation of intermediate data on the same node to reduce network traffic and improve efficiency. During execution, a job is submitted to for and . The framework optimizes for data locality by scheduling tasks on nodes where the input data resides, minimizing data movement. Following the phase, the and sort phase partitions, sorts, and transfers intermediate data to reducers, ensuring keys are grouped correctly for aggregation. This flow supports reliable processing of batch workloads by leveraging to launch framework-specific containers. While MapReduce excels at , alternatives have emerged to address its limitations in complex workflows. Apache Tez provides a DAG-based execution engine that generalizes by allowing arbitrary task dependencies, reducing overhead for multi-stage jobs like those in Hive and , and serving as a for traditional . Apache , running on , offers that can be up to 100 times faster than for iterative algorithms, such as tasks, by caching data in RAM rather than writing intermediates to disk. Other frameworks, like Apache Giraph, extend Hadoop for specialized workloads such as iterative graph processing, integrating via to support , interactive, and pipelines. Performance in is constrained by frequent disk I/O for intermediate results, whereas Spark's memory-centric approach significantly lowers this overhead for repeated computations.

Version History

Hadoop 1

Hadoop 1, the initial stable release series of the Apache Hadoop framework, achieved version 1.0.0 on December 27, 2011, following six years of development from its origins in 2005. This milestone marked the framework's maturity for production use, with subsequent patches like 1.0.4 released in October 2012 to enhance stability and security features, including Kerberos authentication support. The architecture centered on a tightly integrated model where handled both data processing and resource management, limiting its flexibility compared to later iterations. At the core of Hadoop 1 was the framework, comprising a single master JobTracker that coordinated job scheduling, task monitoring, and failure recovery across the cluster, paired with one TaskTracker slave per node responsible for executing individual map and reduce tasks. This design emphasized locality by scheduling tasks on nodes hosting the relevant blocks in HDFS, reducing network overhead and improving efficiency. was achieved through mechanisms like , which launched backup instances of slow-running tasks to mitigate stragglers without diagnosing underlying issues, and rack awareness in HDFS, which distributed replicas across racks to enhance reliability and bandwidth utilization—for instance, placing one replica locally, one on a remote rack, and a third on another node in that remote rack for a replication factor of three. Despite these strengths, Hadoop 1 faced significant limitations due to its monolithic structure. The JobTracker became a bottleneck in large clusters, constraining to approximately 4,000 nodes and 40,000 concurrent tasks, beyond which performance degraded unpredictably from overload in scheduling and monitoring duties. It exclusively supported MapReduce workloads, offering no native accommodation for alternative processing paradigms like streaming or iterative algorithms, and its batch-oriented nature resulted in high latency unsuitable for interactive queries. Early adopters, such as Yahoo, leveraged Hadoop 1 for proofs-of-concept and production tasks like generating the search webmap index from billions of crawled pages, running on clusters exceeding 10,000 cores to process web-scale data for search queries. Hadoop 1 has been largely deprecated in favor of Hadoop 2's for improved scalability and workload diversity, though it persists in some legacy systems where migration costs outweigh benefits.

Hadoop 2 and YARN

Hadoop 2.2.0, released on October 15, 2013, marked a significant in the Hadoop ecosystem by introducing Yet Another Resource Negotiator () as its core innovation, decoupling from data processing to address the limitations of the earlier -centric architecture. rearchitects the cluster to treat as one of many possible processing engines, allowing Hadoop to function as a more flexible data operating system rather than a batch-only framework. YARN enables multi-tenancy by supporting multiple users and applications sharing cluster resources securely through configurable queues and access controls, while accommodating diverse workloads such as with and iterative computations with engines like . For reliability, Hadoop 2.4.0, released on April 7, 2014, introduced (HA) for the YARN ResourceManager via an active-standby mechanism, eliminating the present in prior versions. Additionally, YARN's Timeline Service version 1 provides centralized storage and querying of application history, enabling users to track job progress and diagnostics across the cluster. Compared to Hadoop 1, which was constrained to around 4,000 nodes due to the integrated JobTracker, dramatically improves scalability, supporting clusters of 10,000 or more nodes by distributing to per-node NodeManagers. It also facilitates long-running services, such as interactive queries or graph processing, by allowing applications to hold resources indefinitely without the batch-oriented constraints of earlier implementations. Hadoop 2 enhanced the Hadoop Distributed File System (HDFS) with NameNode federation, allowing multiple independent namespaces to scale metadata operations across larger clusters, and NameNode HA, which uses shared storage and automatic to ensure continuous availability. These improvements transformed Hadoop into a general-purpose platform for processing, fostering integration with emerging ecosystems like frameworks and real-time analytics tools. The Hadoop 2.x series saw iterative minor releases focused on stability, , and enhancements, culminating in version 2.10.0 on October 29, 2019, which included over 360 bug fixes and optimizations while maintaining with earlier 2.x features.

Hadoop 3 and Beyond

Apache Hadoop 3.0.0 was released on December 8, 2017, marking a significant in the framework's capabilities for handling large-scale . A key innovation in this version is the introduction of erasure coding in HDFS, which employs Reed-Solomon encoding to provide while substantially reducing storage requirements. For instance, the RS-6-3 policy uses six nodes to store three blocks with three parity blocks, achieving a 50% storage savings compared to traditional three-way replication for infrequently accessed cold , without compromising durability. Additionally, the Timeline Service v2 enhances monitoring by aggregating application history across clusters, enabling more efficient debugging and resource analysis through a centralized, scalable store. Performance improvements in Hadoop 3.0 build on the foundation by introducing opportunistic containers, which allow pending applications to utilize idle resources on nodes without guaranteed allocation, improving overall cluster utilization by up to 15-20% in mixed workloads. also gained native support for GPU scheduling, enabling for compute-intensive tasks like training directly within the framework. Security was bolstered in Hadoop 3.0 with fine-grained lists (ACLs) in HDFS, allowing precise permission management at the file and directory levels beyond basic semantics. Integration with Kerberos for authentication was deepened, supporting token-based delegation for secure cross-cluster operations, while wire encryption protects data in transit using and SASL mechanisms, addressing previous vulnerabilities in unencrypted communications. Subsequent releases have refined these foundations for modern environments. Hadoop 3.4.0, released on March 17, 2024, optimized cloud integrations with upgrades to the S3A connector using AWS SDK v2 for better performance and reliability in object storage, alongside enhancements to the ABFS connector for Azure Blob Storage, reducing latency in hybrid deployments. Hadoop 3.4.1, released on October 18, 2024, focused on stability with additional bug fixes. The 3.4.2 update on August 29, , introduced a lean binary distribution to minimize footprint, addressed multiple CVEs through dependency upgrades like and , and further improved S3A/ABFS throughput for cloud-native workloads. As of 2025, the 3.5.0-SNAPSHOT branch experiments with enhanced HDFS federation, supporting more scalable namespace management across multiple clusters. Hadoop 3.x maintains backward compatibility with Hadoop 2 APIs, allowing seamless migration of and applications, though certain legacy features like the old Timeline Service have been deprecated in favor of v2. Looking ahead, development efforts as of 2025 emphasize cloud-native adaptations, such as deeper integration for containerized deployments, and AI/ML optimizations including support for distributed training frameworks on . Sustainability initiatives, like energy-efficient scheduling algorithms that prioritize low-power nodes, are also gaining traction to reduce operational costs in large clusters. No plans for a Hadoop 4.0 have been announced, with the focus remaining on iterative enhancements to the 3.x line.

Ecosystem

The Apache Hadoop ecosystem is enriched by several related Apache projects that build upon its core components to provide higher-level abstractions for , storage, and . These projects leverage Hadoop Distributed File System (HDFS) for storage and Yet Another Resource Negotiator () for resource allocation, enabling scalable operations across distributed clusters. Apache Hive serves as a warehousing tool that allows users to query large datasets using HiveQL, a SQL-like , without needing to write complex programs. It includes a metastore for managing table schemas and metadata, translating queries into execution plans that run on backends such as , Tez, or Spark. Hive supports features like partitioning and bucketing for efficient organization, and recent versions, including Hive 4.1.0 released in July 2025, have enhanced capabilities such as full transactions for reliable updates. The project remains under active development by . Apache provides a distributed, scalable, database that models its architecture after Google's , offering column-oriented storage directly on HDFS for handling sparse datasets with billions of rows and millions of columns. It enables random, real-time read and write access, supporting operations like versioning and automatic sharding for and . integrates seamlessly with for and uses HDFS as its underlying storage layer, making it suitable for applications requiring low-latency access to . The project continues active development, with the latest stable release being version 2.5.13 as of November 2025. Apache Pig offers a scripting platform for analyzing large datasets through , a high-level procedural that simplifies the creation of data transformation pipelines, particularly for extract-transform-load (ETL) workflows. Unlike declarative SQL approaches, Pig allows iterative data processing and custom user-defined functions, compiling scripts into or Tez jobs for execution on Hadoop. It excels in handling and complex transformations that are cumbersome in other tools. Pig remains an active top-level project, with version 0.18.0 as the latest stable release supporting integrations with recent versions of the Hadoop ecosystem. Other essential projects include Apache Sqoop, which facilitates efficient bulk data transfer between Hadoop and relational databases via connectors that generate jobs for import and export operations, though it entered the Apache in 2021 and is no longer actively developed. Apache Oozie acts as a scheduler for orchestrating Hadoop jobs as directed acyclic graphs (DAGs), coordinating actions like Hive queries or scripts based on time or data triggers, but it was retired to the in 2025. Apache Ambari provides tools for provisioning, managing, and monitoring Hadoop clusters through a web-based interface and REST APIs, simplifying installation and configuration of ecosystem components; it was briefly retired to the in early 2022 but resurrected later that year and remains active, with the latest release version 3.0.0 as of November 2025. These projects are interdependent, relying on HDFS for persistent storage and for job scheduling and resource isolation, while many have evolved to support modern execution engines like Spark alongside traditional for improved performance. This tight integration forms a cohesive for end-to-end processing, with ongoing enhancements focused on and compatibility.

Integration with Other Technologies

Apache Hadoop integrates seamlessly with various external technologies to extend its functionality in distributed , enabling organizations to incorporate real-time streaming, advanced , and cloud-native storage within their data pipelines. These integrations leverage Hadoop's resource management and HDFS as a foundational storage layer, allowing hybrid environments where can flow between Hadoop and other systems without extensive reconfiguration. One prominent integration is with , an engine that runs natively on , providing faster iterative computations compared to traditional jobs. Spark replaces for many analytics workloads by offering libraries such as Spark SQL for structured queries, MLlib for algorithms, and GraphX for graph processing, all while accessing data in HDFS or compatible stores. This setup allows Spark applications to dynamically allocate resources via , supporting scalable and science tasks. For real-time data ingestion, Hadoop connects with through dedicated connectors like the HDFS Sink in Kafka Connect, facilitating the transfer of streaming events from Kafka topics directly into HDFS for . This integration supports high-throughput, fault-tolerant pipelines where Kafka acts as a buffer for incoming data, enabling Hadoop to handle continuous flows without interrupting ongoing jobs. In the realm of NoSQL and search technologies, Hadoop integrates with Elasticsearch via the Elasticsearch-Hadoop library, which enables indexing of HDFS data into for and . Similarly, data from can be imported into Hadoop using Sqoop with a compatible , allowing bidirectional movement for on wide-column stores. These connectors support efficient data synchronization, enhancing search capabilities and across diverse data models. Hadoop also provides robust connectors to cloud object storage services, treating them as alternatives to HDFS for scalable, cost-effective storage. The S3A connector enables direct read/write access to Amazon S3 buckets over HTTPS, supporting features like multipart uploads for large files. For Microsoft Azure, the Azure Blob File System (ABFS) driver integrates with Azure Data Lake Storage Gen2, offering hierarchical namespace support and OAuth authentication. Likewise, the Google Cloud Storage connector allows Hadoop jobs to operate on GCS buckets, optimizing for high-throughput operations in cloud environments. In and AI workflows, frameworks like and can run on through specialized runtimes such as Apache Submarine or TonY, distributing training across cluster nodes while sharing HDFS data. These integrations support GPU acceleration and fault-tolerant execution for deep learning models. Additionally, can orchestrate ML pipelines that incorporate Hadoop components, such as Spark jobs on with Hadoop configurations, bridging containerized AI with traditional processing. Despite these advancements, integrating Hadoop with external technologies presents challenges, including version compatibility issues that require aligning dependencies across ecosystems to avoid runtime errors. Performance tuning for hybrid workloads often involves optimizing network latency, buffer sizes, and to mitigate bottlenecks in polyglot persistence scenarios, where data spans multiple storage types. Addressing these requires careful configuration and testing to ensure reliable, efficient operations.

Use Cases and Applications

Industry Examples

Yahoo was one of the earliest adopters of Hadoop, deploying it in 2006 to handle massive-scale search indexing and ad technology processing across distributed clusters. By contributing significantly to the Apache project, Yahoo scaled its Hadoop infrastructure to process enormous log files and web data, enabling efficient handling of petabyte-scale workloads for its core operations. followed as an early adopter, leveraging Hadoop for analyzing vast user interaction data and developing Hive—a SQL-like interface for querying large datasets stored in Hadoop's distributed . In the technology sector, adopted Hadoop to power its recommendation systems, processing viewer behavior data through integrations with Spark for real-time analytics on Hadoop clusters. eBay utilizes Hadoop, particularly HBase, for real-time fraud detection by analyzing transaction patterns across terabytes of user-generated data daily. Financial institutions have integrated Hadoop for advanced analytics. previously employed Hadoop to process unstructured data for risk modeling and fraud detection. previously relied on Hadoop distributions like for customer analytics. Retail giants have harnessed Hadoop for operational efficiency. Walmart uses Hadoop to optimize , analyzing sales and inventory data to forecast demand and reduce logistics costs. Alibaba implements Hadoop in its e-commerce data lakes to manage petabytes of transaction and user data, facilitating real-time personalization and inventory decisions. In the public sector, NASA applies Hadoop for high-performance analytics on datasets, using to process climate and satellite data across distributed nodes. CERN operates Hadoop clusters since 2013 for analysis, storing and querying terabytes of experimental data from accelerators like the LHC via HDFS and . Hadoop deployments have achieved massive scale, with some clusters exceeding nodes and handling over 500 petabytes of in production environments. Post-2020, many organizations shifted to hybrid cloud-on-premises models, combining Hadoop's core components with for greater flexibility and cost control; for instance, organizations like and have migrated from on-premises Hadoop to cloud-based platforms such as AWS and . These adoptions have delivered significant outcomes, including cost savings through the use of inexpensive commodity hardware for reliable, fault-tolerant storage and . Hadoop fueled innovation during the boom, enabling scalable analytics that contributed to the growth of the Hadoop market, which reached approximately $25 billion by 2020.

Common Workloads

Apache Hadoop supports a variety of common workloads, leveraging its distributed storage and processing capabilities to handle large-scale data tasks. One of the primary workloads is , where Hadoop excels in extract-transform-load (ETL) jobs for analyzing vast datasets, such as daily aggregation of web logs using or Spark. In these scenarios, input data is divided into independent chunks processed in parallel across a cluster, enabling efficient handling of terabyte-scale logs for tasks like summarization and filtering. Data warehousing represents another key workload, facilitated by tools like Hive, which provides a SQL-like interface for ad-hoc querying and (OLAP) on petabyte-scale datasets stored in HDFS. Hive transforms structured data into tables, allowing users to perform complex joins, aggregations, and multidimensional analysis without writing low-level code, thus supporting operations on historical data. Machine learning workloads on Hadoop involve distributed training of models using libraries such as for scalable algorithms in , clustering, and recommendation systems, or Spark's MLlib for feature extraction and iterative training directly on HDFS data. These frameworks distribute computations across nodes to process massive feature sets, enabling applications like on user behavior datasets. Graph processing and streaming workloads are addressed through integrations like Giraph for iterative analysis of large graphs, such as connections, where algorithms propagate messages across vertices in a distributed manner. For near-real-time processing, Kafka streams data into Hadoop ecosystems, combining batch layers with speed layers to handle continuous event flows, such as real-time log ingestion for . Search and indexing tasks commonly use to build inverted indexes from document collections, mapping terms to their locations across files for efficient retrieval in custom search engines. This process involves a map phase to emit term-document pairs and a reduce phase to consolidate postings lists, supporting scalable text search over web-scale corpora. Performance patterns in Hadoop workloads often include iterative algorithms, such as approximations of , which require multiple MapReduce iterations to converge on node importance in graphs, with each pass refining rank values based on incoming links. Handling skewed data distributions is managed through sampling techniques, where representative subsets of input are analyzed to adjust partitioning and balance load across reducers, mitigating bottlenecks in uneven workloads. Evolving trends in Hadoop workloads reflect a shift toward lambda architectures, which integrate for accurate, comprehensive views with streaming layers for low-latency updates, allowing hybrid systems to serve both historical and real-time insights from the same data sources.

Deployment and Management

Cloud-Based Deployments

Cloud-based deployments of Apache Hadoop enable organizations to leverage scalable, from major cloud providers, eliminating the need for on-premises hardware management while integrating with cloud-native storage and compute resources. These deployments typically use managed Hadoop distributions that support core components like HDFS alternatives (such as or Storage) and for resource orchestration, allowing dynamic scaling based on workload demands. Key providers include Amazon EMR, which offers fully managed Hadoop clusters with auto-scaling capabilities to adjust instance counts based on metrics like CPU utilization or memory pressure, supporting Hadoop versions up to 3.4.1 as of 2025. Azure HDInsight provides managed Hadoop 3.3 clusters (via HDInsight 5.1), enabling integration with Azure services and automatic scaling for . Cloud Dataproc supports ephemeral Hadoop clusters that provision resources on-demand and terminate after job completion, using recent Hadoop 3.x releases, such as 3.3.6 in Dataproc 2.3 images as of 2025, for short-lived workloads. Benefits of cloud-based Hadoop include elasticity through pay-per-use pricing, where users only pay for active compute and storage, and the use of durable object stores like S3 as HDFS alternatives, which offer without local replication overhead. These setups also facilitate seamless integration with cloud services, such as AWS SageMaker or Google Vertex AI, for end-to-end analytics pipelines. Additionally, handle patching, upgrades, and , reducing operational overhead compared to self-managed clusters. Challenges encompass data transfer costs, as ingress/egress fees can accumulate for large-scale Hadoop jobs moving data across regions or hybrid boundaries. arises from provider-specific configurations, potentially complicating migrations, while compatibility issues with object stores require optimizations like the S3A committer to prevent double writes and ensure atomic commits during or Spark operations. Performance tuning for latency is also critical, as object stores exhibit higher access times than traditional HDFS. Hybrid models combine on-premises HDFS for sensitive data with cloud bursting, where allows jobs to overflow from local clusters to cloud resources during peak loads, providing a single namespace across environments. This approach uses 's sub-cluster federation to distribute applications dynamically, maintaining data locality where possible while scaling compute elastically. Best practices include utilizing spot instances in Amazon EMR for non-critical workloads to achieve up to 90% cost savings on compute, while diversifying instance types to mitigate interruptions. Security is enhanced by assigning IAM roles for fine-grained to S3 buckets and services, avoiding static credentials. Monitoring leverages cloud-native tools like AWS CloudWatch for real-time metrics on cluster health, queues, and job performance, with alerts configured for anomalies. As of 2025, trends show a shift toward serverless Hadoop via offerings like Amazon EMR Serverless, which automatically provisions and scales resources without cluster management, ideal for intermittent workloads. Market analyses indicate that Hadoop-as-a-Service deployments are growing rapidly, with the global market projected to reach $1,091 billion by 2034, driven by over 90% enterprise cloud adoption and a preference for managed services in new big data initiatives.

Commercial Support and Distributions

Commercial distributions of Apache Hadoop provide enterprise-grade enhancements, including management tools, security features, and certified integrations, built on the open-source core to support large-scale . Major vendors include , which offers the Cloudera Data Platform (CDP) integrating Apache Hadoop 3.1.1 (with Cloudera patches) with tools like Cloudera Manager for cluster automation and Ranger for fine-grained authorization and auditing. Following the 2019 merger of Hortonworks into , the legacy Hortonworks Data Platform (HDP) has been phased out, with end-of-support announced for older versions to encourage migration to CDP, which maintains compatibility with Hadoop's and HDFS components. Amazon Web Services (AWS) delivers Hadoop through Amazon EMR, a managed service that deploys fully configured clusters running Apache Hadoop alongside other frameworks, emphasizing scalability and integration with AWS storage like S3. IBM provides support via its watsonx platform, incorporating an Execution Engine for Apache Hadoop updated in March 2025, which builds on the earlier (IOP) for enterprise distributions with added analytics and governance features. These distributions often include certified hardware compatibility and testing to ensure adherence to Apache standards. Enterprise support models from these vendors typically encompass 24/7 technical assistance, proactive patching for (CVEs), and migration services from pure open-source Hadoop setups to managed environments. For instance, offers comprehensive support contracts that include rapid response for production issues and customized upgrades. The Apache Hadoop , owned by (ASF), requires commercial variants to comply with branding guidelines, prohibiting unauthorized use of "Hadoop" in product names without ; distributions must pass ASF conformance tests for guarantees. Post-2020 market consolidation, exemplified by the Cloudera-Hortonworks merger, has shifted focus toward cloud-native offerings like CDP's public cloud edition, reducing on-premises dependency while preserving Hadoop's distributed processing capabilities. By 2025, the Hadoop distribution market emphasizes AI-enhanced management, with features like automated cluster optimization and integrated into platforms such as CDP to improve by up to 25% in AI workloads.

References

  1. https://hadoop.apache.org/
  2. https://www.dataversity.net/articles/a-brief-history-of-the-hadoop-ecosystem/
  3. https://aws.amazon.com/what-is/hadoop/
  4. https://hadoop.apache.org/releases.html
  5. https://news.apache.org/foundation/entry/the_apache_software_foundation_announces21
  6. https://engineering.fb.com/2010/12/10/core-infra/looking-at-the-code-behind-our-three-uses-of-apache-hadoop/
  7. https://innovation.ebayinc.com/stories/eagle-is-your-hadoop-data-secured/
  8. https://www.ibm.com/think/topics/hadoop
  9. https://cloud.google.com/dataproc
  10. https://research.google.com/archive/mapreduce-osdi04.pdf
  11. https://research.google.com/archive/gfs-sosp2003.pdf
  12. http://infolab.stanford.edu/~ragho/hive-icde2010.pdf
  13. https://www.cio.com/article/243204/apache-hadoop-turns-10.html
  14. https://adtmag.com/articles/2016/01/28/hadoop-turns-10.aspx
  15. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/release/0.1.0/RELEASENOTES.0.1.0.html
  16. https://tom-e-white.com/2008/01/hadoop-is-now-apache-top-level-project.html
  17. https://news.apache.org/foundation/entry/the_apache_software_foundation_announces48
  18. https://hadoop.apache.org/release/3.0.0.html
  19. https://hadoop.apache.org/release/3.4.2.html
  20. http://apache.github.io/hadoop/
  21. https://sudipchakrabarti.substack.com/p/when-hadoop-was-king-and-yahoo-was
  22. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
  23. https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
  24. https://hadoop.apache.org/docs/r2.7.4/hadoop-project-dist/hadoop-common/RackAwareness.html
  25. https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
  26. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html
  27. https://ieeexplore.ieee.org/document/11153845
  28. https://hadoop.apache.org/docs/r3.1.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
  29. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/Federation.html
  30. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html
  31. https://www.sciencedirect.com/science/article/pii/S187705092503159X
  32. https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html
  33. https://hadoop.apache.org/docs/r1.2.1/capacity_scheduler.html
  34. https://hadoop.apache.org/docs/r1.2.1/fair_scheduler.html
  35. https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
  36. https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
  37. https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
  38. https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html
  39. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Metrics.html
  40. https://ambari.apache.org/
  41. https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
  42. https://tez.apache.org/
  43. https://spark.apache.org/mllib/
  44. https://giraph.apache.org/
  45. https://hadoop.apache.org/release/1.0.0.html
  46. https://endoflife.date/apache-hadoop
  47. https://courses.grainger.illinois.edu/cs423/sp2016/lectures/Hadoop.pdf
  48. https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
  49. https://jeremy.zawodny.com/blog/archives/009992.html
  50. https://dl.acm.org/doi/10.1145/2523616.2523633
  51. https://hadoop.apache.org/release/2.4.0.html
  52. https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/TimelineServer.html
  53. https://hadoop.apache.org/release/2.10.0.html
  54. https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/release/3.0.0/RELEASENOTES.3.0.0.html
  55. https://hadoop.apache.org/docs/r3.4.0/hadoop-project-dist/hadoop-common/release/3.4.0/RELEASENOTES.3.4.0.html
  56. https://hadoop.apache.org/docs/stable/
  57. https://hive.apache.org/
  58. https://hbase.apache.org/
  59. https://hbase.apache.org/downloads.html
  60. https://pig.apache.org/
  61. https://sqoop.apache.org/
  62. https://oozie.apache.org/
  63. https://spark.apache.org/docs/latest/
  64. https://kafka.apache.org/documentation/#connect
  65. https://github.com/confluentinc/kafka-connect-hdfs
  66. https://hadoop.apache.org/docs/stable/hadoop-azure/index.html
  67. https://www.kubeflow.org/docs/components/spark-operator/user-guide/writing-sparkapplication/
  68. https://www.wired.com/2011/10/how-yahoo-spawned-hadoop/
  69. https://www.netcomlearning.com/blog/what-is-hadoop
  70. https://www.projectpro.io/article/apache-hadoop-turns-10-the-rise-and-glory-of-hadoop/211
  71. https://aquare.la/en/what-is-hdfs-and-what-is-its-impact-on-business/
  72. https://data-flair.training/blogs/big-data-case-studies/
  73. https://www.projectpro.io/article/how-jpmorgan-uses-hadoop-to-leverage-big-data-analytics/142
  74. https://www.nextplatform.com/2015/07/28/lack-of-legacy-lets-capital-one-build-nimble-infrastructure/
  75. https://hdfstutorial.com/blog/big-data-ecommerce-case-studies/
  76. https://ntrs.nasa.gov/citations/20170000324
  77. https://indico.cern.ch/event/716795/contributions/2946453/attachments/1645011/2628931/BigData_at_CERN_JP.pdf
  78. https://www.huaweicloud.com/intl/en-us/product/mrs.html
  79. https://www.researchgate.net/publication/394021647_The_Evolution_of_Big_Data_Workflows_From_On-_Premise_Hadoop_to_Cloud-Based_Architectures
  80. https://medium.com/next-at-chase/how-chase-transitioned-its-data-lake-from-hadoop-to-aws-part-1-7cf4186d034a
  81. https://medium.com/capital-one-tech/understanding-the-evolution-of-data-lakes-37d13b809be9
  82. https://www.sciencedirect.com/topics/computer-science/hadoop
  83. https://www.marketsandmarkets.com/Market-Reports/hadoop-big-data-analytics-market-766.html
  84. https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-introduction
  85. https://kafka.apache.org/documentation/streams/
  86. https://dl.acm.org/doi/10.1145/1989323.1989425
  87. https://dl.acm.org/doi/10.1145/2391229.2391245
  88. https://www.databricks.com/glossary/lambda-architecture
  89. https://aws.amazon.com/emr/features/hadoop/
  90. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-7110-release.html
  91. https://docs.cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.3
  92. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop.html
  93. https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-5x-component-versioning
  94. https://aws.amazon.com/emr/features/
  95. https://intellias.com/big-data-cloud/
  96. https://cloud.google.com/blog/products/storage-data-transfer/hdfs-vs-cloud-storage-pros-cons-and-migration-tips
  97. https://www.cloudzero.com/blog/cloud-computing-statistics/
  98. https://hadoop.apache.org/docs/r3.3.4/hadoop-aws/tools/hadoop-aws/committers.html
  99. https://juicefs.medium.com/from-hadoop-to-cloud-why-and-how-to-decouple-storage-and-compute-in-big-data-platforms-8e01e47e9ea0
  100. https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/Federation.html
  101. https://onlinelibrary.wiley.com/doi/10.1002/cpe.5662
  102. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html
  103. https://aws.amazon.com/blogs/compute/best-practices-to-optimize-your-amazon-ec2-spot-instances-usage/
  104. https://aws.amazon.com/blogs/big-data/petabyte-scale-data-migration-made-simple-appsflyers-best-practice-journey-with-amazon-emr-serverless/
  105. https://www.precedenceresearch.com/hadoop-as-a-service-market
  106. https://sqmagazine.co.uk/cloud-adoption-statistics/
  107. https://docs.cloudera.com/runtime/7.3.1/public-release-notes/topics/rt-runtime-component-versions_731.html
  108. https://www.cloudera.com/products/cloudera-data-platform.html
  109. https://www.cloudera.com/about/news-and-blogs/press-releases/2019-01-03-cloudera-and-hortonworks-complete-planned-merger.html
  110. https://www.acceldata.io/guide/data-observability-support-hadoop
  111. https://www.ibm.com/docs/en/watsonx/w-and-w/2.1.0?topic=new-execution-engine-apache-hadoop
  112. https://cwiki.apache.org/confluence/display/HADOOP2/Distributions%2Band%2BCommercial%2BSupport
  113. https://www.openlogic.com/solutions/apache-hadoop-support
  114. https://www.gartner.com/reviews/market/hadoop-distributions
  115. https://www.apache.org/foundation/marks/list/
  116. https://www.apache.org/foundation/marks/
  117. https://www.futuremarketinsights.com/reports/hadoop-distribution-market
  118. https://www.cloudera.com/blog/business/enterprise-ai-and-data-architecture-in-2025-from-experimentation-to-integration.html
Add your contribution
Related Hubs
User Avatar
No comments yet.