Hubbry Logo
Apache MesosApache MesosMain
Open search
Apache Mesos
Community hub
Apache Mesos
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Apache Mesos
Apache Mesos
Apache Mesos
View on Wikipedia
from Wikipedia
Apache Mesos
DeveloperApache Software Foundation
Stable release
1.11.0 / November 24, 2020; 5 years ago (2020-11-24)[1]
Written inC++
TypeCluster management software
LicenseApache License 2.0
Websitemesos.apache.org Edit this on Wikidata
RepositoryMesos Repository

Apache Mesos is an open-source project to manage computer clusters. It was developed at the University of California, Berkeley.

History

[edit]

Mesos began as a research project in the UC Berkeley RAD Lab by then PhD students Benjamin Hindman, Andy Konwinski, and Matei Zaharia, as well as professor Ion Stoica. The students started working on the project as part of a course taught by David Culler. It was originally named Nexus but due to a conflict with another university's project, was renamed to Mesos.[2]

Mesos was first presented in 2009 (while still named Nexus) by Andy Konwinski at HotCloud '09 in a talk accompanying the first paper published about the project.[3] In 2011 a more developed version was presented in a talk by Zaharia at the Usenix Symposium on Networked Systems Design and Implementation. Specifically, he presented the paper on Mesos authored by Benjamin Hindman, Andy Konwinski Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica and Zaharia himself.[4]

On July 27, 2016, the Apache Software Foundation announced version 1.[5] It added the ability to centrally supply Docker, rkt and appc instances.[6]

On April 5, 2021, it was voted to move Mesos to the Apache Attic,[7] however the vote was cancelled two days later due to increased interest.[8]

As of August 2025, this project was retired [9] and placed into the attic October 2025.[10] As such, all development on this project has ceased.

Technology

[edit]

Mesos uses Linux cgroups to provide isolation for CPU, memory, I/O and file system.[11] Mesos is comparable to Google's Borg scheduler, a platform used internally to manage and distribute Google's services.[12]

Apache Aurora
DeveloperApache Software Foundation
Final release
0.22.0 / December 12, 2019; 6 years ago (2019-12-12)[13]
Written inJava, Python
TypeMesos Framework
LicenseApache License 2.0
Websiteaurora.apache.org
RepositoryAurora Repository

Apache Aurora

[edit]

Apache Aurora is a Mesos framework for both long-running services and cron jobs, originally developed by Twitter starting in 2010 and open sourced in late 2013.[14] It can scale to tens of thousands of servers, and holds many similarities to Borg[15][16] including its rich domain-specific language (DSL) for configuring services. As of February 2020 the project was retired to the Attic.[17] A fork of the project was maintained by former members, hosted on GitHub under the name Aurora Scheduler.[18]

Chronos

[edit]

Chronos is a distributed cron-like system which is elastic and capable of expressing dependencies between jobs.[19]

Marathon

[edit]

Marathon is promoted for platform as a service or container orchestration system scaling to thousands of physical servers. It is fully REST-based and allows canary-style deployments and deployment topologies. It is written in the programming language Scala.[20]

Users

[edit]

The social networking site Twitter began using Mesos and Apache Aurora in 2010, after Hindman gave a presentation to a group of Twitter engineers.[12]

Airbnb said in July 2013 that it uses Mesos to run data processing systems like Apache Hadoop and Apache Spark.[21]

The Internet auction website eBay stated in April 2014 that it used Mesos to run continuous integration on a per-developer basis. They accomplish this by using a custom Mesos plugin that allows developers to launch their own private Jenkins instance.[22]

In April 2015, it was announced that Apple service Siri is using its own Mesos framework called Jarvis.[23]

In August 2015, it was announced that Verizon selected Mesosphere's DC/OS, which is based on open source Apache Mesos, for data center service orchestration.[24]

In November 2015, Yelp announced they had been using Mesos and Marathon for a year and a half for production services.[25]

Commercial support

[edit]

Software startup Mesosphere, Inc. sells the Datacenter Operating System, a distributed operating system, based on Apache Mesos.[26] In September 2015, Microsoft announced a commercial partnership with Mesosphere to build container scheduling and orchestration services for Microsoft Azure.[27] In October 2015, Oracle announced support for Mesos through Oracle Container Cloud Service.[28]

See also

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Apache Mesos

View on Grokipedia
from Grokipedia
Apache Mesos is an open-source project that functions as a distributed systems kernel, providing efficient resource isolation and sharing across diverse cluster frameworks such as Hadoop, Spark, and MPI, by abstracting CPU, memory, disk, and other resources in large-scale datacenter environments. It enables fine-grained through a two-level scheduling architecture, where a central master offers available resources to application-specific frameworks that then manage their own task scheduling. Developed to address the inefficiencies of siloed resource usage in multi-framework clusters, Mesos supports scaling to tens of thousands of nodes and integrates with container technologies like Docker for deploying workloads across clouds and on-premises infrastructure. Originating as a research project at the in 2009, Mesos was initially proposed in a 2011 paper by Benjamin Hindman, Andy Konwinski, , , Anthony D. Joseph, Randy H. Katz, , and , who aimed to create a platform for sharing commodity clusters among diverse workloads while improving utilization by up to 1.5x compared to traditional approaches. The project entered the Incubator in 2010 and graduated to a top-level project on July 24, 2013, after demonstrating production use at organizations like and for running and web services. Mesos reached its 1.0 stable release on July 27, 2016, incorporating features like high-availability masters using , pluggable isolators for resource constraints, and HTTP APIs for framework development and monitoring. At its core, Mesos employs a : the master daemon manages resource offers to frameworks, while agent daemons on cluster nodes enforce isolation and execute tasks, supporting cross-platform operation on , macOS, and Windows. It facilitated the development of ecosystems like DC/OS for cloud-native orchestration, though adoption shifted toward alternatives like in later years. Following declining community activity, the project was retired by in August 2025 and moved to the in October 2025, with read-only archives preserved and community forks like Clusterd encouraged for continued maintenance.

History

Origins and Development

Apache Mesos originated as a research project in 2009 at the , developed by Benjamin Hindman, Andy Konwinski, Matei Zaharia, and , along with collaborators including , Anthony D. Joseph, Randy Katz, and . The project emerged from efforts to address the growing challenges of managing large-scale data centers, where commodity clusters were increasingly underutilized due to the silos created by specialized frameworks. The initial motivation stemmed from the inefficiencies in resource sharing across diverse workloads, such as Hadoop for and MPI for , which often led to low cluster utilization—typically around 10-20% in practice—because each framework monopolized entire nodes. Inspired by operating system kernels that abstract hardware for multiple applications, the team aimed to create a platform for fine-grained resource sharing that improved utilization while preserving data locality and avoiding costly data replication across frameworks. Early prototypes focused on enabling multiple frameworks to coexist on shared clusters without interference, demonstrating up to 2.1-fold improvements in job completion times for Hadoop workloads in evaluations on a 50-node cluster. These prototypes culminated in the seminal 2011 NSDI paper, "Mesos: A Platform for Fine-Grained Resource Sharing in the ," which detailed the system's and empirical results from real-world deployments, including integration with Hadoop and MPI. In 2010, Mesos entered the Incubator as an open-source project, marking its shift from academic research to broader community development. It graduated to become a top-level project in July 2013, reflecting its maturity and adoption by organizations like for production-scale cluster management. At its core, Mesos introduced two-level scheduling to decouple from task placement: a central Mesos scheduler offers available resources to framework-specific schedulers, which then decide how to utilize them, enabling flexible policies like fair sharing or capacity guarantees. Resource isolation was achieved through OS-level mechanisms, such as , to ensure tasks from different frameworks do not interfere with each other's performance on shared nodes. These principles laid the foundation for Mesos as a distributed kernel-like layer, prioritizing and adaptability for multi-framework environments.

Key Milestones and Releases

Apache Mesos entered the Apache Incubator in 2010, with its initial development stemming from a research project at the . The project's first incubator release occurred in 2012, marking the beginning of its formal open-source evolution under Apache governance. A significant milestone came on July 24, 2013, when Mesos graduated to become a top-level Apache project, recognizing its maturity and growing community adoption for resource management in large-scale clusters. In September 2014, Mesos 0.20.0 introduced native support for Docker containers, allowing frameworks to launch tasks using Docker images and a subset of Docker options, which broadened its appeal for containerized workloads. The integration of with Mesos around 2013 enabled efficient resource sharing for data processing frameworks, with Spark 0.5.0 explicitly supporting Mesos 0.9 for running analytics workloads on shared clusters. Mesos 1.0.0, released on July 27, 2016, represented a major maturation point, featuring a new HTTP API for improved interoperability, a unified containerizer supporting multiple runtimes including Docker and AppC, and enhanced high-availability for the master process through integration for and state replication. This version solidified Mesos as a production-ready platform for fault-tolerant distributed systems.
VersionRelease DateKey Features
0.20.0September 3, 2014Native Docker container support
1.0.0July 27, 2016HTTP API, unified containerizer, ZooKeeper-based HA master
1.4.0September 18, 2017Enhanced GPU resource isolation and disk isolation for better support of compute-intensive tasks
1.9.0September 2019Improvements to persistent volumes, agent draining, and quota limits for more reliable stateful workloads
Integration with Apache Kafka emerged prominently in 2015–2016, with frameworks like Kafka on Mesos enabling elastic scaling of Kafka brokers across cluster resources, supporting high-throughput streaming applications. By 2016, the project had surpassed 100 contributors for recent releases, reflecting robust community growth that peaked in the late 2010s with hundreds of active participants driving enhancements. The final active release, 1.11.0, arrived on November 24, 2020, incorporating bug fixes and minor improvements amid declining development activity; subsequent maintenance focused on patches rather than new features.

Retirement

In July 2025, the Apache Mesos Committee (PMC) initiated and concluded a formal binding vote to the project on July 22, 2025, citing prolonged inactivity and a lack of active maintainers as primary reasons. This decision followed years of declining community contributions, with GitHub commit activity dropping significantly after 2019 and no substantial updates since then, as well as an earlier unsuccessful vote in April 2021 that was cancelled after two days due to renewed interest. The reflected broader industry shifts toward for container orchestration, which had gained dominance in managing distributed systems. A key factor in Mesos' decline was the strategic pivot by its primary backer, (rebranded as D2iQ in ), which shifted focus to Kubernetes-based solutions like Konvoy starting in , effectively ending support for Mesos-centric products such as DC/OS by 2021. This commercial redirection reduced funding and development resources for the open-source project, exacerbating the maintainer shortage. The retirement process continued with Apache Board approval on August 20, 2025, moving Mesos to the Apache Attic for archival purposes. Project resources, including mailing lists, JIRA issue tracker, and the Git repository, were subsequently made read-only to preserve historical data while preventing further changes. The official announcement of the retirement was issued on October 17, 2025. Mesos had no new feature releases after version 1.11.0, issued on , 2020, though minor security patches were applied sporadically until early 2021. In the immediate aftermath, the Mesos website was redirected to its Apache Attic page, providing read-only access to documentation and archives. The retirement notice encouraged users to consider community forks, such as Clusterd, an active continuation of Mesos maintained on since early 2025, for ongoing needs in resource isolation and cluster management.

Architecture

Core Components

Apache Mesos is built around a distributed comprising master nodes, agent nodes, and framework-specific components that enable efficient across clusters. The system employs a two-level scheduling model where the Mesos master allocates to frameworks, which then manage their own task scheduling. This design allows multiple diverse frameworks to coexist on the same physical infrastructure while providing fine-grained resource isolation. The master node serves as the central coordinator in the Mesos cluster, responsible for managing agent daemons, tracking the overall state of resources, and offering available resources to registered frameworks based on configurable allocation policies such as fair sharing or strict priority. Masters support through a replicated setup with , ensuring by allowing backup masters to take over seamlessly if the active leader fails. This replication is orchestrated via , which handles , configuration management, and state synchronization across multiple masters, agents, and schedulers. Agent nodes (previously known as slave nodes) operate on each machine in the cluster, reporting available resources—such as CPUs, memory, disk, and ports—to the master and enforcing resource isolation for tasks launched on that node. Agents execute tasks through framework-provided executors and utilize pluggable isolators to manage and limit resource usage, including CPU shares, memory limits, disk volumes, network ports, and GPU allocation, primarily leveraging Linux control groups (cgroups) and namespaces for isolation on supported platforms. This modular isolation mechanism allows operators to customize enforcement for specific environments without altering the core Mesos codebase. Framework-specific schedulers register with the master to receive resource offers and decide how to allocate those resources to tasks, enabling frameworks to implement their own scheduling logic independently of Mesos. Once resources are allocated, executors—also framework-defined—run on agent nodes to launch and monitor individual tasks, handling the actual execution and reporting status back through the agent to the scheduler. These components decouple from task execution, allowing Mesos to support a wide variety of workloads efficiently. Mesos provides HTTP APIs for programmatic interaction with the cluster, including operator endpoints for managing masters and agents, as well as monitoring endpoints to query tasks, resources, and cluster state; these APIs form the basis for developing distributed applications and integrating with external tools. A web-based is accessible via the master's HTTP interface, offering a visual overview of cluster utilization, active tasks, and resource distribution to aid in monitoring and debugging. Mesos demonstrates cross-platform compatibility, running on Linux (64-bit), macOS (64-bit), and Windows (experimental support for agents only, requiring Windows 10 Creators Update or Windows Server 2016 and later). This support is facilitated by the pluggable isolators and containerizers, which adapt to platform-specific mechanisms for resource isolation, such as POSIX compliance on Unix-like systems and experimental features on Windows.

Resource Management and Scheduling

Apache Mesos abstracts cluster resources such as CPU, , disk, and as commoditized units that can be offered to frameworks in a fine-grained manner. These resources are represented using three types: scalars for floating-point values like 1.5 CPUs or 8192 MB of (with three decimal places of precision), ranges for continuous intervals such as port numbers (e.g., [21000-24000]), and sets for discrete items like custom resource identifiers. Predefined scalar resources include cpus, mem (in MB), disk (in MB), and gpus (whole numbers only), while ports use ranges; frameworks receive these abstractions via or key-value pairs to enable efficient allocation across diverse workloads. Mesos employs a two-level scheduling model to facilitate multi-tenant cluster operation, where the Mesos master allocates resources to registered frameworks, and the frameworks' schedulers make acceptance decisions based on their specific needs. In this model, the master periodically detects unused resources on agents and issues resource offers—bundles containing available units like 4 CPUs and 4 GB of memory—to subscribed frameworks. The offer cycle operates continuously: upon receiving an offer via the SUBSCRIBE call, a framework's scheduler can accept it using an ACCEPT call with the offer ID, applying filters to reject insufficient or unsuitable resources (e.g., based on location or attributes), and specifying operations to launch tasks either as individual processes or grouped containers. Accepted offers trigger task launches, where tasks execute via executors on agents, supporting data locality optimizations like delay scheduling to achieve up to 95% locality with minimal wait times. To ensure secure multi-tenancy, Mesos implements resource isolation through Linux-specific mechanisms, including control groups () for limiting CPU and memory usage, namespaces for process and network isolation, and seccomp filters to enforce security policies by restricting system calls. These isolators are modular, allowing operators to enable or customize them, and integrate with container runtimes such as Docker for image-based launches or AppC for composable isolation layers. This setup prevents interference between tasks from different frameworks while maintaining lightweight overhead. Mesos demonstrates strong , supporting clusters with over 10,000 nodes through its distributed architecture and low-latency operations, such as task launches under 1 second even at 50,000 emulated nodes. is achieved via periodic agent reregistration with the master (every 10 seconds by default) and automatic task relaunch upon recovery, complemented by for replicated master election with 4-8 second times. Agents handle disconnections gracefully by buffering updates and resynchronizing state, ensuring minimal disruption in large-scale environments. For guaranteed allocation in multi-tenant settings, Mesos supports resource reservations tied to , which represent groups of frameworks or users. Static reservations, configured at agent startup via flags like --resources='cpus([role](/page/Role)):8;mem([role](/page/Role)):4096', dedicate resources to specific and require restarts to modify. Dynamic reservations, introduced in version 0.23.0, allow runtime adjustments through framework operations (e.g., Offer::Operation::Reserve) or operator HTTP endpoints, enabling partial unreservations without interrupting active tasks. Hierarchical , such as eng/backend, facilitate and refinement of reservations, while role-based quotas enforce upper limits on total allocatable resources per to prevent overcommitment. Fair sharing among uses weighted Dominant Resource Fairness (wDRF), where weights (default 1) determine proportional allocation, configurable via the /weights endpoint.

Frameworks and Ecosystem

Integrated Frameworks

Apache Mesos employs a framework model where external schedulers, independent of Mesos internals, register with the master to receive resource offers—proposals of available CPU, , and other from cluster nodes—and decide how to allocate them for launching and managing tasks. This approach allows frameworks to handle task execution via executors, which run on agent nodes to supervise and report on job progress, enabling efficient resource utilization across heterogeneous workloads. The model inherently supports long-running services, such as continuously operating applications, through persistent task , as well as batch jobs that execute finite computations, by leveraging either custom s for complex logic or the built-in command for simple shell commands and launches. Frameworks in the Mesos fall into key categories tailored to specific needs: orchestration, as seen in Marathon for deploying and scaling Docker-based services; batch scheduling, exemplified by for cron-like job orchestration with dependency graphs; and application-specific designs like Aurora, optimized for fault-tolerant, Twitter-scale management. Mesos integrates natively with big data processing tools via dedicated modes, allowing to operate in coarse-grained or fine-grained scheduling for distributed analytics, Hadoop to distribute tasks across the cluster, and Kafka to manage scalable message brokers as a framework for pipelines. This enables seamless among these tools without dedicated , leveraging Mesos' offer-based allocation for elastic scaling. Extensibility is a core strength, provided by the Mesos SDKs in C++, , and Python, which abstract scheduler and executor APIs to simplify custom framework development for diverse applications, including for stream and batch processing and for real-time data computation. Complementing these, ecosystem tools like Mesos-DNS facilitate by dynamically mapping framework tasks to DNS-resolvable hostnames and IP addresses, while integration via the Mesos exporter collects metrics on masters, agents, and tasks for . Overall, this framework ecosystem allows Mesos to unify diverse workloads—avoiding fragmented resource pools—though it demands framework-specific configurations for tuning isolation, fault tolerance, and performance.

Apache Aurora

Apache Aurora is a Mesos framework designed for scheduling long-running services, providing fault-tolerant management of applications across shared clusters. Originally developed internally at starting in 2010 by engineer Bill Farner as a simplified alternative to their proprietary scheduler—inspired by Google's Borg system—it was open-sourced in late 2013 and entered the Apache Incubator the same year. By February 2015, it had reached version 0.7.0, incorporating features like Docker integration and an improved command-line client, and it became a top-level Apache project thereafter. Key features of Aurora include declarative job definitions written in Python using a domain-specific language (DSL) in .aurora configuration files, which specify tasks, processes, and resources. These configurations support sophisticated service management, such as rolling updates with automatic health-based rollback, resource quotas for multi-user environments, and integration with for . Aurora also enables canary deployments to test updates on a subset of instances, autoscaling through dynamic task rescheduling on healthy nodes, and built-in health checks to monitor and maintain service availability. Additionally, it handles cron jobs for periodic execution and ad-hoc one-off tasks alongside persistent services. In its architecture, the Aurora scheduler operates as a Mesos framework, receiving resource offers from the Mesos master and launching tasks accordingly to ensure efficient allocation across the cluster. It leverages Mesos' resource isolation mechanisms, such as for CPU, memory, and disk limits, to enforce constraints on tasks. For task execution and intra-task process orchestration, Aurora uses Thermos, an execution engine that acts as a constraint solver to manage dependencies, ordering, and lifecycle events like on-success or on-failure hooks within jobs. This setup allows precise placement decisions based on constraints, such as rack affinity or node attributes, while maintaining through via . At Twitter, Aurora managed thousands of services, powering over 95% of stateless applications including the ad-serving platform, by automating deployments across tens of thousands of machines and handling hundreds of daily updates with minimal human intervention. It supported cron jobs for scheduled data processing and one-off tasks for temporary workloads, improving cluster utilization and reducing operational costs through automated failure recovery. With its last release, version 0.22.0, occurring on December 12, 2019, the project was officially retired by Apache in February 2020 due to inactivity and moved to the Apache Attic in April 2021. Compared to generic Mesos schedulers, Aurora offered advantages in fine-grained control over replica counts, update strategies, and failure handling, tailored for large-scale, service-oriented environments like Twitter's, enabling resilient operations without extensive custom scripting.

Chronos

Chronos is a distributed and fault-tolerant job scheduler designed as a Mesos framework, enabling cron-like batch processing across clusters with resource-aware execution. Developed by Airbnb engineers, it addresses limitations of traditional cron by providing dependency management, retries, and scalability for complex workflows in distributed environments. Airbnb open-sourced Chronos in March 2013 to manage batch jobs on Mesos, integrating it for efficient resource allocation in data-intensive workflows. The project leverages Mesos to distribute tasks, allowing for fault-tolerant scheduling without single points of failure. Unlike native cron, which operates on individual machines, Chronos enables distributed execution across clusters, incorporating Mesos's resource isolation to prevent bottlenecks and ensure reliable job orchestration. Key features of include JSON-based job specifications that define commands, schedules, and resources; support for job dependencies to form chains or graphs; configurable retries and error handling for robustness; and parallelism through multiple concurrent tasks on Mesos agents. Scheduling uses ISO8601 notation for flexible, repeating intervals, akin to syntax but adapted for distributed use. These elements allow users to define complex batch jobs via a RESTful API and a web UI for monitoring and visualization. In its architecture, the scheduler registers as a Mesos framework and launches tasks on available agents, relying on Mesos for offers and task relaunch upon failures. Job state and history are persisted in a backend, such as for reporting and long-term storage, ensuring across restarts. This design supports integration with external systems like Hadoop via custom executors or wrappers, without requiring Mesos agents to host those dependencies directly. At , powered ETL pipelines for extracting data from diverse sources, transforming it through multi-step processes, and loading into storage like S3, alongside Hadoop job orchestration. It scaled to handle extensive batch workflows without central chokepoints, distributing execution across Mesos clusters for efficient data processing. Despite its strengths, lacks built-in mechanisms, requiring external tools for inter-job communication in dynamic environments. Development activity tapered off around 2018, with the last major release in , partly due to the broader decline in Mesos adoption. As of November 2025, following the retirement of Apache Mesos, remains inactive with no further development or releases.

Marathon

Marathon is a production-grade orchestration platform designed to run on top of Apache Mesos, enabling the deployment and management of long-running services at scale. Developed by as an open-source project starting in 2013, it was created by Tobias Knaup and Florian Leibert to address the need for a simple, RESTful interface for containerized applications on Mesos clusters. By 2015, Marathon had matured sufficiently for production integration with DC/OS, Mesosphere's distribution of Mesos, allowing reliable in enterprise environments. Key features of Marathon include application definitions specified in JSON format via its REST API, which simplifies starting, stopping, and scaling services without complex configuration files. It provides automatic horizontal scaling based on resource availability and demand, along with integrated health checks using HTTP, TCP, or command-based probes to ensure task reliability. Load balancing is facilitated through Marathon-LB, an extension that dynamically generates HAProxy configurations to distribute traffic across instances, supporting both internal and external routing. Additionally, Marathon enables zero-downtime deployments via rolling updates, where new versions replace instances incrementally, and blue-green strategies that switch traffic between environments for safer rollouts. Architecturally, Marathon functions as a Mesos framework scheduler, registering with the Mesos master to receive resource offers and launching tasks accordingly in a two-level scheduling process. It achieves through an active/passive cluster model using for and state persistence, ensuring without service interruption. Placement constraints, such as rack affinity or operator-specified rules, allow fine-grained control over where tasks run to optimize for latency or . Marathon natively supports Docker containers for portability and Mesos containers for lightweight isolation, with the ability to bind persistent volumes for stateful applications. In practice, Marathon excels in use cases like , where it deploys and scales interdependent services across distributed clusters, handling and recovery automatically. Across all installations worldwide, Marathon has managed applications on more than 100,000 nodes, with individual production deployments handling over 10,000 tasks, making it suitable for large-scale web applications and backends in data centers. The project evolved significantly with the release of version 1.0 in March 2016, which enhanced stability, consistency, and support for advanced deployment patterns to meet enterprise demands. Active development continued through contributions from and the community until 2021, when DC/OS integration support ended on October 31, 2021, amid Mesosphere's strategic pivot to new platforms under D2iQ. The repository was archived and made read-only in October 2024. As of November 2025, following the retirement of Apache Mesos, Marathon receives no further maintenance. Security in Marathon includes support for basic authentication and SSL/TLS encryption on its REST API to secure communications and access. It leverages Mesos' module with access control lists (ACLs) for , allowing operators to define permissions for principals like registering frameworks or launching tasks. Secrets management is handled through Mesos' built-in features, enabling tasks to securely retrieve sensitive data as environment variables or volumes without exposing them in plain text.

Adoption and Impact

Notable Users

Twitter pioneered the use of Apache Mesos in production, deploying the Aurora framework to manage web services and batch jobs across large-scale clusters. By 2016, 's Mesos clusters typically handled tens of thousands of tasks, enabling efficient resource sharing and fault-tolerant scheduling for its high-traffic platform. adopted Mesos to run frameworks like for orchestrating data pipelines, integrating with tools such as Hadoop, Storm, and to process petabytes of data daily. This setup supported 's complex needs, providing fault-tolerant scheduling as a replacement for traditional jobs. Verizon integrated Mesos, via Mesosphere DC/OS, as a nationwide platform for data center orchestration, powering media services like video streaming on its FiOS entertainment platform during its pre-2020 peak usage. This deployment accelerated product rollouts and supported scalable containerized applications for network services. eBay utilized Mesos to scale its (CI) infrastructure, running Jenkins farms in Docker containers to handle build workloads for e-commerce applications. This approach improved developer productivity and supported the dynamic demands of eBay's . Other prominent adopters included for orchestration, for content delivery scaling, and Apple for internal cluster , reflecting Mesos's broad appeal during its peak adoption period from 2017 to 2019. However, by 2024, many organizations, including and , migrated to due to its maturing ecosystem, richer tooling, and wider community support.

Commercial Offerings and Support

Mesosphere launched DC/OS in 2015 as its flagship commercial platform, bundling as the core resource manager with Marathon for container orchestration, Edge-LB for load balancing, and various administrative tools to simplify cluster management for enterprise deployments. By 2017, DC/OS had attracted more than 100 enterprise customers, including , , and Verizon, enabling them to run data-intensive workloads at scale across hybrid environments. In August 2019, Mesosphere rebranded to D2iQ to emphasize Day 2 operations in cloud-native ecosystems, pivoting its primary focus to Kubernetes-based solutions like Konvoy (later rebranded as DKP) while maintaining initial support for Mesos and DC/OS. However, D2iQ announced the sunset of DC/OS in 2020, with an end-of-life date of October 31, 2021, marking the cessation of official updates, patches, and commercial backing for Mesos-integrated products by early 2022. Commercial engagement extended beyond Mesosphere through ecosystem contributions, such as sponsorships of MesosCon conferences by companies including and others up to 2019, fostering community-driven adoption. Enterprise support models under and D2iQ encompassed paid subscriptions for advanced features, security patches, and dedicated consulting services to assist with deployment and optimization. Complementing these were community resources, including discussions and on platforms like , which remained active until the project's full retirement. With Mesos entering retirement in August 2025 and moving to the Apache Attic by October 2025, no official commercial support is available, leaving users to either the —such as the ongoing Clusterd project—or migrate to successor technologies like . Prior to its strategic pivot, Mesosphere had raised approximately $251 million in venture funding across multiple rounds to fuel its growth in distributed systems.

Legacy

Influence on Distributed Computing

Apache Mesos pioneered the concept of resource unification in distributed systems by treating datacenter resources as a shared pool managed like an operating system kernel, enabling efficient multi-tenancy across diverse workloads. This approach abstracted CPU, memory, disk, and network resources into a unified layer, allowing multiple frameworks to share clusters without silos, a departure from traditional siloed deployments. The original Mesos design emphasized fine-grained sharing, where resources are offered dynamically to frameworks via a resource offer model, achieving utilization improvements of 10% for CPU and 18% for memory over static partitioning, along with 95% data locality for Hadoop jobs in benchmarks. This innovation influenced subsequent systems by establishing a blueprint for treating the datacenter as a single programmable entity, fostering multi-tenant architectures in cloud-native computing. In the realm of processing, Mesos significantly impacted frameworks like and Hadoop by enabling shared clusters that reduced resource fragmentation and improved efficiency. Spark, for instance, leveraged Mesos for fine-grained , allowing jobs to dynamically utilize idle resources across the cluster, which enhanced performance for iterative algorithms common in . Similarly, Hadoop's evolution toward incorporated elements of Mesos' resource negotiation, shifting from a monolithic scheduler to a more flexible two-level model that supports diverse applications beyond . This sharing capability addressed key pain points in ecosystems, such as underutilized hardware in siloed deployments, and inspired 's development to handle multi-framework workloads more effectively. Mesos' two-level scheduling model, where a central allocator offers resources to framework-specific schedulers, became a foundational adopted in hybrid schedulers and influenced tools like and federation mechanisms. In this architecture, the first level handles coarse-grained allocation across the cluster, while the second level allows frameworks to optimize task placement based on application-specific needs, enabling scalability and flexibility for heterogeneous workloads. drew from this model to support diverse job types, including batch and service-oriented tasks, by implementing a similar hierarchical . federation, which coordinates multiple clusters, echoed Mesos' decentralized decision-making to manage resources across distributed environments, though with a focus on container . This design has been credited as the first practical two-level scheduler, shaping modern hybrid approaches that balance global efficiency with local . The Mesos community played a pivotal role in standardizing practices through events like MesosCon, held annually from 2014 to 2018, which brought together developers and users to collaborate on ecosystem growth and best practices. These conferences, starting with the inaugural event in co-located with LinuxCon, facilitated discussions on integrations, scalability, and extensions, ultimately strengthening the open-source ecosystem around cluster management. Mesos also contributed to container standards via support for the App Container (AppC) specification, enabling interoperability with tools like rkt and influencing the broader shift toward portable container runtimes in the . Mesosphere, a key contributor, joined as a founding member of the OCI in 2015, helping consolidate standards for image formats and runtimes that Mesos natively supported. Mesos demonstrated remarkable , supporting clusters of over 10,000 nodes in production environments, as evidenced by deployments at companies like and , where it managed thousands of tasks with low latency. The original paper has garnered over 5,000 academic citations, underscoring its enduring influence on distributed systems research. However, Mesos faced criticisms for its setup complexity, including the need for coordination and framework-specific configurations, which contributed to slower adoption compared to more streamlined alternatives like . This operational overhead, while offering flexibility, required significant expertise for reliable deployment and maintenance in diverse settings.

Alternatives and Successors

Kubernetes has emerged as the primary successor to Apache Mesos in container orchestration, offering a single-scheduler model that simplifies compared to Mesos' two-level . Many organizations migrated from Mesos to between 2018 and 2023, often employing pattern to incrementally replace legacy components while maintaining operational continuity. For instance, completed a full migration of its stateless container orchestration platform from Mesos to in 2024, transitioning across multiple data centers to leverage ' ecosystem and scalability. Other notable migrations include those by Adevinta in 2020 and mPharma in 2023, highlighting ' dominance in modern distributed systems. Alternative orchestration tools have also served as viable replacements for Mesos, particularly for specific workloads. HashiCorp Nomad provides multi-workload scheduling capabilities, supporting containers, virtual machines, and non-containerized applications with a simpler, single-binary deployment model that contrasts with Mesos' complexity. Apache YARN remains a strong option for Hadoop-centric environments, focusing on for processing through its application-level scheduler, which offers fine-grained control over and Spark jobs without Mesos' broader abstraction layer. Docker Swarm emphasizes simplicity for container , enabling easy cluster management with built-in and load balancing, making it suitable for smaller-scale deployments where Mesos' overhead is unnecessary. Following Mesos' retirement to the Apache Attic in August 2025, community forks have emerged to preserve its core functionality for niche applications. Clusterd, a 2025 GitHub fork maintained by Andreas Peters, continues development of Mesos' resource isolation and sharing mechanisms, though it exhibits limited activity and seeks additional contributors for ongoing support. Migration guidance from the Apache community encourages transitioning to or , with practical tools and strategies derived from real-world case studies, such as Uber's playbook for large-scale shifts, facilitating the conversion of Mesos frameworks to Kubernetes manifests. As of 2025, Mesos persists in legacy hybrid setups but holds less than 1% in container , dwarfed by ' over 90% dominance. Future prospects for Mesos may involve revival through forks like Clusterd, particularly if emerging AI workloads require its fine-grained resource sharing in specialized scenarios.

References

  1. https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf
  2. https://mesos.apache.org/
  3. https://news.apache.org/foundation/entry/the_apache_software_foundation_announces45
  4. https://news.apache.org/foundation/entry/the_apache_software_foundation_announces97
  5. https://mesos.apache.org/documentation/latest/architecture/
  6. https://attic.apache.org/projects/mesos.html
  7. https://www.usenix.org/conference/nsdi11/mesos-platform-fine-grained-resource-sharing-data-center
  8. https://blog.x.com/engineering/en_us/a/2012/incubating-apache-mesos
  9. https://mesos.apache.org/blog/mesos-0-20-0-released/
  10. https://spark.apache.org/releases/spark-release-0-5-0.html
  11. https://mesos.apache.org/blog/mesos-1-0-0-released/
  12. https://mesos.apache.org/blog/mesos-1-4-0-released/
  13. https://mesos.apache.org/documentation/latest/gpu-support/
  14. https://mesos.apache.org/blog/
  15. https://d2iq.com/blog/making-apache-kafka-elastic-with-apache-mesos
  16. https://mesos.apache.org/blog/dev-community-status/
  17. https://whimsy.apache.org/board/minutes/Mesos.html
  18. https://mvnrepository.com/artifact/org.apache.mesos/mesos/1.11.0
  19. https://lists.apache.org/thread/d0cob6pp8xkw06qyckzv52nc20n0p7wt
  20. https://news.ycombinator.com/item?id=26713082
  21. https://www.techtarget.com/searchitoperations/news/252468124/Mesospheres-Kubernetes-pivot-sets-stage-for-IT-vendor-attrition
  22. https://d2iq.com/blog/apache-mesos-deep-dive
  23. https://www.datacenterknowledge.com/business/mesosphere-becomes-d2iq-moves-into-kubernetes-big-data
  24. https://lists.apache.org/thread/vr36mqspo636tht9jrfrm8zpfstotzx9
  25. https://www.mail-archive.com/[email protected]/msg10402.html
  26. https://attic.apache.org/projects.html
  27. https://github.com/m3scluster/clusterd
  28. https://mesos.apache.org/documentation/latest/high-availability/
  29. https://mesos.apache.org/documentation/latest/mesos-containerizer/
  30. https://mesos.apache.org/documentation/latest/endpoints/
  31. https://mesos.apache.org/documentation/latest/windows/
  32. https://mesos.apache.org/documentation/latest/building/
  33. https://mesos.apache.org/documentation/latest/attributes-resources/
  34. https://mesos.apache.org/documentation/latest/running-workloads/
  35. https://mesos.apache.org/documentation/latest/roles/
  36. https://mesos.apache.org/documentation/latest/reservation/
  37. https://mesos.apache.org/documentation/latest/app-framework-development-guide/
  38. https://mesos.apache.org/documentation/latest/frameworks/
  39. https://github.com/mesos/hadoop
  40. https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/mesos/
  41. https://github.com/mesos/storm
  42. https://github.com/mesos/mesos_exporter
  43. https://blog.x.com/engineering/en_us/a/2015/all-about-apache-aurora
  44. https://aurora.apache.org/documentation/latest/reference/configuration-tutorial/
  45. https://aurora.apache.org/documentation/latest/features/service-discovery/
  46. https://aurora.apache.org/documentation/latest/features/resource-isolation/
  47. https://aurora.apache.org/documentation/latest/reference/configuration/
  48. https://issues.apache.org/jira/browse/AURORA-167
  49. https://blog.x.com/engineering/en_us/a/2016/overview-of-the-twitter-cloud-platform-compute
  50. https://aurora.apache.org/documentation/latest/features/cron-jobs/
  51. https://aurora.apache.org/blog/
  52. https://attic.apache.org/projects/aurora.html
  53. https://s.apache.org/d7emy
  54. https://medium.com/airbnb-engineering/chronos-a-replacement-for-cron-f05d7d986a9d
  55. https://github.com/mesos/chronos
  56. https://mesos.github.io/chronos/
  57. https://mesos.github.io/chronos/docs/api.html
  58. https://d2iq.com/blog/cassandra-on-mesos-performance-reliability-mesosphere-hackweek
  59. https://github.com/mesos/chronos/releases
  60. https://github.com/d2iq-archive/marathon
  61. https://d2iq.com/blog/marathon-0-6-0-released
  62. https://d2iq.com/blog/marathon-production-ready-containers
  63. https://archive-docs-old.d2iq.com/mesosphere/dcos/services/marathon-lb/1.14/mlb-advanced-tutorial
  64. https://aventer-ug.github.io/marathon/index.html
  65. https://www.baeldung.com/ops/mesos-kubernetes-comparison
  66. https://mesos.apache.org/documentation/latest/authorization/
  67. https://mesos.apache.org/documentation/latest/secrets/
  68. https://archive.fosdem.org/2016/schedule/event/twitter/
  69. http://nerds.airbnb.com/introducing-chronos/
  70. https://www.prnewswire.com/news-releases/verizon-selects-mesosphere-dcos-as-nationwide-platform-for-data-center-service-orchestration-300131523.html
  71. https://www.nextplatform.com/2015/09/02/verizon-satisfies-google-envy-with-mesos/
  72. https://www.youtube.com/watch?v=ncuIj9_EL7o
  73. https://innovation.ebayinc.com/stories/delivering-ebays-ci-solution-with-apache-mesos-part-i/
  74. https://innovation.ebayinc.com/stories/delivering-ebays-ci-solution-with-apache-mesos-part-ii/
  75. https://thenewstack.io/apache-mesos-narrowly-avoids-a-move-to-the-attic-for-now/
  76. https://www.forrester.com/blogs/kubernetes-k8s-from-orchestration-war-to-platform-showdown-to-os-standard/
  77. https://www.uber.com/blog/migrating-ubers-compute-platform-to-kubernetes-a-technical-journey/
  78. https://www.alibabacloud.com/blog/twitter-announced-switch-from-mesos-to-kubernetes_595156
  79. https://d2iq.com/blog/recapping-an-amazing-year-for-mesosphere
  80. https://siliconangle.com/2015/06/10/mesospheres-data-center-os-launches-in-general-availability/
  81. https://www.prnewswire.com/news-releases/at-one-year-anniversary-mesosphere-dcos-hits-major-milestones-300440277.html
  82. https://d2iq.com/blog/mesosphere-is-now-d2iq
  83. https://d2iq.com/products/dcos
  84. https://stackoverflow.com/questions/44171100/whats-difference-between-apache-mesos-mesosphere-and-dcos
  85. https://stackoverflow.com/questions/tagged/mesos
  86. https://www.datacenterknowledge.com/business/-125-million-richer-mesosphere-tackles-big-data-at-the-edge
  87. https://amplab.cs.berkeley.edu/wp-content/uploads/2011/06/Mesos-A-Platform-for-Fine-Grained-Resource-Sharing-in-the-Data-Center.pdf
  88. https://www.databricks.com/blog/2014/01/21/spark-and-hadoop.html
  89. https://www.oreilly.com/content/a-tale-of-two-clusters-mesos-and-yarn/
  90. https://www.cse.ust.hk/~weiwa/teaching/Fall16-COMP6611B/reading_list/YARN.pdf
  91. https://arxiv.org/pdf/2511.01860
  92. https://globaljournals.org/GJCST_Volume19/4-A-Taxonomy-of-Schedulers-Operating.pdf
  93. https://issues.apache.org/jira/browse/MESOS-1166
  94. https://mesos.apache.org/blog/mesoscon-2014-program-announced/
  95. https://mesos.apache.org/documentation/latest/container-image/
  96. https://opencontainers.org/posts/announcements/2015-06-20-industry-leaders-unite-to-create-project-for-open-container-standard/
  97. https://www.quora.com/In-the-near-future-would-you-rather-use-Apache-Mesos-or-Kubernetes-for-container-oriented-big-data-stacks-Why-so
Add your contribution
Related Hubs
User Avatar
No comments yet.