Hubbry Logo
Apache SparkApache SparkMain
Open search
Apache Spark
Community hub
Apache Spark
logo
8 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Apache Spark
Apache Spark
Apache Spark
View on Wikipedia
from Wikipedia
Apache Spark
Original authorMatei Zaharia
DeveloperApache Spark
Initial releaseMay 26, 2014; 11 years ago (2014-05-26)
Stable release
4.0.1 (Scala 2.13) / September 6, 2025; 5 months ago (2025-09-06)
Written inScala[1]
Operating systemWindows, macOS, Linux
Available inScala, Java, SQL, Python, R, C#, F#
TypeData analytics, machine learning algorithms
LicenseApache License 2.0
Websitespark.apache.org Edit this at Wikidata
RepositorySpark Repository

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of California, Berkeley's AMPLab starting in 2009, in 2013, the Spark codebase was donated to the Apache Software Foundation, which has maintained it since.

Overview

[edit]

Apache Spark has its architectural foundation in the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.[2] The Dataframe API was released as an abstraction on top of the RDD, followed by the Dataset API. In Spark 1.x, the RDD was the primary application programming interface (API), but as of Spark 2.x use of the Dataset API is encouraged[3] even though the RDD API is not deprecated.[4][5] The RDD technology still underlies the Dataset API.[6][7]

Spark and its RDDs were developed in 2012 in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.[8]

Inside Apache Spark the workflow is managed as a directed acyclic graph (DAG). Nodes represent RDDs while edges represent the operations on the RDDs.

Spark facilitates the implementation of both iterative algorithms, which visit their data set multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications may be reduced by several orders of magnitude compared to Apache Hadoop MapReduce implementation.[2][9] Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark.[10]

Apache Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone native Spark, Hadoop YARN, Apache Mesos or Kubernetes.[11] A standalone native Spark cluster can be launched manually or by the launch scripts provided by the install package. It is also possible to run the daemons on a single machine for testing. For distributed storage Spark can interface with a wide variety of distributed systems, including Alluxio, Hadoop Distributed File System (HDFS),[12] MapR File System (MapR-FS),[13] Cassandra,[14] OpenStack Swift, Amazon S3, Kudu, Lustre file system,[15] or a custom solution can be implemented. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in such a scenario, Spark is run on a single machine with one executor per CPU core.

Spark Core

[edit]

Spark Core is the foundation of the overall project. It provides distributed task dispatching, scheduling, and basic I/O functionalities, exposed through an application programming interface (for Java, Python, Scala, .NET[16] and R) centered on the RDD abstraction (the Java API is available for other JVM languages, but is also usable for some other non-JVM languages that can connect to the JVM, such as Julia[17]). This interface mirrors a functional/higher-order model of programming: a "driver" program invokes parallel operations such as map, filter or reduce on an RDD by passing a function to Spark, which then schedules the function's execution in parallel on the cluster.[2] These operations, and additional ones such as joins, take RDDs as input and produce new RDDs. RDDs are immutable and their operations are lazy; fault-tolerance is achieved by keeping track of the "lineage" of each RDD (the sequence of operations that produced it) so that it can be reconstructed in the case of data loss. RDDs can contain any type of Python, .NET, Java, or Scala objects.

Besides the RDD-oriented functional style of programming, Spark provides two restricted forms of shared variables: broadcast variables reference read-only data that needs to be available on all nodes, while accumulators can be used to program reductions in an imperative style.[2]

A typical example of RDD-centric functional programming is the following Scala program that computes the frequencies of all words occurring in a set of text files and prints the most common ones. Each map, flatMap (a variant of map) and reduceByKey takes an anonymous function that performs a simple operation on a single data item (or a pair of items), and applies its argument to transform an RDD into a new RDD. 

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.rdd.RDD

val conf: SparkConf = new SparkConf().setAppName("wiki_test") // create a spark config object
val sc: SparkContext = new SparkContext(conf) // Create a spark context
val data: RDD[String] = sc.textFile("/path/to/somedir") // Read files from "somedir" into an RDD of (filename, content) pairs.
val tokens: RDD[String] = data.flatMap(_.split(" ")) // Split each file into a list of tokens (words).
val wordFreq: RDD[(String, Int)] = tokens.map((_, 1)).reduceByKey(_ + _) // Add a count of one to each token, then sum the counts per word type.
val topWords: Array[(Int, String)] = wordFreq.sortBy(s => -s._2).map(x => (x._2, x._1)).top(10) // Get the top 10 words. Swap word and count to sort by count.

Spark SQL

[edit]

Spark SQL is a component on top of Spark Core that introduced a data abstraction called DataFrames,[a] which provides support for structured and semi-structured data. Spark SQL provides a domain-specific language (DSL) to manipulate DataFrames in Scala, Java, Python or .NET.[16] It also provides SQL language support, with command-line interfaces and ODBC/JDBC server. Although DataFrames lack the compile-time type-checking afforded by RDDs, as of Spark 2.0, the strongly typed DataSet is fully supported by Spark SQL as well. 

import org.apache.spark.sql.{DataFrame, SparkSession}

val url: String = "jdbc:mysql://yourIP:yourPort/test?user=yourUsername;password=yourPassword" // URL for your database server.
val spark: SparkSession = SparkSession.builder().getOrCreate() // Create a Spark session object

val df: DataFrame = spark
  .read
  .format("jdbc")
  .option("url", url)
  .option("dbtable", "people")
  .load()

df.printSchema() // Looks at the schema of this DataFrame.
val countsByAge: DataFrame = df.groupBy("age").count() // Counts people by age

Or alternatively via SQL: 

df.createOrReplaceTempView("people")
val countsByAge: DataFrame = spark.sql("SELECT age, count(*) FROM people GROUP BY age")

Spark Streaming

[edit]

Spark Streaming uses Spark Core's fast scheduling capability to perform streaming analytics. It ingests data in mini-batches and performs RDD transformations on those mini-batches of data. This design enables the same set of application code written for batch analytics to be used in streaming analytics, thus facilitating easy implementation of lambda architecture.[19][20] However, this convenience comes with the penalty of latency equal to the mini-batch duration. Other streaming data engines that process event by event rather than in mini-batches include Storm and the streaming component of Flink.[21] Spark Streaming has support built-in to consume from Kafka, Flume, Twitter, ZeroMQ, Kinesis, and TCP/IP sockets.[22]

In Spark 2.x, a separate technology based on Datasets, called Structured Streaming, that has a higher-level interface is also provided to support streaming.[23]

Spark can be deployed in a traditional on-premises data center as well as in the cloud.[24]

MLlib machine learning library

[edit]

Spark MLlib is a distributed machine-learning framework on top of Spark Core that, due in large part to the distributed memory-based Spark architecture, is as much as nine times as fast as the disk-based implementation used by Apache Mahout (according to benchmarks done by the MLlib developers against the alternating least squares (ALS) implementations, and before Mahout itself gained a Spark interface), and scales better than Vowpal Wabbit.[25] Many common machine learning and statistical algorithms have been implemented and are shipped with MLlib which simplifies large scale machine learning pipelines, including:

GraphX

[edit]

GraphX is a distributed graph-processing framework on top of Apache Spark. Because it is based on RDDs, which are immutable, graphs are immutable and thus GraphX is unsuitable for graphs that need to be updated, let alone in a transactional manner like a graph database.[27] GraphX provides two separate APIs for implementation of massively parallel algorithms (such as PageRank): a Pregel abstraction, and a more general MapReduce-style API.[28] Unlike its predecessor Bagel, which was formally deprecated in Spark 1.6, GraphX has full support for property graphs (graphs where properties can be attached to edges and vertices).[29]

Like Apache Spark, GraphX initially started as a research project at UC Berkeley's AMPLab and Databricks, and was later donated to the Apache Software Foundation and the Spark project.[30]

Language support

[edit]

Apache Spark has built-in support for Scala, Java, SQL, R, Python, and Swift with 3rd party support for the .NET CLR,[31] Julia,[32] and more.

History

[edit]

Spark was initially started by Matei Zaharia at UC Berkeley's AMPLab in 2009, and open sourced in 2010 under a BSD license.[33]

In 2013, the project was donated to the Apache Software Foundation and switched its license to Apache 2.0. In February 2014, Spark became a Top-Level Apache Project.[34]

In November 2014, Spark founder M. Zaharia's company Databricks set a new world record in large scale sorting using Spark.[35][33]

Spark had in excess of 1000 contributors in 2015,[36] making it one of the most active projects in the Apache Software Foundation[37] and one of the most active open source big data projects.

Version Original release date Latest version Release date
Unsupported: 0.5 2012-06-12 0.5.2 2012-11-22
Unsupported: 0.6 2012-10-15 0.6.2 2013-02-07
Unsupported: 0.7 2013-02-27 0.7.3 2013-07-16
Unsupported: 0.8 2013-09-25 0.8.1 2013-12-19
Unsupported: 0.9 2014-02-02 0.9.2 2014-07-23
Unsupported: 1.0 2014-05-26 1.0.2 2014-08-05
Unsupported: 1.1 2014-09-11 1.1.1 2014-11-26
Unsupported: 1.2 2014-12-18 1.2.2 2015-04-17
Unsupported: 1.3 2015-03-13 1.3.1 2015-04-17
Unsupported: 1.4 2015-06-11 1.4.1 2015-07-15
Unsupported: 1.5 2015-09-09 1.5.2 2015-11-09
Unsupported: 1.6 2016-01-04 1.6.3 2016-11-07
Unsupported: 2.0 2016-07-26 2.0.2 2016-11-14
Unsupported: 2.1 2016-12-28 2.1.3 2018-06-26
Unsupported: 2.2 2017-07-11 2.2.3 2019-01-11
Unsupported: 2.3 2018-02-28 2.3.4 2019-09-09
Unsupported: 2.4 LTS 2018-11-02 2.4.8 2021-05-17[38]
Unsupported: 3.0 2020-06-18 3.0.3 2021-06-01[39]
Unsupported: 3.1 2021-03-02 3.1.3 2022-02-18[40]
Unsupported: 3.2 2021-10-13 3.2.4 2023-04-13[41]
Unsupported: 3.3 2022-06-16 3.3.3 2023-08-21[42]
Unsupported: 3.4 2023-04-13 3.4.4 2024-10-27[43]
Supported: 3.5 LTS 2023-09-09 3.5.8 2026-01-15[44]
Latest version: 4.0 2025-05-23 4.0.1 2025-09-06[45]
Latest version: 4.1 2025-12-16 4.1.1 2026-01-09[46]
Future version: 4.2 2026-01-11 4.2.0-preview1 2026-01-11[47]
Legend:
Unsupported
Supported
Latest version
Preview version

Scala version

[edit]

Spark 3.5.2 is based on Scala 2.13 (and thus works with Scala 2.12 and 2.13 out-of-the-box), but it can also be made to work with Scala 3.[48]

Developers

[edit]

Apache Spark is developed by a community. The project is managed by a group called the "Project Management Committee" (PMC).[49]

Maintenance releases and EOL

[edit]

Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months. For example, branch 2.3.x is no longer considered maintained as of September 2019, 18 months after the release of 2.3.0 in February 2018. No more 2.3.x releases should be expected after that point, even for bug fixes.

The last minor release within a major a release will typically be maintained for longer as an “LTS” release. For example, 2.4.0 was released on November 2, 2018, and had been maintained for 31 months until 2.4.8 was released in May 2021. 2.4.8 is the last release and no more 2.4.x releases should be expected even for bug fixes.[50]

See also

[edit]

Notes

[edit]

References

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Apache Spark

View on Grokipedia
from Grokipedia
Apache Spark is an open-source, unified analytics engine designed for large-scale , enabling efficient execution of , , and workloads on single-node machines or distributed clusters. It originated as a research project at the , Berkeley's AMPLab in 2009, led by during his PhD program, and was open-sourced in early 2010 to address limitations in existing big data frameworks like Hadoop MapReduce by introducing in-memory computing for faster iterative algorithms. In 2013, the project joined as an incubating project and became a top-level Apache project in February 2014, fostering a global community of over 2,000 contributors from industry and academia. At its core, Spark revolves around resilient distributed datasets (RDDs), a fault-tolerant for in-memory cluster that supports parallel operations and fault recovery without data replication across the entire dataset. Key features include high-level APIs in multiple languages—, Scala, and Python (via PySpark)—along with built-in libraries such as Spark SQL for structured data querying, Structured Streaming for processing, MLlib for scalable , and GraphX for graph . Its optimized engine leverages in-memory caching and adaptive query execution to achieve up to 100 times faster performance than disk-based systems for certain workloads, making it suitable for batch processing, interactive queries, and streaming applications. Today, Apache Spark powers for thousands of organizations, including 80% of the Fortune 500 companies, and integrates seamlessly with various storage systems and cloud platforms.

Introduction

Definition and Purpose

Apache Spark is an open-source, framework that provides a unified engine for large-scale , leveraging in-memory to handle vast datasets efficiently across clusters. Developed under , it supports high-level APIs in multiple languages including Scala, , Python, and , allowing developers to perform complex data operations without managing low-level distributed systems details. The primary purposes of Apache Spark include enabling for large static datasets, real-time streaming for continuous data flows, workflows through its MLlib library, and graph via GraphX, all integrated into a single cohesive platform that reduces the need for multiple specialized tools. This unified approach facilitates seamless transitions between different data processing paradigms, supporting data engineering, science, and tasks at scale. In comparison to predecessors like Hadoop , which rely on disk-based processing, Spark delivers up to 100 times faster performance in memory and 10 times faster on disk by employing (DAG) execution for optimized task scheduling and to minimize unnecessary computations. At the heart of Spark's design lies the Resilient Distributed Dataset (RDD), a fault-tolerant, immutable distributed collection that serves as the core abstraction for parallel data operations, ensuring reliability without full data replication. Spark's versatility makes it suitable for high-level use cases such as extract-transform-load (ETL) operations to prepare data pipelines, interactive queries for ad-hoc analysis, and real-time analytics to derive insights from streaming sources, powering applications in industries like finance, healthcare, and e-commerce.

Key Features and Benefits

Apache Spark's in-memory computation capability allows data to be cached directly in RAM across cluster nodes, significantly reducing disk I/O overhead and accelerating iterative algorithms such as training. This approach enables Spark to achieve up to 100 times faster performance compared to disk-based systems like Hadoop for multi-pass applications. By keeping intermediate results in memory, Spark minimizes data movement, making it particularly effective for workloads requiring repeated data access. As a unified analytics engine, Spark provides a single framework for diverse tasks, including , real-time streaming, , and graph , thereby reducing the need for multiple specialized tools and simplifying development workflows. This integration fosters developer productivity through high-level APIs that abstract low-level distributed systems complexities, allowing code written on a single machine to scale seamlessly to large clusters. Spark ensures through its Resilient Distributed Datasets (RDDs), which track to automatically recompute lost partitions upon node failures without requiring manual checkpoints or data replication. This mechanism supports reliable operation across distributed environments, enhancing system resilience for large-scale deployments. Spark's scalability extends to petabyte-scale datasets distributed over thousands of nodes, leveraging efficient execution graphs for high-throughput processing. Key benefits include substantial cost savings from accelerated processing times, with in-memory operations often yielding orders-of-magnitude improvements in iterative tasks, and improved developer efficiency via intuitive APIs in languages like Scala, Python, , and . Additionally, Spark's interoperability allows it to run on various cluster managers such as Hadoop YARN, , and standalone modes, as well as integrate with storage systems like HDFS.

Architecture

Spark Core and RDDs

Spark Core serves as the foundational engine of Apache Spark, providing distributed task dispatching, scheduling, and basic input/output functionalities that underpin all higher-level components. It manages the execution of tasks across a cluster, handling in coordination with external cluster managers, and supports in-memory computation to accelerate . This core layer enables Spark's unified analytics capabilities by abstracting the complexities of distributed computing. At the heart of Spark Core is the Resilient Distributed Dataset (RDD), an immutable, partitioned collection of objects that can be processed in parallel across nodes in a cluster. RDDs provide a fault-tolerant for in-memory cluster , allowing users to explicitly cache data in and control its partitioning to optimize performance. They can be created from external data sources like Hadoop files or by transforming other RDDs, ensuring data locality and parallelism. Examples include parallelizing a local collection into an RDD or loading data from HDFS. RDDs support two primary types of operations: transformations and actions. Transformations, such as map, filter, and join, create a new RDD from an existing one without immediately computing the result, enabling composition of complex pipelines. Actions, like count, collect, and reduce, trigger the execution of the computation and return a value or write data to an output system. These operations allow developers to express data processing workflows in a functional style. The execution model of Spark Core relies on , where transformations build a lineage graph without performing computations until an action is invoked. This lineage is represented as a (DAG) of dependencies between RDDs, which tracks the sequence of transformations applied. Upon an action, the DAG scheduler divides the graph into stages of tasks, optimizing for pipelining narrow dependencies while handling wide dependencies separately. This approach minimizes unnecessary work and supports iterative algorithms efficiently. RDD lineage enables by allowing lost partitions to be recomputed from the original data using the recorded transformations, without requiring data replication. In case of node , Spark uses the DAG to regenerate only the affected partitions, ensuring reliability in large-scale distributed environments. This recomputation mechanism contrasts with traditional checkpointing, offering low-overhead recovery for in-memory data. Data in RDDs is distributed across the cluster via partitioning, where each RDD consists of multiple partitions stored on different nodes to enable parallelism. The number of partitions can be controlled during creation or via operations like repartition, influencing the of parallel tasks. occurs during wide dependencies, such as groupByKey or reduceByKey, where data is redistributed across partitions based on keys, often involving network I/O and disk spills if is insufficient; this is one of the most costly operations in Spark. To mitigate shuffle overhead, users can tune partition sizes and use combiners for aggregations. Spark Core's memory management is enhanced by the project, which optimizes through columnar in-memory layouts and whole-stage code generation for CPU efficiency. Tungsten also supports off-heap memory allocation to store data outside the JVM heap, reducing garbage collection pauses and enabling larger working sets. These features, including efficient binary and cache-aware computation, significantly improve performance for memory-intensive workloads.

Cluster Managers and Deployment

Apache Spark relies on external cluster managers to distribute workloads across a cluster of machines, handling resource allocation, task scheduling, and fault recovery independently of Spark's core engine. The cluster manager allocates resources such as CPU, memory, and optionally GPUs to Spark applications by launching executor processes on worker nodes. These executors perform computations and store data in memory or disk, while the manager ensures fault tolerance through mechanisms like restarting failed executors or reallocating tasks upon node failures. This separation allows Spark to integrate with various resource management frameworks without modifying its internal execution model. Spark supports several cluster managers, each suited to different environments. The built-in Standalone mode provides a simple, self-contained option for deployments without external dependencies, using a master-worker where the master tracks resources and workers host executors. Installation involves placing Spark binaries on cluster nodes and starting the master daemon, which then accepts worker registrations; it supports dynamic resource allocation and fault recovery by relaunching executors on healthy nodes. For Hadoop ecosystems, Spark integrates with (Yet Another Resource Negotiator), leveraging YARN's resource management to schedule Spark applications alongside other workloads; this integration, introduced in Spark 0.6.0, allows Spark to request containers for executors and handles fault tolerance via YARN's application master process. Historically, Spark supported Apache Mesos for fine-grained or coarse-grained resource sharing, where coarse-grained mode (default) allocates entire executors statically and fine-grained mode (deprecated) shares resources at the task level for more dynamic allocation; however, Mesos support was deprecated in Spark 3.2.0 and fully removed in Spark 4.0.0 to streamline maintenance. Since Spark 2.3.0, Spark offers native integration with Kubernetes for containerized deployments, using Kubernetes' scheduler to manage pods for drivers and executors; this enables dynamic resource allocation, scaling, and portability across cloud-native environments, with production readiness achieved in Spark 3.1.0. Spark applications can be deployed in client mode or cluster mode, determining the location of the driver program—the process that creates the SparkContext and coordinates execution. In client mode, the driver runs on the machine submitting the application (e.g., a developer's ), suitable for interactive sessions where output is needed locally, while executors run on cluster nodes. In cluster mode, the driver runs inside the cluster (e.g., as a application master or pod), allowing the client to disconnect after submission; this mode is ideal for production jobs to avoid driver failures due to client connectivity issues. Security features, such as Kerberos authentication, are supported across managers to secure access to protected resources like HDFS; configuration involves setting properties like spark.kerberos.renewal.credentials to use ticket caches for authentication without exposing credentials. Monitoring Spark applications occurs primarily through the built-in Web UI, accessible at http://<driver-node>:4040 during execution, which provides tabs for jobs (showing scheduler stages and task progress), stages (detailing task metrics like duration and shuffle data), executors (listing active instances with and disk usage), storage (RDD persistence details), and environment (configuration and dependencies). For completed applications, the Spark History Server (port 18080) aggregates logs if event is enabled via spark.eventLog.enabled=true. Integration with cluster managers extends monitoring; for example, provides executor logs via its ResourceManager UI, enhancing visibility into resource utilization and failures.

Data Processing Modules

Spark SQL and DataFrames

Spark SQL is a module within Apache Spark designed for processing structured data, enabling users to query data using SQL or the DataFrame API while leveraging Spark's distributed execution engine. It supports HiveQL syntax and provides seamless integration for structured data operations, allowing queries on data from various sources such as files, databases, and existing RDDs. Unlike the low-level RDD API, Spark SQL exposes more information about and computations to enable advanced optimizations. DataFrames represent distributed collections of data organized into named columns, akin to relational database tables or data frames in and Python, but with built-in optimizations for distributed processing. They are lazily evaluated and implemented on top of RDDs, supporting operations like filtering, aggregation, and joining through a . DataFrames can be created from structured data files, Hive tables, external databases, or RDDs, and are available across Scala, , Python, and APIs. The API extends DataFrames by providing a type-safe interface, primarily in Scala and , where users can work with strongly typed objects while retaining the benefits of structured operations and optimizations. Datasets combine the capabilities of RDDs with the of DataFrames, allowing transformations using both functional and SQL-like methods. In Python and , the DataFrame API serves a similar role without the type safety of Datasets. Query optimization in Spark SQL is handled by the optimizer, which performs logical and physical planning to generate efficient execution plans. uses a tree-based representation of queries and applies rule-based and cost-based optimizations, leveraging Scala's and quasiquotes for extensibility. It resolves tables and columns, applies predicate pushdown, and selects join algorithms based on data statistics, significantly improving performance over unoptimized plans. Key features of Spark SQL include support for user-defined functions (UDFs), which allow custom logic on rows or aggregates, and window functions for computations over a set of rows related to the current row, such as ranking or cumulative sums. It also natively handles data formats like , where schemas can be inferred and loaded as DataFrames, and , preserving schema and enabling efficient columnar storage and compression. In Spark 4.0 (released May 2025), ANSI SQL mode is enabled by default for improved compliance, and support for the VARIANT data type was added for handling like and . When spark.sql.ansi.enabled is set to false (via Spark configuration or session setting; for example, in Databricks Runtime for compatibility), to_date(expr) or CAST(expr AS DATE) provides safe string-to-date conversion by returning NULL for malformed or invalid dates instead of raising an error. For specific formats, to_date(expr, 'fmt') behaves the same way when ANSI mode is disabled, returning NULL on invalid inputs regardless of the mode. Spark SQL integrates with the Hive metastore to access metadata for Hive tables, supporting Hive and UDFs for compatibility with existing Hive warehouses. It also exposes a engine through JDBC and ODBC servers, enabling external tools to connect and execute queries directly without writing Spark code. Introduced in Spark 3.0, Adaptive Query Execution (AQE) enhances runtime performance by dynamically adjusting execution plans based on runtime statistics, including handling data skew in joins and aggregations through techniques like splitting skewed partitions. AQE is enabled by default in recent versions and includes features like dynamic coalesce for reducing shuffle partitions post-aggregation.

Structured Streaming

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine, introduced in Apache Spark 2.0 as an evolution from the earlier Spark Streaming model that relied on Discretized Streams (DStreams). It replaces the DStream-based API with a Dataset/DataFrame-based API, enabling unified batch and streaming processing while providing end-to-end exactly-once semantics through integration with replayable data sources and idempotent sinks. This shift allows developers to express streaming computations using the same high-level APIs as batch processing, simplifying development and maintenance. At its core, Structured Streaming treats input data streams as unbounded tables that are continuously appended, where streaming computations are expressed as operations on these tables, similar to static DataFrames. The engine processes data incrementally using a micro-batch approach by default, where incoming data is batched into small, discrete units and processed as a series of deterministic batch jobs triggered at configurable intervals. Triggers control the scheduling, such as processing available data upon arrival (default micro-batch) or at fixed intervals, enabling control over latency and throughput. Since Spark 2.3, an experimental continuous mode has been available, allowing sub-millisecond end-to-end latencies by continuously running tasks without batching, though it supports only at-least-once guarantees and a limited set of operations; this mode remains experimental in Spark 3.x and has seen refinements in Spark 4.0 for improved and . In Spark 4.0 (May 2025), the Arbitrary State API v2 was introduced for more flexible stateful , along with the State Data Source for easier debugging of streaming state. Structured Streaming supports a variety of input sources for ingesting continuous data, including for message queues, file systems for monitoring directories, and TCP sockets for simple network streams, among others that provide replayability for fault recovery. For file-based sources such as JSON and CSV, schema inference is supported by setting the inferSchema option to true on the DataStreamReader. There is no global configuration property named spark.sql.streaming.schemaInference. It is recommended as a best practice to explicitly define the schema rather than relying on inference, as inferred schemas can change over time across incoming files and cause runtime failures. Outputs, or , allow writing processed results to destinations such as the console for , in-memory tables for testing, or custom logic via the foreach sink for integration with external systems, with support for idempotent operations to ensure reliable delivery. Fault tolerance is achieved through checkpointing, where the engine periodically saves the current progress, including offsets from sources and intermediate state, to a reliable storage like HDFS or cloud object stores, enabling recovery from failures by restarting from the last checkpoint. For handling late-arriving or out-of-order data based on event time, watermarking defines a threshold beyond which old data is discarded, allowing the system to bound state size in aggregations and joins while processing within acceptable delays— for instance, a 10-minute watermark retains data up to 10 minutes late. The processing model guarantees at-least-once delivery by default through output modes like or complete, but achieves end-to-end exactly-once semantics when using replayable sources (e.g., Kafka with committed offsets) and idempotent sinks, combined with Write-Ahead Logs and checkpointing to prevent duplicates or losses during failures. This ensures reliable even in distributed environments. In terms of , Structured Streaming enables low-latency with sub-second trigger intervals in micro-batch mode, often achieving latencies under 250 milliseconds for operational workloads through optimizations like adaptive query execution. It integrates seamlessly with Spark SQL, allowing streaming DataFrames to join with static tables or other streams using familiar query expressions, supporting complex operations like aggregations and windowing without custom code. For scalability, queries can be deployed on Spark cluster managers like or , distributing across nodes to handle high-throughput streams.

MLlib Machine Learning Library

MLlib is Apache Spark's open-source distributed library, designed to enable scalable implementation of common algorithms and workflows across clusters. It integrates seamlessly with Spark's core APIs, supporting languages such as Scala, , Python, and , and operates on distributed data structures like DataFrames for efficient processing of large-scale datasets. The library emphasizes ease of use by providing high-level abstractions for data preparation, model training, and evaluation, while leveraging Spark's in-memory computation to achieve performance gains over traditional disk-based systems. MLlib supports a range of algorithms for core machine learning tasks, including classification with logistic regression and decision trees, regression via generalized linear models, clustering through K-means, dimensionality reduction using Principal Component Analysis (PCA), and recommendation systems employing Alternating Least Squares (ALS). These algorithms are implemented to distribute computations across cluster nodes, allowing them to handle datasets that exceed single-machine memory limits. For instance, logistic regression in MLlib uses stochastic gradient descent for training on massive feature vectors, while K-means employs scalable variants like k-means++ for initialization to ensure convergence on billions of data points. Decision trees and their ensembles, such as random forests, support both classification and regression by recursively partitioning data based on feature splits, optimized for distributed execution. Feature engineering in MLlib is facilitated by a suite of transformers and estimators that process into suitable formats for modeling. Transformers such as Tokenizers convert text into word arrays, while StandardScaler normalizes features to unit variance, enabling consistent scaling across distributed partitions. The Pipeline API allows users to chain these stages—such as feature extraction, transformation, and selection—into reusable workflows, treating the entire sequence as a single estimator for simplified management and serialization. This design promotes modularity, where stages like vector assemblers combine multiple feature columns into a single vector, streamlining input to downstream algorithms. Model training in MLlib utilizes distributed optimization techniques tailored for large datasets, including Limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) for smooth convex problems like , which iteratively approximates the Hessian to converge efficiently on terabyte-scale data. Hyperparameter tuning is supported through tools like CrossValidator, which performs k-fold cross-validation over parameter grids in a distributed manner, evaluating models on held-out subsets to select optimal configurations without manual intervention. For advanced features, MLlib includes Gaussian Mixture Models for probabilistic clustering via expectation-maximization, modeling data as mixtures of multivariate Gaussians, and survival regression through Accelerated Failure Time (AFT) models based on Weibull distributions for time-to-event analysis. Integration with external libraries, such as , is possible via community wrappers like TensorFlowOnSpark, enabling hybrid workflows where Spark handles data preprocessing and performs inference. In Spark 3.x releases, MLlib received enhancements for improved GPU support through integration with the RAPIDS Accelerator, which offloads compatible algorithms like and K-means to GPUs for up to 10x speedups on certain workloads, alongside optimizations in vector assembly for faster feature construction. In Spark 4.0 (May 2025), MLlib gained support for execution via Spark Connect, allowing remote ML pipeline operations. metrics are built into MLlib's evaluators, providing functions to compute accuracy for tasks and F1-score as a of , applicable to both binary and multiclass scenarios via DataFrame-based APIs. These metrics facilitate model assessment on distributed test sets, often combined with pipelines for end-to-end validation. preparation for MLlib commonly leverages Spark SQL DataFrames for querying and transforming structured data prior to modeling.

GraphX Graph Processing

GraphX is Apache Spark's API for graph-parallel computation, providing abstractions for representing and processing large-scale graphs within the Spark ecosystem. Built directly on top of Spark's resilient distributed datasets (RDDs), it enables developers to perform graph analytics alongside other data processing tasks in a unified framework. This integration allows seamless mixing of graph operations with Spark's core dataflow capabilities, facilitating efficient handling of graph data in distributed environments. In GraphX, graphs are modeled as property graphs, where vertices are represented as an RDD of tuples in the form (VertexId, VertexValue), with VertexId being a unique 64-bit long identifier and VertexValue holding arbitrary user-defined attributes. Edges are similarly represented as an RDD of Edge[EdgeValue] objects, each containing source and destination VertexIds along with an EdgeValue for properties like weights. This dual-RDD structure supports flexible property graphs while leveraging Spark's fault-tolerant distribution and in-memory computation. GraphX enforces no specific ordering on vertex identifiers, allowing broad applicability to various graph topologies. GraphX offers a range of operators for manipulating graphs, categorized into basic and structural types. Basic operators include joinVertices, which augments vertex attributes based on neighbor information, and subgraph, which filters vertices and edges to create induced subgraphs. Structural operators provide high-level analytics such as connectedComponents for identifying disjoint graph partitions and triangleCounting for measuring local clustering. These operators are optimized for parallel execution, reducing the need for low-level RDD manipulations. For iterative algorithms, GraphX implements the Pregel API, an abstraction for message-passing computations inspired by the Pregel system. It supports algorithms like , which iteratively propagates rankings across vertices, and single-source shortest paths, where messages accumulate distances from a source. The API operates in supersteps, sending messages along edges and updating vertex states until convergence, all while maintaining Spark's distribution and fault tolerance. Graphs in GraphX can be constructed from existing RDDs of vertices and edges or built from collections such as edge lists using utility methods like GraphLoader.edgeListFile. Optimizations include edge partitioning strategies, such as random vertex cut, which balance computational load by co-partitioning vertices with high-degree neighbors to minimize communication overhead. These builders ensure efficient loading and partitioning for large-scale graphs. Performance-wise, GraphX leverages Spark's in-memory processing to handle graphs with billions of edges; for instance, it computes on a 1-billion-edge graph across 32 machines in under 40 seconds. It integrates with Spark's MLlib for graph-based tasks, such as community detection, by converting graphs to compatible formats. This enables scalable analytics without data movement. Despite its strengths, GraphX has limitations, particularly in handling dynamic graphs with frequent updates, where its static representation can lead to inefficiencies. For scenarios requiring SQL-like declarative queries on graphs, the GraphFrames library, built on DataFrames, is recommended as a more flexible alternative.

Language Support and APIs

Supported Programming Languages

Apache Spark primarily supports Scala as its native programming language, in which the framework is implemented on the (JVM). As of Spark 4.0, applications using the Scala API must use Scala 2.13 for compatibility. This allows developers familiar with paradigms to leverage Scala's concise syntax for building Spark applications directly against the core abstractions. The API provides a full-featured interface tailored for enterprise environments, offering semantics closely aligned with the Scala API to ensure consistency in operations like data transformations and job submissions. Spark 4.0 requires 17 or 21. It enables Java developers to interact with Spark's resilient distributed datasets (RDDs) and higher-level structures without needing to learn Scala-specific constructs. PySpark offers a high-level Python API through Pythonic wrappers around Spark's JVM-based components, making it particularly popular among data scientists for its integration with libraries like and . As of Spark 4.0, Python 3.9 or higher is required, building on support for Python 3.6+ in early 3.x releases and 3.7+ in later 3.x versions. Key features include Pandas User-Defined Functions (UDFs), which enable vectorized operations on batches of data using for efficient transfer between Spark and Pandas DataFrames. Additionally, PySpark supports broadcast joins via hints that mark small DataFrames for efficient distribution across the cluster, optimizing performance in join operations. Spark 4.0 introduces enhancements such as an improved pandas API on Spark for better compatibility with pandas workloads, native Python Data Source API for custom data sources in Python, and a lightweight pyspark-client (1.5 MB) for easier remote connections. In Spark 4.1.0, further improvements to PySpark UDFs and Data Sources include new Arrow-native UDF and UDTF decorators for efficient PyArrow execution without Pandas conversion overhead, and Python Data Source filter pushdown to reduce data movement. SparkR integrates R's DataFrame with Spark, allowing R users to perform distributed with operations resembling those in base R and packages like , such as filtering, aggregation, and grouping. As of Spark 4.0, the R requires R 3.5+ but is deprecated and scheduled for removal in a future release. It supports column-based access through functions like select() and column(), enabling intuitive manipulation of Spark DataFrames in an R-native style. These language APIs are utilized across Spark's modules, including Spark SQL and MLlib, to provide unified access to and capabilities. Language support evolves with Spark releases; for instance, the deprecation of SparkR in Spark 4.0 signals a shift toward stronger Python and Scala focus.

API Design and Usage

Apache Spark's APIs are structured in layers to cater to different levels of abstraction and use cases. The low-level Resilient Distributed Dataset (RDD) API provides fine-grained control over distributed , allowing developers to perform arbitrary operations on data partitions across a cluster. In contrast, the high-level and DataFrame APIs, introduced for structured and , offer domain-specific optimizations, such as optimizer integration for SQL-like queries and in languages like Scala and . These higher-level APIs abstract away much of the RDD complexity, enabling more for tasks like and . Central to Spark's design are principles of immutability, , and . RDDs and Datasets are immutable collections, ensuring that transformations create new objects without modifying originals, which supports and parallelism. The functional style emphasizes operations like map, filter, and reduce, drawing from functional languages to promote composable and concise code. defers computation until necessary, building a (DAG) of transformations that the optimizer can rearrange for efficiency, reducing unnecessary data movement and I/O. A core pattern in Spark APIs distinguishes between transformations and actions. Transformations, such as or join, are lazy operations that return a new RDD or without immediate execution, allowing Spark to optimize the overall plan. Actions, like collect or count, trigger eager computation, materializing results and initiating the DAG execution across the cluster. For performance-critical workflows, caching or persistence enables reusing intermediate results in memory or disk, avoiding recomputation in iterative algorithms. Spark APIs facilitate cross-module usage through unified entry points. The SparkSession serves as a shared interface for SQL operations, providing access to DataFrames and optimizer without needing a separate SQLContext. For streaming, the legacy StreamingContext for DStream-based has been deprecated in favor of Structured Streaming, which integrates directly with SparkSession for continuous and SQL queries. In , MLlib pipelines chain transformers and estimators using APIs, leveraging SparkSession for end-to-end workflows from preparation to model evaluation. A notable addition in recent releases is Spark Connect, introduced in Spark 3.4 and enhanced in 4.0, which provides a client-server for remote execution. It supports the DataFrame API in PySpark and DataFrame/ APIs in Scala, allowing thin-client connections to Spark clusters without full Spark installation on the driver. Best practices in Spark API usage emphasize minimizing resource-intensive operations. Developers should avoid wide transformations, such as groupBy or join on non-partitioned keys, which trigger costly shuffles redistributing data across nodes; instead, prefer narrow transformations like mapPartitions for local operations. For efficient lookups, broadcast variables distribute small read-only datasets (typically under 100 MB) to all nodes, eliminating repeated joins and reducing network traffic. Error handling in Spark APIs involves anticipating common exceptions and leveraging built-in tools for diagnostics. OutOfMemoryError often arises from insufficient executor memory during shuffles or caching large datasets, resolvable by tuning spark.executor.memory or increasing partitions to balance load. The Spark UI provides a web-based interface to monitor job stages, task metrics, and storage usage, aiding in debugging by visualizing DAG execution and identifying bottlenecks like skewed partitions. A significant evolution in Spark's API design occurred in version 2.0 with the unification of DataFrame and APIs. This merged the untyped DataFrame for SQL interoperability with the typed for compile-time safety in Scala and , eliminating the need for separate contexts and streamlining development across modules. Python and users continued with DataFrames only, as full Dataset typing was not supported, but the change enhanced overall API consistency and performance through shared optimizations.

History and Development

Origins and Initial Development

Apache Spark originated as a research project at the , Berkeley's AMPLab in , initiated to overcome the limitations of in handling iterative and interactive data processing tasks. Hadoop's model, while effective for , incurred high overheads due to disk-based intermediate storage, making it inefficient for applications requiring multiple passes over the same data, such as machine learning algorithms and scientific simulations. The project aimed to enable faster in-memory analytics, supporting workloads in scientific computing and by introducing a unified engine for both batch and iterative computations. The core development was led by , with significant contributions from , , Patrick Wendell, and other members of the AMPLab team, including Mosharaf Chowdhury, Tathagata Das, and . Early work focused on the Resilient Distributed Datasets (RDD) abstraction, a fault-tolerant mechanism for in-memory cluster computing, which formed the foundation of Spark's design and was detailed in a seminal paper presented at the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI) in 2012, earning the conference's Best Paper Award. This publication highlighted how RDDs could accelerate iterative jobs by up to 20 times compared to Hadoop. The project was initially funded through grants from the (NSF), including a $10 million Expeditions in , and the Defense Advanced Research Projects Agency (), supporting AMPLab's broader initiatives. Spark was first released as in early 2010 under the BSD license, with version 0.5.0 following in June 2012, introducing features like Mesos support and improved Hadoop integration. In 2013, the project transitioned to the Apache Software Foundation's Incubator program, marking its shift toward broader community governance. Implemented primarily in Scala, Spark quickly gained traction among researchers for its expressive and performance gains.

Major Releases and Evolution

Apache Spark achieved top-level project status within on February 27, 2014, enabling independent governance and community-driven development. Shortly thereafter, version 1.0.0 was released on May 30, 2014, marking the first stable release with guaranteed API compatibility across the 1.x series, including core APIs in Scala, , and Python. This version introduced Spark SQL as an alpha feature for structured data processing, laying the groundwork for DataFrames, alongside optimizations in Spark Streaming and expansions to the MLlib library, such as support for sparse vectors and new algorithms like decision trees. The 1.x series, spanning 2014 to 2016, focused on maturing core components, with DataFrames formally introduced in version 1.3 (September 2015) to enable relational operations on distributed datasets via a SQL-like interface. Spark Streaming saw performance enhancements for stateful operations, while MLlib evolved with additional algorithms and better scalability, solidifying its role as a distributed framework. By the end of the series with version 1.6 (July 2016), these modules provided robust foundations for batch, streaming, and analytical workloads. Spark 2.x, from 2016 to 2019, emphasized unification and optimization, beginning with (July 2016), which merged the DataFrame and APIs into a single, type-safe interface supporting both structured and . Structured Streaming debuted as a scalable, fault-tolerant engine built on the API, representing a shift from the earlier DStreams model by treating streams as unbounded tables for unified batch and streaming queries. Project Tungsten introduced whole-stage code generation and off-heap to boost execution efficiency, reducing JVM overhead and improving CPU utilization by up to 10x in some workloads. Later releases, such as 2.4 (November 2018), added barrier execution mode to synchronize tasks across stages, facilitating integration with iterative algorithms like those in . The 3.x series, released from 2020 to 2025, advanced query optimization and ecosystem compatibility, starting with version 3.0 (June 2020), which introduced Adaptive Query Execution (AQE) to dynamically adjust query plans at runtime based on , alongside dynamic partition to eliminate unnecessary data scans during joins. Version 3.2 (October 2021) brought the Pandas API on Spark, allowing Python users to scale pandas operations across clusters with minimal code changes, while enhancing support for Python 3.8 and beyond. integration improved with better operator support and resource management, enabling native deployment on containerized environments. The series culminated in version 3.5 (September 2023), incorporating further AQE refinements and ANSI SQL compliance for broader interoperability. Spark maintains a bi-annual release cadence for major versions, with branches cut in and , followed by patch releases addressing vulnerabilities and critical bugs. This rhythm ensures timely feature delivery while sustaining long-term support for prior series. In 3.x, integrations like Delta Lake for transactions on data lakes were deepened, enhancing reliability for production workloads.

Current Maintenance and Future Directions

Apache Spark is governed by under a Project Management Committee (PMC) that oversees development, releases, and community contributions, with active committers numbering 92 as of 2025, including a significant portion affiliated with as a leading corporate contributor. engineers, such as Michael Armbrust and Tathagata Das, hold key roles in the PMC, driving substantial code contributions and feature advancements. Maintenance follows a structured versioning , with (LTS) branches like 3.5.x receiving bug fixes and security updates for approximately 31 months; for instance, Spark 3.5.0, released in September 2023, is supported until April 2026. Older series, such as 2.x branches, reached end-of-life (EOL) by 2023, with the final releases concluding around that time to focus resources on newer versions. The community exceeds over 2,000 contributors overall, fostering collaborative development through platforms like JIRA for issue tracking and pull requests. Annual events, including the Data + AI Summit, facilitate roadmap discussions, with sessions dedicated to Spark's evolution, performance optimizations, and integration strategies. Looking ahead, Spark emphasizes enhanced AI and integration, exemplified by Spark Connect—introduced in version 3.4—which enables remote client-server APIs for decoupled, access to Spark clusters, streamlining distributed ML workflows. Future enhancements prioritize cloud-native compatibility, with tighter integrations to platforms like AWS, Azure, and Google Cloud for seamless deployment in containerized environments. As of 2025, Spark 4.0 introduces the VARIANT data type for efficient handling of semi-structured data, supporting schema evolution without rigid upfront definitions, alongside performance boosts like improved vectorized execution in SQL operators to accelerate query processing on large datasets. In December 2025, Spark 4.1.0 was released, featuring significant PySpark enhancements, including Python Arrow-native User-Defined Functions (UDFs) with decorators for efficient PyArrow execution, vectorized UDFs via the @udf decorator, improved support for User-Defined Types (UDTs) in Arrow-optimized UDFs, filter pushdown for Python Data Sources, and Arrow writer support for streaming Python data sources, aimed at boosting performance and usability in Python-based distributed processing. Amid these advancements, Spark faces challenges from serverless competitors like AWS Glue, which offers simpler ETL for non-experts through managed scaling and visual interfaces, contrasting Spark's steeper for custom distributed processing. To address usability, ongoing efforts focus on intuitive APIs and tools like PySpark enhancements in 4.0 and 4.1.0, aiming to broaden adoption beyond specialists while maintaining its edge in scalable, real-time analytics.

Ecosystem and Integrations

Apache Spark integrates closely with several other Apache projects to enhance its capabilities in data storage, processing, and execution within ecosystems. These integrations leverage Spark's core engine to provide seamless , enabling users to build robust data pipelines without custom middleware. Apache provides an in-memory columnar format that facilitates zero-copy data exchange between Spark and other tools, such as in Python, reducing overhead and improving performance for data transfer across processes. This integration is natively supported in Spark's PySpark API, enabling efficient handling of large datasets in hybrid JVM-Python environments. Spark acts as an execution engine for queries, sharing the Hive Metastore to access structured data catalogs and execute SQL operations on Hive tables without requiring Hive's native backend. This setup allows Spark to process Hive-compatible data sources directly, supporting input/output formats defined in Hive for seamless migration and hybrid usage. Native connectors enable Spark SQL to interact with for distributed, scalable storage, treating HBase tables as external data sources for read/write operations via DataFrames. The Spark Connector library supports this integration, allowing Spark applications to query HBase regions efficiently while maintaining Spark's distributed execution model. Similarly, the Spark Connector provides native support for , enabling Spark to expose Cassandra tables as RDDs or DataFrames and perform bulk reads/writes using CQL queries within Spark jobs. This connector optimizes data movement between Spark and Cassandra clusters, supporting wide-column storage for high-throughput applications. Apache Iceberg is an open table format that enables reliable data lakes with features like transactions, schema evolution, and time travel, natively integrated with Spark via Spark SQL and DataFrames. As an top-level project since 2020, Iceberg allows Spark users to manage large analytic datasets across storage systems like on object stores, with built-in support for partitioning and hidden partitioning to optimize query performance.

Third-Party Integrations and Tools

Delta Lake serves as an open-source storage layer that brings transactions, , and schema enforcement to Spark DataFrames, allowing reliable data management on data lakes. Originally developed by and open-sourced in 2019, Delta Lake evolved into a standalone project under the Delta Lake initiative, with significant adoption for its reliability features in Spark workflows. SynapseML, contributed by , extends Apache Spark with scalable capabilities, including integrations for , cognitive services, and distributed algorithms, hosted as an open-source library compatible with Spark's ecosystem. It unifies multiple ML frameworks under a single , enabling AI workflows on Spark clusters. is a unified analytics platform built on Apache Spark, offering collaborative notebooks for interactive data exploration and AutoML capabilities to automate workflows. This integration enhances Spark's usability by providing a managed environment that supports scalable data processing, AI model training, and deployment across cloud infrastructures. Cloudera Data Platform (CDP), resulting from the merger of and Hortonworks, incorporates Apache Spark into its enterprise distribution for secure management, enabling features like and multi-cloud deployments. CDP's Spark runtime supports distributed processing with built-in protocols, facilitating compliance in regulated industries such as and healthcare. Alluxio serves as a distributed file system that caches data across heterogeneous storage systems, accelerating Apache Spark's I/O operations by reducing latency in data access from sources like HDFS or cloud object stores. By positioning Alluxio as an intermediary layer, Spark jobs can achieve up to 4x faster processing in certain workloads, such as reading RDDs, through intelligent caching policies. Konduit Serving enables the deployment of models trained in Apache Spark to production environments, supporting inference via APIs and integration with frameworks like for distributed execution. This tool bridges Spark's with real-time serving, allowing scalable model inference without custom infrastructure. Commercial connectors extend Spark's ecosystem through ; for instance, AWS Glue utilizes Spark for serverless ETL jobs, automating data preparation and integration from over 100 sources with minimal setup. Similarly, Google Cloud Dataproc provides a fully managed Spark service, optimizing cluster provisioning and auto-scaling for cost-efficient . Among open-source extensions, GraphFrames offers DataFrame-based graph processing for Apache Spark, enabling expressive queries and algorithms like using Spark SQL syntax for easier integration with structured data pipelines. Koalas, a legacy library implementing the API on Spark, has been superseded by the native API on Spark introduced in Spark 3.2, which provides similar functionality with improved performance and official maintenance. As of 2025, integrations with vector databases like are rising to support AI workloads, with the Spark-Milvus connector facilitating efficient ingestion and querying of high-dimensional vectors for applications such as and recommendation systems. This trend enhances Spark's role in generative AI pipelines by combining its distributed processing with vector similarity search capabilities.

References

  1. https://spark.apache.org/
  2. https://spark.apache.org/history.html
  3. https://www.databricks.com/dataaisummit/speaker/matei-zaharia
  4. https://aws.amazon.com/what-is/apache-spark/
  5. https://spark.apache.org/research.html
  6. https://spark.apache.org/docs/latest/
  7. https://spark.apache.org/faq.html
  8. https://people.eecs.berkeley.edu/~yelick/talks/data/DataDance-JulianCon17.pdf
  9. https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf
  10. https://spark.apache.org/docs/latest/rdd-programming-guide.html
  11. https://spark.apache.org/docs/latest/cluster-overview.html
  12. https://spark.apache.org/docs/latest/api/python/index.html
  13. https://www.databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html
  14. https://spark.apache.org/docs/latest/spark-standalone.html
  15. https://spark.apache.org/docs/latest/running-on-yarn.html
  16. https://spark.apache.org/docs/3.5.3/running-on-mesos.html
  17. https://spark.apache.org/releases/spark-release-4-0-0.html
  18. https://spark.apache.org/docs/latest/running-on-kubernetes.html
  19. https://spark.apache.org/docs/latest/submitting-applications.html
  20. https://spark.apache.org/docs/latest/security.html
  21. https://spark.apache.org/docs/latest/monitoring.html
  22. https://spark.apache.org/docs/latest/sql-programming-guide.html
  23. https://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf
  24. https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_df.html
  25. https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/Dataset.html
  26. https://www.databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html
  27. https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html
  28. https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-window.html
  29. https://spark.apache.org/docs/latest/sql-data-sources-json.html
  30. https://spark.apache.org/docs/latest/sql-data-sources-parquet.html
  31. https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html
  32. https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html
  33. https://spark.apache.org/sql/
  34. https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html
  35. https://spark.apache.org/docs/latest/sql-performance-tuning.html
  36. https://spark.apache.org/releases/spark-release-3-0-0.html
  37. https://spark.apache.org/docs/3.5.1/structured-streaming-programming-guide.html
  38. https://www.databricks.com/blog/introducing-apache-spark-40
  39. https://www.databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html
  40. https://spark.apache.org/docs/latest/ml-guide.html
  41. https://www.jmlr.org/papers/volume17/15-237/15-237.pdf
  42. https://spark.apache.org/docs/latest/ml-classification-regression.html
  43. https://spark.apache.org/docs/latest/ml-clustering.html
  44. https://spark.apache.org/mllib/
  45. https://spark.apache.org/docs/latest/ml-features.html
  46. https://spark.apache.org/docs/latest/ml-pipeline.html
  47. https://github.com/yahoo/TensorFlowOnSpark
  48. https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html#evaluation
  49. https://spark.apache.org/graphx/
  50. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-gonzalez.pdf
  51. https://spark.apache.org/docs/latest/graphx-programming-guide.html
  52. https://graphframes.io/01-about/01-index.html
  53. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.pandas_udf.html
  54. https://spark.apache.org/releases/spark-release-4.1.0.html
  55. https://spark.apache.org/docs/latest/streaming-programming-guide.html
  56. https://spark.apache.org/docs/latest/spark-connect-overview.html
  57. https://spark.apache.org/docs/latest/tuning.html
  58. https://spark.apache.org/docs/latest/configuration.html
  59. https://spark.apache.org/releases/spark-release-2-0-0.html
  60. https://people.eecs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf
  61. https://amplab.cs.berkeley.edu/wp-content/uploads/2011/06/Spark-Cluster-Computing-with-Working-Sets.pdf
  62. https://obamawhitehouse.archives.gov/the-press-office/2015/11/19/fact-sheet-data-knowledge-action
  63. https://wordpress.cels.anl.gov/atpesc/wp-content/uploads/sites/96/2017/08/ATPESC_2017_Dinner_Talk_7_8-8_Franklin-Big_Data_Analytics.pdf
  64. https://www.educba.com/spark-versions/
  65. https://spark.apache.org/news/spark-accepted-into-apache-incubator.html
  66. https://news.apache.org/foundation/entry/the_apache_software_foundation_announces50
  67. https://spark.apache.org/releases/spark-release-1-0-0.html
  68. https://www.databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html
  69. https://www.databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html
  70. https://spark.apache.org/releases/spark-release-2-4-0.html
  71. https://www.databricks.com/blog/2021/10/04/pandas-api-on-upcoming-apache-spark-3-2.html
  72. https://spark.apache.org/releases/spark-release-3-5-0.html
  73. https://spark.apache.org/versioning-policy.html
  74. https://spark.apache.org/committers.html
  75. https://www.databricks.com/dataaisummit/speaker/tathagata-das
  76. https://endoflife.date/apache-spark
  77. https://www.ibm.com/think/topics/apache-spark
  78. https://www.databricks.com/dataaisummit/session/apache-spark-ask-us-anything
  79. https://www.linkedin.com/pulse/apache-spark-2025-look-ahead-signore-technologies-qkdbc
  80. https://engineering.datagalaxy.com/apache-spark-4-0-is-here-top-features-revolutionizing-data-engineering-and-analytics-8d273d7c4621
  81. https://www.linkedin.com/pulse/aws-glue-vs-apache-spark-choosing-right-tool-your-data-shashank-k-olylc
  82. https://www.analyticsvidhya.com/blog/2024/08/apache-spark-4-0/
  83. https://spark.apache.org/docs/latest/api/python/tutorial/sql/arrow_pandas.html
  84. https://hive.apache.org/docs/latest/admin/hive-on-spark-getting-started/
  85. https://github.com/apache/hbase-connectors
  86. https://github.com/apache/cassandra-spark-connector
  87. https://iceberg.apache.org/spark-quickstart/
  88. https://iceberg.apache.org/
  89. https://delta.io/
  90. https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf
  91. https://www.microsoft.com/en-us/research/project/synapseml/
  92. https://github.com/microsoft/SynapseML
  93. https://www.databricks.com/glossary/what-is-apache-spark
  94. https://docs.databricks.com/aws/en/spark/
  95. https://docs.cloudera.com/runtime/7.3.1/spark-overview/topics/spark-overview.html
  96. https://documentation.alluxio.io/os-en/compute/spark
  97. https://www.alluxio.io/blog/effective-spark-rdds-with-alluxio
  98. https://github.com/KonduitAI/konduit-serving
  99. https://deeplearning4j.konduit.ai/
  100. https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html
  101. https://cloud.google.com/dataproc
  102. https://graphframes.io/
  103. https://spark.apache.org/docs/latest/api/python/migration_guide/koalas_to_pyspark.html
  104. https://milvus.io/docs/integrate_with_spark.md
  105. https://github.com/zilliztech/spark-milvus
Add your contribution
Related Hubs
User Avatar
No comments yet.