Community Hub0 Subscribers
Community hub
Data engineering
View on Wikipediafrom Wikipedia
| This article is part of a series on |
| Engineering |
|---|
 |
Data engineering is a software engineering approach to the building of data systems, to enable the collection and usage of data. This data is usually used to enable subsequent analysis and data science, which often involves machine learning.[1][2] Making the data usable usually involves substantial computing and storage, as well as data processing.
History
[edit]Around the 1970s/1980s the term information engineering methodology (IEM) was created to describe database design and the use of software for data analysis and processing.[3] These techniques were intended to be used by database administrators (DBAs) and by systems analysts based upon an understanding of the operational processing needs of organizations for the 1980s. In particular, these techniques were meant to help bridge the gap between strategic business planning and information systems. A key early contributor (often called the "father" of information engineering methodology) was the Australian Clive Finkelstein, who wrote several articles about it between 1976 and 1980, and also co-authored an influential Savant Institute report on it with James Martin.[4][5][6] Over the next few years, Finkelstein continued work in a more business-driven direction, which was intended to address a rapidly changing business environment; Martin continued work in a more data processing-driven direction. From 1983 to 1987, Charles M. Richter, guided by Clive Finkelstein, played a significant role in revamping IEM as well as helping to design the IEM software product (user data), which helped automate IEM.
In the early 2000s, the data and data tooling was generally held by the information technology (IT) teams in most companies.[7] Other teams then used data for their work (e.g. reporting), and there was usually little overlap in data skillset between these parts of the business.
In the early 2010s, with the rise of the internet, the massive increase in data volumes, velocity, and variety led to the term big data to describe the data itself, and data-driven tech companies like Facebook and Airbnb started using the phrase data engineer.[3][7] Due to the new scale of the data, major firms like Google, Facebook, Amazon, Apple, Microsoft, and Netflix started to move away from traditional ETL and storage techniques. They started creating data engineering, a type of software engineering focused on data, and in particular infrastructure, warehousing, data protection, cybersecurity, mining, modelling, processing, and metadata management.[3][7] This change in approach was particularly focused on cloud computing.[7] Data started to be handled and used by many parts of the business, such as sales and marketing, and not just IT.[7]
Tools
[edit]Compute
[edit]High-performance computing is critical for the processing and analysis of data. One particularly widespread approach to computing for data engineering is dataflow programming, in which the computation is represented as a directed graph (dataflow graph); nodes are the operations, and edges represent the flow of data.[8] Popular implementations include Apache Spark, and the deep learning specific TensorFlow.[8][9][10] More recent implementations, such as Differential/Timely Dataflow, have used incremental computing for much more efficient data processing.[8][11][12]
Storage
[edit]Data is stored in a variety of ways, one of the key deciding factors is in how the data will be used. Data engineers optimize data storage and processing systems to reduce costs. They use data compression, partitioning, and archiving.
Databases
[edit]If the data is structured and some form of online transaction processing is required, then databases are generally used.[13] Originally mostly relational databases were used, with strong ACID transaction correctness guarantees; most relational databases use SQL for their queries. However, with the growth of data in the 2010s, NoSQL databases have also become popular since they horizontally scaled more easily than relational databases by giving up the ACID transaction guarantees, as well as reducing the object-relational impedance mismatch.[14] More recently, NewSQL databases — which attempt to allow horizontal scaling while retaining ACID guarantees — have become popular.[15][16][17][18]
Data warehouses
[edit]If the data is structured and online analytical processing is required (but not online transaction processing), then data warehouses are a main choice.[19] They enable data analysis, mining, and artificial intelligence on a much larger scale than databases can allow,[19] and indeed data often flow from databases into data warehouses.[20] Business analysts, data engineers, and data scientists can access data warehouses using tools such as SQL or business intelligence software.[20]
Data lakes
[edit]A data lake is a centralized repository for storing, processing, and securing large volumes of data. A data lake can contain structured data from relational databases, semi-structured data, unstructured data, and binary data. A data lake can be created on premises or in a cloud-based environment using the services from public cloud vendors such as Amazon, Microsoft, or Google.
Files
[edit]If the data is less structured, then often they are just stored as files. There are several options:
- File systems represent data hierarchically in nested folders.[21]
- Block storage splits data into regularly sized chunks;[21] this often matches up with (virtual) hard drives or solid state drives.
- Object storage manages data using metadata;[21] often each file is assigned a key such as a UUID.[22]
Management
[edit]The number and variety of different data processes and storage locations can become overwhelming for users. This inspired the usage of a workflow management system (e.g. Airflow) to allow the data tasks to be specified, created, and monitored.[23] The tasks are often specified as a directed acyclic graph (DAG).[23]
Lifecycle
[edit]Business planning
[edit]Business objectives that executives set for what's to come are characterized in key business plans, with their more noteworthy definition in tactical business plans and implementation in operational business plans. Most businesses today recognize the fundamental need to grow a business plan that follows this strategy. It is often difficult to implement these plans because of the lack of transparency at the tactical and operational degrees of organizations. This kind of planning requires feedback to allow for early correction of problems that are due to miscommunication and misinterpretation of the business plan.
Systems design
[edit]The design of data systems involves several components such as architecting data platforms, and designing data stores.[24][25]
Data modeling
[edit]Data modeling is the analysis and representation of data requirements for an organisation. It produces a data model—an abstract representation that organises business concepts and the relationships and constraints between them. The resulting artefacts guide communication between business and technical stakeholders and inform database design.[26][27]
A common convention distinguishes three levels of models:[26]
- Conceptual model – a technology-independent view of the key business concepts and rules.
- Logical model – a detailed representation in a chosen paradigm (most commonly the relational model) specifying entities, attributes, keys, and integrity constraints.[27]
- Physical model – an implementation-oriented design describing tables, indexes, partitioning, and other operational considerations.[27]
Approaches include entity–relationship (ER) modeling for operational systems,[28] dimensional modeling for analytics and data warehousing,[29] and the use of UML class diagrams to express conceptual or logical models in general-purpose modeling tools.[30]
Well-formed data models aim to improve data quality and interoperability by applying clear naming standards, normalisation, and integrity constraints.[27][26]
Roles
[edit]Data engineer
[edit]A data engineer is a type of software engineer who creates big data ETL pipelines to manage the flow of data through the organization. This makes it possible to take huge amounts of data and translate it into insights.[31] They are focused on the production readiness of data and things like formats, resilience, scaling, and security. Data engineers usually hail from a software engineering background and are proficient in programming languages like Java, Python, Scala, and Rust.[32][3] They will be more familiar with databases, architecture, cloud computing, and Agile software development.[3]
Data scientist
[edit]Data scientists are more focused on the analysis of the data, they will be more familiar with mathematics, algorithms, statistics, and machine learning.[3][33]
See also
[edit]References
[edit]- ^ "What is Data Engineering? | A Quick Glance of Data Engineering". EDUCBA. January 5, 2020. Retrieved July 31, 2022.
- ^ "Introduction to Data Engineering". Dremio. Retrieved July 31, 2022.
- ^ a b c d e f Black, Nathan (January 15, 2020). "What is Data Engineering and Why Is It So Important?". QuantHub. Retrieved July 31, 2022.
- ^ "Information engineering," part 3, part 4, part 5, Part 6" by Clive Finkelstein. In Computerworld, In depths, appendix. May 25 – June 15, 1981.
- ^ Christopher Allen, Simon Chatwin, Catherine Creary (2003). Introduction to Relational Databases and SQL Programming.
- ^ Terry Halpin, Tony Morgan (2010). Information Modeling and Relational Databases. p. 343
- ^ a b c d e Dodds, Eric. "The History of the Data Engineering and the Megatrends". Rudderstack. Retrieved July 31, 2022.
- ^ a b c Schwarzkopf, Malte (March 7, 2020). "The Remarkable Utility of Dataflow Computing". ACM SIGOPS. Retrieved July 31, 2022.
- ^ "sparkpaper" (PDF). Retrieved July 31, 2022.
- ^ Abadi, Martin; Barham, Paul; Chen, Jianmin; Chen, Zhifeng; Davis, Andy; Dean, Jeffrey; Devin, Matthieu; Ghemawat, Sanjay; Irving, Geoffrey; Isard, Michael; Kudlur, Manjunath; Levenberg, Josh; Monga, Rajat; Moore, Sherry; Murray, Derek G.; Steiner, Benoit; Tucker, Paul; Vasudevan, Vijay; Warden, Pete; Wicke, Martin; Yu, Yuan; Zheng, Xiaoqiang (2016). "TensorFlow: A system for large-scale machine learning". 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). pp. 265–283. Retrieved July 31, 2022.
- ^ McSherry, Frank; Murray, Derek; Isaacs, Rebecca; Isard, Michael (January 5, 2013). "Differential dataflow". Microsoft. Retrieved July 31, 2022.
- ^ "Differential Dataflow". Timely Dataflow. July 30, 2022. Retrieved July 31, 2022.
- ^ "Lecture Notes | Database Systems | Electrical Engineering and Computer Science | MIT OpenCourseWare". ocw.mit.edu. Retrieved July 31, 2022.
- ^ Leavitt, Neal (February 2010). "Will NoSQL Databases Live Up to Their Promise?". Computer. 43 (2): 12–14. doi:10.1109/MC.2010.58.
- ^ Aslett, Matthew (2011). "How Will The Database Incumbents Respond To NoSQL And NewSQL?" (PDF). 451 Group (published April 4, 2011). Retrieved February 22, 2020.
- ^ Pavlo, Andrew; Aslett, Matthew (2016). "What's Really New with NewSQL?" (PDF). SIGMOD Record. Retrieved February 22, 2020.
- ^ Stonebraker, Michael (June 16, 2011). "NewSQL: An Alternative to NoSQL and Old SQL for New OLTP Apps". Communications of the ACM Blog. Retrieved February 22, 2020.
- ^ Hoff, Todd (September 24, 2012). "Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In". Retrieved February 22, 2020.
- ^ a b "What is a Data Warehouse?". www.ibm.com. Retrieved July 31, 2022.
- ^ a b "What is a Data Warehouse? | Key Concepts | Amazon Web Services". Amazon Web Services, Inc. Retrieved July 31, 2022.
- ^ a b c "File storage, block storage, or object storage?". www.redhat.com. Retrieved July 31, 2022.
- ^ "Cloud Object Storage – Amazon S3 – Amazon Web Services". Amazon Web Services, Inc. Retrieved July 31, 2022.
- ^ a b "Home". Apache Airflow. Retrieved July 31, 2022.
- ^ "Introduction to Data Engineering". Coursera. Retrieved July 31, 2022.
- ^ Finkelstein, Clive. What are The Phases of Information Engineering.
- ^ a b c Simsion, Graeme; Witt, Graham (2015). Data Modeling Essentials (4th ed.). Morgan Kaufmann. ISBN 9780128002025.
- ^ a b c d Date, C. J. (2004). An Introduction to Database Systems (8th ed.). Addison-Wesley. ISBN 9780321197849.
- ^ Chen, Peter P. (1976). "The Entity–Relationship Model—Toward a Unified View of Data". ACM Transactions on Database Systems. 1 (1): 9–36. doi:10.1145/320434.320440.
- ^ Kimball, Ralph; Ross, Margy (2013). The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling (3rd ed.). Wiley. ISBN 9781118530801.
- ^ Unified Modeling Language (UML) Version 2.5.1 (Report). Object Management Group. 2017.
- ^ Tamir, Mike; Miller, Steven; Gagliardi, Alessandro (December 11, 2015). The Data Engineer (Report). SSRN 2762013.
- ^ "Data Engineer vs. Data Scientist". Springboard Blog. February 7, 2019. Retrieved March 14, 2021.
- ^ "What is Data Science and Why it's Important". Edureka. January 5, 2017.
Further reading
[edit]- Hares, John S. (1992). Information Engineering for the Advanced Practitioner. Wiley. ISBN 978-0-471-92810-2.
- Finkelstein, Clive (1989). An Introduction to Information Engineering: From Strategic Planning to Information Systems. Addison-Wesley. ISBN 978-0-201-41654-1.
- Finkelstein, Clive (1992). Information Engineering: Strategic Systems Development. Addison-Wesley. ISBN 978-0-201-50988-5.
- Ian Macdonald (1986). "Information engineering". in: Information Systems Design Methodologies. T.W. Olle et al. (ed.). North-Holland.
- Ian Macdonald (1988). "Automating the Information engineering methodology with the Information Engineering Facility". In: Computerized Assistance during the Information Systems Life Cycle. T.W. Olle et al. (ed.). North-Holland.
- James Martin and Clive Finkelstein. (1981). Information engineering. Technical Report (2 volumes), Savant Institute, Carnforth, Lancs, UK.
- James Martin (1989). Information engineering. (3 volumes), Prentice-Hall Inc.
- Finkelstein, Clive (2006). Enterprise Architecture for Integration: Rapid Delivery Methods and Technologies. Artech House. ISBN 978-1-58053-713-1.
- Reis, Joe; Housley, Matt (2022). Fundamentals of Data Engineering. O'Reilly Media. ISBN 978-1-0981-0827-4.
External links
[edit]
Wikimedia Commons has media related to Information Engineering.
- The Complex Method IEM Archived July 20, 2019, at the Wayback Machine
- Rapid Application Development
- Enterprise Engineering and Rapid Delivery of Enterprise Architecture
Data engineering
View on Grokipediafrom Grokipedia
Definition and Overview
Definition
Data engineering is the discipline focused on designing, building, and maintaining scalable data infrastructure and pipelines to collect, store, process, and deliver data for analysis and decision-making.[4] This practice involves creating systems that handle large volumes of data efficiently, ensuring it is accessible and usable by downstream consumers such as analytics teams and machine learning models.[1] Key components of data engineering include data ingestion, which involves collecting raw data from diverse sources; transformation, where data is cleaned, structured, and enriched to meet specific requirements; storage in appropriate systems like databases or data lakes; and ensuring accessibility through optimized querying and delivery mechanisms.[5] Fundamental goals of data engineering encompass ensuring data quality through validation and cleansing, reliability via robust pipeline designs that minimize failures, scalability to accommodate growing data volumes using cloud and distributed systems, and efficiency in data flow to support timely insights.[4] These objectives are guided by frameworks emphasizing quality, reliability, scalability, and governance to systematically evaluate and improve data systems.[6]Importance
Data engineering is pivotal in enabling data-driven decision-making within organizations, particularly through its foundational role in business intelligence. By constructing scalable pipelines that process and deliver high-quality data in real time, it empowers real-time analytics, which allows businesses to respond swiftly to market changes and operational needs. Furthermore, data engineering facilitates the preparation and curation of datasets essential for training artificial intelligence (AI) and machine learning (ML) models, ensuring these systems operate on reliable, accessible information. This infrastructure also underpins personalized services, such as tailored customer experiences, by integrating diverse data sources to generate actionable insights at scale.[7][8][9] The economic significance of data engineering is amplified by the explosive growth of data worldwide, with projections estimating a total volume of 182 zettabytes by 2025, driven by increasing digital interactions and IoT proliferation.[10] This surge necessitates efficient data management to avoid overwhelming storage and processing costs, where data engineering intervenes by optimizing pipelines to reduce overall data expenditures by 5 to 20 percent through automation, deduplication, and resource allocation strategies.[11] Such efficiencies not only lower operational expenses but also enhance return on investment for data initiatives, positioning data engineering as a key driver of economic value in knowledge-based economies. Across industries, data engineering unlocks transformative applications by ensuring seamless data flow and integration. In finance, it supports fraud detection systems that analyze transaction data in real time to identify anomalous patterns and prevent losses, integrating disparate sources like payment logs and customer profiles for comprehensive monitoring. In healthcare, it enables patient data integration from electronic health records, wearables, and imaging systems, fostering unified views that improve diagnostics, treatment planning, and population health management. Similarly, in e-commerce, data engineering powers recommendation systems by processing user behavior, purchase history, and inventory data to deliver personalized product suggestions, thereby boosting customer engagement and sales conversion rates.[12][13][14] In the context of digital transformation, data engineering is instrumental in supporting cloud migrations and hybrid architectures, which allow organizations to blend on-premises and cloud environments for greater flexibility and scalability. This integration accelerates agility by enabling seamless data mobility across platforms, reducing latency in analytics workflows and facilitating adaptive responses to evolving business demands.[15][16]History
Early Developments
The field of data engineering traces its roots to the 1960s and 1970s, when the need for systematic data management in large-scale computing environments spurred the development of early database management systems (DBMS). One of the pioneering systems was IBM's Information Management System (IMS), introduced in 1968 as a hierarchical DBMS designed for mainframe computers, initially to support the NASA Apollo space program's inventory and data tracking requirements.[17] IMS represented a shift from file-based storage to structured data organization, enabling efficient access and updates in high-volume transaction processing, which laid foundational principles for handling enterprise data.[18] This era's innovations addressed the limitations of earlier tape and disk file systems, emphasizing data independence and hierarchical navigation to support business operations.[19] A pivotal advancement came in 1970 with Edgar F. Codd's proposal of the relational model, which revolutionized data storage by organizing information into tables with rows and columns connected via keys, rather than rigid hierarchies.[20] Published in the Communications of the ACM, Codd's model emphasized mathematical relations and normalization to reduce redundancy and ensure data integrity, influencing the design of future DBMS.[21] Building on this, in 1974, IBM researchers Donald D. Chamberlin and Raymond F. Boyce developed SEQUEL (later renamed SQL), a structured query language for relational databases that allowed users to retrieve and manipulate data using declarative English-like statements. SQL's introduction simplified data access for non-programmers, becoming essential for business reporting.[22] Concurrently, in mainframe environments during the 1970s and 1980s, rudimentary ETL (Extract, Transform, Load) concepts emerged through batch processing jobs that pulled data from disparate sources, applied transformations for consistency, and loaded it into centralized repositories for analytical reporting.[23] These processes, often implemented in COBOL on systems like IMS, supported decision-making in industries such as finance and manufacturing by consolidating transactional data.[24] In the 1980s, data engineering benefited from broader software engineering principles, particularly modularity, which promoted breaking complex data systems into independent, reusable components to enhance maintainability and scalability.[25] This approach was facilitated by the rise of Computer-Aided Software Engineering (CASE) tools, first conceptualized in the early 1980s and widely adopted by the late decade, which automated aspects of database design, modeling, and code generation for data handling tasks.[26] CASE tools, such as those for entity-relationship diagramming, integrated modularity with data flow analysis, allowing engineers to manage growing volumes of structured data more effectively in enterprise settings.[27] By the 1990s, the transition to client-server architectures marked a significant evolution, distributing data processing across networked systems where clients requested data from centralized servers, reducing mainframe dependency and enabling collaborative access.[28] This paradigm, popularized with the advent of personal computers and local area networks, supported early forms of distributed querying and data sharing, setting the stage for more scalable engineering practices while still focusing on structured data environments.[29]Big Data Era and Modern Evolution
The big data era emerged in the 2000s as organizations grappled with exponentially growing volumes of data that exceeded the capabilities of traditional relational databases. In 2006, Yahoo developed Hadoop, an open-source framework for distributed storage and processing, building on Google's MapReduce paradigm introduced in a 2004 research paper.[30] MapReduce enabled parallel processing of large datasets across clusters of inexpensive hardware, facilitating fault-tolerant handling of petabyte-scale data. This innovation addressed key challenges in scalability and cost, laying the foundation for modern distributed computing in data engineering. Complementing Hadoop, NoSQL databases gained traction to manage unstructured and semi-structured data varieties. MongoDB, launched in 2009, offered a flexible, document-based model that supported dynamic schemas and horizontal scaling, rapidly becoming integral to big data ecosystems.[31] The 2010s brought refinements in processing efficiency and real-time capabilities, propelled by the maturation of cloud infrastructure. Apache Spark achieved top-level Apache project status in 2014, introducing in-memory computation to dramatically reduce latency compared to Hadoop's disk I/O reliance, enabling faster iterative algorithms for analytics and machine learning.[32] Apache Kafka, initially created at LinkedIn in 2011 and open-sourced shortly thereafter, established a robust platform for stream processing, supporting high-throughput ingestion and distribution of real-time event data with durability guarantees.[33] Cloud storage solutions scaled accordingly; AWS Simple Storage Service (S3), introduced in 2006, saw widespread adoption in the 2010s for its elastic, durable object storage, underpinning cost-effective data lakes and pipelines that handled exabyte-level growth.[34][35] Concurrently, the role of the data engineer emerged as a distinct profession in the early 2010s, driven by the need for specialized skills in managing big data infrastructures.[36] In the 2020s, data engineering evolved toward seamless integration with artificial intelligence and operational efficiency. The incorporation of AI/ML operations (MLOps) automated model training, deployment, and monitoring within data pipelines, bridging development and production environments for continuous intelligence.[37] Serverless architectures, exemplified by AWS Lambda's application to data tasks since its 2014 launch, enabled on-demand execution of ETL jobs and event-driven workflows without provisioning servers, reducing overhead in dynamic environments.[38] The data mesh paradigm, first articulated by Zhamak Dehghani in 2019, advocated for domain-oriented, decentralized data products to foster interoperability and ownership, countering monolithic architectures in enterprise settings.[39] Regulatory and security milestones further influenced the field. The European Union's General Data Protection Regulation (GDPR), enforced from May 2018, mandated robust data governance frameworks, including privacy-by-design principles and accountability measures that reshaped global data handling practices.[40] By 2025, trends emphasize resilience against emerging threats, with efforts to integrate quantum-resistant encryption algorithms—standardized by NIST in 2024—into data pipelines to protect against quantum decryption risks.[41]Core Concepts
Data Pipelines
Data pipelines form the foundational architecture in data engineering, enabling the systematic movement, processing, and storage of data from diverse sources to downstream systems for analysis and decision-making.[42] At their core, these pipelines consist of interconnected stages that ensure data flows reliably and efficiently, typically encompassing ingestion, transformation, and loading.[43] Ingestion involves capturing data from sources such as databases, APIs, or sensors, which can occur in batch mode for periodic collection of large volumes or streaming mode for continuous real-time intake.[44] The transformation stage follows, where data undergoes cleaning to remove inconsistencies, normalization, aggregation for summarization, and enrichment to add context, preparing it for usability.[42] Finally, loading delivers the processed data into target storage systems like data lakes or warehouses, ensuring accessibility for querying and analytics.[43] In environments with scarce APIs, such as for certain public financial data sources, web scraping serves as an effective ingestion method. Python libraries like BeautifulSoup and Scrapy enable extraction of structured data from websites. Supplementary data can be incorporated via available open APIs. The ingested data is typically stored in databases such as PostgreSQL augmented with the TimescaleDB extension, which optimizes handling of time-series data common in financial applications. Compliance with rate limits and terms of service is essential to ensure legal and ethical data acquisition.[45][46][47][48] Data pipelines are categorized into batch and streaming types based on processing paradigms. Batch pipelines process fixed datasets at scheduled intervals, ideal for non-time-sensitive tasks like daily reports, handling terabytes of historical data efficiently.[49] In contrast, streaming pipelines handle unbounded, continuous data flows in real-time, enabling immediate insights such as fraud detection, often using frameworks like Apache Flink for low-latency event processing.[50] This distinction allows data engineers to select architectures suited to workload demands, with streaming supporting applications requiring sub-second responsiveness.[44] Effective data pipeline design adheres to key principles that ensure robustness at scale. Idempotency guarantees that re-executing a pipeline with the same inputs produces identical outputs without duplication or errors, facilitating safe retries in distributed environments.[51] Fault tolerance incorporates mechanisms like checkpointing and error handling to recover from failures without data loss, maintaining pipeline integrity during hardware issues or network disruptions.[52] Scalability is achieved through horizontal scaling, where additional nodes or resources are added to process petabyte-scale datasets, distributing workloads across clusters for linear performance gains.[53] These principles collectively enable pipelines to support growing data volumes and varying velocities in production systems.[52] Success in data pipelines is evaluated through critical metrics that quantify operational health. Throughput measures the volume of data processed per unit time, such as records per second, indicating capacity to handle workload demands.[54] Latency tracks the end-to-end time from data ingestion to availability, essential for time-sensitive applications where delays can impact outcomes.[55] Reliability is assessed via uptime, targeting high availability like 99.9% to minimize disruptions and ensure consistent data delivery.[56] Monitoring these metrics allows engineers to optimize pipelines for efficiency and dependability.[54]ETL and ELT Processes
Extract, Transform, Load (ETL) is a data integration process that collects raw data from various sources, applies transformations to prepare it for analysis, and loads it into a target repository such as a data warehouse.[57] The workflow begins with the extract phase, where data is copied from heterogeneous sources—including databases, APIs, and flat files—into a temporary staging area to avoid impacting source systems.[57] In the transform phase, data undergoes cleaning and structuring operations, such as joining disparate datasets, filtering irrelevant records, deduplication, format standardization, and aggregation, often in the staging area to ensure quality before final storage.[58] The load phase then transfers the refined data into the target system, using methods like full loads for initial population or incremental loads for ongoing updates.[57] This approach is particularly suitable for on-premises environments with limited storage capacity in the target system, as transformations reduce data volume prior to loading.[59] Extract, Load, Transform (ELT) reverses the transformation timing in the ETL process, loading raw data directly into the target system first and performing transformations afterward within that system's compute environment.[60] During the extract phase, unchanged raw data is pulled from sources and immediately loaded into scalable storage like a cloud data warehouse.[61] Transformations—such as joining, filtering, and aggregation—occur post-load, leveraging the target's processing power for efficiency.[61] Platforms like Snowflake exemplify ELT by enabling in-warehouse transformations on large datasets, offering advantages in scalability for big data scenarios where raw data volumes exceed traditional staging limits.[62] Both ETL and ELT incorporate tools-agnostic steps to ensure reliability and efficiency. Data validation rules, including schema enforcement to verify structural consistency and business logic checks for data integrity, are applied during extraction or transformation to reject non-compliant records early.[63] Error handling mechanisms, such as automated retry logic for transient failures like network issues, prevent full pipeline halts and log exceptions for auditing.[64] Performance optimization often involves parallel processing, where extraction, transformation, or loading tasks are distributed across multiple nodes to reduce latency and handle high-volume data flows.[65] Choosing between ETL and ELT depends on organizational needs: ETL is preferred in compliance-heavy environments requiring rigorous pre-load validation and cleansing to meet regulatory standards like GDPR or HIPAA.[66] Conversely, ELT suits analytics-focused setups with access to powerful cloud compute resources, allowing flexible, on-demand transformations for rapid insights on vast datasets.[62]Tools and Technologies
Compute and Processing
In data engineering, compute and processing refer to the frameworks and platforms that execute data transformations, analytics, and computations at scale, handling vast volumes of structured and unstructured data efficiently across distributed systems. These systems support both batch-oriented workloads, where data is processed in discrete chunks, and streaming workloads, where data arrives continuously in real time. Key frameworks emphasize fault tolerance, scalability, and integration with various data sources to enable reliable processing pipelines. Batch processing is a foundational paradigm in data engineering, enabling the handling of large, static datasets through distributed computing. Apache Spark serves as a prominent open-source framework for this purpose, providing an in-memory computation engine that distributes data across clusters for parallel processing. Spark supports high-level APIs for SQL queries via Spark SQL, allowing declarative data manipulation on petabyte-scale datasets, and includes MLlib, a scalable machine learning library for tasks like feature extraction, classification, and clustering on distributed data. By processing data in resilient distributed datasets (RDDs) or structured DataFrames, Spark achieves up to 100x faster performance than traditional disk-based systems like Hadoop MapReduce for iterative algorithms.[67] Stream processing complements batch methods by enabling real-time analysis of unbounded data flows, such as sensor logs or user interactions. Apache Kafka Streams is a client-side library built on Apache Kafka that processes event streams with low latency, treating input data as infinite sequences for transformations like filtering, joining, and aggregation. It incorporates windowing to group events into time-based or count-based segments for computations, such as tumbling windows that aggregate every 30 seconds, and state management to store and update keyed data persistently across processing nodes, ensuring fault-tolerant operations. Apache Flink, another leading framework, extends stream processing with native support for stateful computations over both bounded and unbounded streams, using checkpoints for exactly-once processing guarantees and state backends like RocksDB for efficient local storage and recovery. Flink's event-time processing handles out-of-order arrivals accurately, making it suitable for applications requiring sub-second latency.[68][69][50][70] Cloud-based compute options simplify deployment by managing infrastructure for these frameworks. AWS Elastic MapReduce (EMR) offers fully managed Spark clusters that auto-scale based on workload demands, integrating seamlessly with other AWS services for hybrid batch-streaming jobs. Google Cloud Dataproc provides similar managed environments for Spark and Flink, enabling rapid cluster creation in minutes with built-in autoscaling and ephemeral clusters to minimize idle costs. Databricks offers a unified platform for Apache Spark-based processing across multiple clouds, supporting scalable compute with autoscaling and integration for batch and real-time data engineering workflows.[71] For serverless architectures, AWS Glue delivers on-demand ETL processing without cluster provisioning, automatically allocating resources for Spark-based jobs and scaling to handle terabytes of data per run. These platforms often pair with distributed storage systems for input-output efficiency, though processing logic remains independent.[72][73][74][75] Optimizing compute performance is critical in data engineering to balance speed, cost, and reliability. Resource allocation involves tuning CPU cores and memory per executor in frameworks like Spark to match workload intensity, with GPU acceleration available for compute-heavy tasks such as deep learning integrations via libraries like RAPIDS. Cloud providers employ pay-per-use cost models, charging based on instance hours or data processed— for instance, AWS EMR bills per second of cluster runtime—allowing dynamic scaling to avoid over-provisioning. Key optimization techniques include data partitioning, which divides datasets into smaller chunks by keys like date or region to enable parallel execution and reduce shuffle overhead, potentially cutting job times by 50% or more in large-scale queries. Additional strategies, such as broadcast joins for small datasets and predicate pushdown, further minimize data movement across nodes.[76][77]Storage Systems
In data engineering, storage systems are essential for persisting data at rest, ensuring durability, accessibility, and performance tailored to diverse workloads such as transactional processing and analytical queries. These systems vary in structure, from row-oriented databases for operational data to columnar formats optimized for aggregation, allowing engineers to select paradigms that align with data volume, schema rigidity, and query patterns. Key considerations include scalability for petabyte-scale datasets, cost-efficiency in cloud environments, and integration with extraction, transformation, and loading (ETL) processes for data ingestion. Relational databases form a foundational storage paradigm for structured data in data engineering workflows, employing SQL for querying and maintaining data integrity through ACID (Atomicity, Consistency, Isolation, Durability) properties. Systems like PostgreSQL, an open-source object-relational database management system, support ACID transactions to ensure reliable updates even in concurrent environments, preventing partial commits or data inconsistencies. Additionally, PostgreSQL utilizes indexing mechanisms, such as B-tree and hash indexes, to accelerate query retrieval by organizing data for efficient lookups on columns like primary keys or frequently filtered attributes.[78] An extension like TimescaleDB enhances PostgreSQL for time-series data, making it suitable for ingestion in API-scarce environments, such as financial terminals relying on web scraping for public data sources, while ensuring compliance with rate limits and terms of service through robust data pipelines.[47] This row-oriented storage excels in scenarios requiring frequent reads and writes, such as real-time operational analytics, though it may incur higher costs for very large-scale aggregations compared to specialized analytical stores. Data warehouses represent purpose-built OLAP (Online Analytical Processing) systems designed for complex analytical queries on large, historical datasets in data engineering pipelines. Amazon Redshift, a fully managed petabyte-scale data warehouse service, leverages columnar storage to store data by columns rather than rows, which minimizes disk I/O and enhances compression for aggregation-heavy operations like sum or average calculations across billions of records.[79] This architecture supports massive parallel processing, enabling sub-second query responses on terabytes of data for business intelligence tasks, while automating tasks like vacuuming and distribution key management to maintain performance. Google BigQuery, a serverless data warehouse on Google Cloud Platform, employs columnar storage and decouples storage from compute for petabyte-scale analysis, with automatic scaling for efficient querying in data pipelines.[80] Snowflake, a multi-cloud data platform, separates storage and compute to enable scalable data warehousing, supporting ETL/ELT processes with near-zero maintenance across clouds.[81][82] Data lakes provide a flexible, schema-on-read storage solution for raw and unstructured data in data engineering, accommodating diverse formats without upfront schema enforcement to support exploratory analysis. Delta Lake, an open-source storage layer built on Apache Parquet files and often deployed on Amazon S3, enables ACID transactions on object storage, allowing reliable ingestion of semi-structured data like JSON logs or images alongside structured Parquet datasets.[83] By applying schema enforcement and time travel features at read time, Delta Lake mitigates issues like data corruption in lakes holding exabytes of heterogeneous data from IoT sensors or web streams, fostering a unified platform for machine learning and analytics.[84] Distributed file systems and object storage offer scalable alternatives for big data persistence in data engineering, balancing cost, durability, and access latency. The Hadoop Distributed File System (HDFS) provides fault-tolerant, block-based storage across clusters, ideal for high-throughput workloads in on-premises environments where data locality to compute nodes reduces network overhead. In contrast, object storage like Amazon S3 achieves near-infinite scalability for cloud-native setups, storing unstructured files durably with 99.999999999% availability, though it trades faster sequential reads for lower costs—often 5-10 times cheaper than HDFS per gigabyte[85]—making it preferable for archival or infrequently accessed data. Engineers must weigh these trade-offs, as S3's eventual consistency model can introduce slight delays in write-heavy scenarios compared to HDFS's immediate visibility.[86]Orchestration and Workflow Management
Orchestration and workflow management in data engineering involve tools that automate the scheduling, execution, and oversight of complex data pipelines, ensuring dependencies are handled efficiently and failures are managed proactively. Apache Airflow serves as a foundational open-source platform for this purpose, allowing users to define workflows as Directed Acyclic Graphs (DAGs) in Python code, where tasks represent individual operations and dependencies are explicitly modeled to dictate execution order.[87] For instance, dependencies can be set using operators liketask1 >> task2, ensuring task2 runs only after task1 completes successfully, which supports scalable batch-oriented processing across distributed environments.[88]
Modern alternatives to Airflow emphasize asset-oriented approaches, shifting focus from task-centric orchestration to data assets such as tables or models, which enhances observability and maintainability. Dagster, for example, models pipelines around software-defined assets, enabling automatic lineage tracking across transformations and built-in testing at development stages rather than solely in production, thereby reducing debugging time in complex workflows.[89] Similarly, Prefect provides a Python-native orchestration engine that supports dynamic flows with conditional logic and event-driven triggers, offering greater flexibility than rigid DAG structures while maintaining reproducibility through state tracking and caching mechanisms.[90]
Monitoring features in these tools are essential for maintaining pipeline reliability, including real-time alerting on failures, comprehensive logging, and visual representations of data flows. Airflow's web-based UI includes Graph and Grid views for visualizing DAG status and task runs, with logs accessible for failed instances and support for custom callbacks to alert on completion states, helping enforce service level agreements (SLAs) for uptime through operational oversight.[87] Dagster integrates lineage visualization and freshness checks directly into its asset catalog, allowing teams to monitor data quality and dependencies end-to-end without additional tooling.[91] Prefect enhances this with a modern UI for dependency graphs, real-time logging, and automations for failure alerts, enabling rapid recovery and observability in dynamic environments.[90]
Integration with continuous integration/continuous deployment (CI/CD) pipelines further bolsters orchestration by facilitating automated deployment and versioning for reproducible workflows. Airflow DAGs can be synchronized and deployed via CI/CD tools like GitHub Actions, where code changes trigger testing and updates to production environments, ensuring version control aligns with infrastructure changes.[92] Dagster supports CI/CD through Git-based automation for asset definitions, promoting reproducibility by versioning code alongside data lineage.[93] Prefect extends this with built-in deployment versioning, allowing rollbacks to prior states without manual Git edits, which integrates seamlessly with GitHub Actions for end-to-end pipeline automation.[94] These integrations align orchestration with the deployment phase of the data engineering lifecycle, minimizing manual interventions.