Hubbry Logo
logo
Big data avatar
Big data
Community hub

Big data

logo
0 subscribers
Read side by side
Big data
View on Wikipedia
from Wikipedia

Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate.[1]

A diagram of the generation and common application of big data.

Big data analysis challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy, and data source. Big data was originally associated with three key concepts: volume, variety, and velocity.[2] The analysis of big data presents challenges in sampling, and thus previously allowing for only observations and sampling. Thus a fourth concept, veracity, refers to the quality or insightfulness of the data.[3] Without sufficient investment in expertise for big data veracity, the volume and variety of data can produce costs and risks that exceed an organization's capacity to create and capture value from big data.[4]

Current usage of the term big data tends to refer to the use of predictive analytics, user behavior analytics, or certain other advanced data analytics methods that extract value from big data, and seldom to a particular size of data set. "There is little doubt that the quantities of data now available are indeed large, but that's not the most relevant characteristic of this new data ecosystem."[5] Analysis of data sets can find new correlations to "spot business trends, prevent diseases, combat crime and so on".[6] Scientists, business executives, medical practitioners, advertising and governments alike regularly meet difficulties with large data-sets in areas including Internet searches, fintech, healthcare analytics, geographic information systems, urban informatics, and business informatics. Scientists encounter limitations in e-Science work, including meteorology, genomics,[7] connectomics, complex physics simulations, biology, and environmental research.[8]

The size and number of available data sets have grown rapidly as data is collected by devices such as mobile devices, cheap and numerous information-sensing Internet of things devices, aerial (remote sensing) equipment, software logs, cameras, microphones, radio-frequency identification (RFID) readers and wireless sensor networks.[9][10] The world's technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s;[11] as of 2012, every day 2.5 exabytes (2.17×260 bytes) of data are generated.[12] Based on an IDC report prediction, the global data volume was predicted to grow exponentially from 4.4 zettabytes to 44 zettabytes between 2013 and 2020. By 2025, IDC predicts there will be 163 zettabytes of data.[13] According to IDC, global spending on big data and business analytics (BDA) solutions is estimated to reach $215.7 billion in 2021.[14][15] Statista reported that the global big data market is forecasted to grow to $103 billion by 2027.[16] In 2011 McKinsey & Company reported, if US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year.[17] In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data.[17] And users of services enabled by personal-location data could capture $600 billion in consumer surplus.[17] One question for large enterprises is determining who should own big-data initiatives that affect the entire organization.[18]

Relational database management systems and desktop statistical software packages used to visualize data often have difficulty processing and analyzing big data. The processing and analysis of big data may require "massively parallel software running on tens, hundreds, or even thousands of servers".[19] What qualifies as "big data" varies depending on the capabilities of those analyzing it and their tools. Furthermore, expanding capabilities make big data a moving target. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."[20]

Definition

[edit]

The term big data has been in use since the 1990s, with some giving credit to John Mashey for popularizing the term.[21][22] Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time.[23][page needed] Big data philosophy encompasses unstructured, semi-structured and structured data; however, the main focus is on unstructured data.[24] Big data "size" is a constantly moving target; as of 2012 ranging from a few dozen terabytes to many zettabytes of data.[25] Big data requires a set of techniques and technologies with new forms of integration to reveal insights from data-sets that are diverse, complex, and of a massive scale.[26] Variability is often included as an additional quality of big data.

A 2018 definition states "Big data is where parallel computing tools are needed to handle data", and notes, "This represents a distinct and clearly defined change in the computer science used, via parallel programming theories, and losses of some of the guarantees and capabilities made by Codd's relational model."[27]

In a comparative study of big datasets, Kitchin and McArdle found that none of the commonly considered characteristics of big data appear consistently across all of the analyzed cases.[28] For this reason, other studies identified the redefinition of power dynamics in knowledge discovery as the defining trait.[29] Instead of focusing on the intrinsic characteristics of big data, this alternative perspective pushes forward a relational understanding of the object claiming that what matters is the way in which data is collected, stored, made available and analyzed.

Big data vs. business intelligence

[edit]

The growing maturity of the concept more starkly delineates the difference between "big data" and "business intelligence":[30]

Characteristics

[edit]
This image shows the growth of big data's primary characteristics of volume, velocity, and variety.

Big data can be described by the following characteristics:

Volume
The quantity of generated and stored data. The size of the data determines the value and potential insight, and whether it can be considered big data or not. The size of big data is usually larger than terabytes and petabytes.[34]
Variety
The type and nature of the data. Earlier technologies like RDBMSs were capable to handle structured data efficiently and effectively. However, the change in type and nature from structured to semi-structured or unstructured challenged the existing tools and technologies. Big data technologies evolved with the prime intention to capture, store, and process the semi-structured and unstructured (variety) data generated with high speed (velocity), and huge in size (volume). Later, these tools and technologies were explored and used for handling structured data also but preferable for storage. Eventually, the processing of structured data was still kept as optional, either using big data or traditional RDBMSs. This helps in analyzing data towards effective usage of the hidden insights exposed from the data collected via social media, log files, sensors, etc. Big data draws from text, images, audio, video; plus it completes missing pieces through data fusion.
Velocity
The speed at which the data is generated and processed to meet the demands and challenges that lie in the path of growth and development. Big data is often available in real-time. Compared to small data, big data is produced more continually. Two kinds of velocity related to big data are the frequency of generation and the frequency of handling, recording, and publishing.[35]
Veracity
The truthfulness or reliability of the data, which refers to the data quality and the data value.[36] Big data must not only be large in size, but also must be reliable in order to achieve value in the analysis of it. The data quality of captured data can vary greatly, affecting an accurate analysis.[37]
Value
The worth in information that can be achieved by the processing and analysis of large datasets. Value also can be measured by an assessment of the other qualities of big data.[38] Value may also represent the profitability of information that is retrieved from the analysis of big data.
Variability
The characteristic of the changing formats, structure, or sources of big data. Big data can include structured, unstructured, or combinations of structured and unstructured data. Big data analysis may integrate raw data from multiple sources. The processing of raw data may also involve transformations of unstructured data to structured data.

Other possible characteristics of big data are:[39]

Exhaustive
Whether the entire system (i.e., =all) is captured or recorded or not. Big data may or may not include all the available data from sources.
Fine-grained and uniquely lexical
Respectively, the proportion of specific data of each element per element collected and if the element and its characteristics are properly indexed or identified.
Relational
If the data collected contains common fields that would enable a conjoining, or meta-analysis, of different data sets.
Extensional
If new fields in each element of the data collected can be added or changed easily.
Scalability
If the size of the big data storage system can expand rapidly.

Architecture

[edit]

Big data repositories have existed in many forms, often built by corporations with a special need. Commercial vendors historically offered parallel database management systems for big data beginning in the 1990s. For many years, WinterCorp published the largest database report.[40][promotional source?]

Teradata Corporation in 1984 marketed the parallel processing DBC 1012 system. Teradata systems were the first to store and analyze 1 terabyte of data in 1992. Hard disk drives were 2.5 GB in 1991 so the definition of big data continuously evolves. Teradata installed the first petabyte class RDBMS based system in 2007. As of 2017, there are a few dozen petabyte class Teradata relational databases installed, the largest of which exceeds 50 PB. Systems up until 2008 were 100% structured relational data. Since then, Teradata has added semi structured data types including XML, JSON, and Avro.

In 2000, Seisint Inc. (now LexisNexis Risk Solutions) developed a C++-based distributed platform for data processing and querying known as the HPCC Systems platform. This system automatically partitions, distributes, stores and delivers structured, semi-structured, and unstructured data across multiple commodity servers. Users can write data processing pipelines and queries in a declarative dataflow programming language called ECL. Data analysts working in ECL are not required to define data schemas upfront and can rather focus on the particular problem at hand, reshaping data in the best possible manner as they develop the solution. In 2004, LexisNexis acquired Seisint Inc.[41] and their high-speed parallel processing platform and successfully used this platform to integrate the data systems of Choicepoint Inc. when they acquired that company in 2008.[42] In 2011, the HPCC systems platform was open-sourced under the Apache v2.0 License.

CERN and other physics experiments have collected big data sets for many decades, usually analyzed via high-throughput computing rather than the map-reduce architectures usually meant by the current "big data" movement.

In 2004, Google published a paper on a process called MapReduce that uses a similar architecture. The MapReduce concept provides a parallel processing model, and an associated implementation was released to process huge amounts of data. With MapReduce, queries are split and distributed across parallel nodes and processed in parallel (the "map" step). The results are then gathered and delivered (the "reduce" step). The framework was very successful,[43] so others wanted to replicate the algorithm. Therefore, an implementation of the MapReduce framework was adopted by an Apache open-source project named "Hadoop".[44] Apache Spark was developed in 2012 in response to limitations in the MapReduce paradigm, as it adds in-memory processing and the ability to set up many operations (not just map followed by reducing).

MIKE2.0 is an open approach to information management that acknowledges the need for revisions due to big data implications identified in an article titled "Big Data Solution Offering".[45] The methodology addresses handling big data in terms of useful permutations of data sources, complexity in interrelationships, and difficulty in deleting (or modifying) individual records.[46]

Studies in 2012 showed that a multiple-layer architecture was one option to address the issues that big data presents. A distributed parallel architecture distributes data across multiple servers; these parallel execution environments can dramatically improve data processing speeds. This type of architecture inserts data into a parallel DBMS, which implements the use of MapReduce and Hadoop frameworks. This type of framework looks to make the processing power transparent to the end-user by using a front-end application server.[47]

The data lake allows an organization to shift its focus from centralized control to a shared model to respond to the changing dynamics of information management. This enables quick segregation of data into the data lake, thereby reducing the overhead time.[48][49]

Technologies

[edit]

A 2011 McKinsey Global Institute report characterizes the main components and ecosystem of big data as follows:[50]

Multidimensional big data can also be represented as OLAP data cubes or, mathematically, tensors. Array database systems have set out to provide storage and high-level query support on this data type. Additional technologies being applied to big data include efficient tensor-based computation,[51] such as multilinear subspace learning,[52] massively parallel-processing (MPP) databases, search-based applications, data mining,[53] distributed file systems, distributed cache (e.g., burst buffer and Memcached), distributed databases, cloud and HPC-based infrastructure (applications, storage and computing resources),[54] and the Internet.[citation needed] Although, many approaches and technologies have been developed, it still remains difficult to carry out machine learning with big data.[55]

Some MPP relational databases have the ability to store and manage petabytes of data. Implicit is the ability to load, monitor, back up, and optimize the use of the large data tables in the RDBMS.[56][promotional source?]

DARPA's Topological Data Analysis program seeks the fundamental structure of massive data sets and in 2008 the technology went public with the launch of a company called "Ayasdi".[57][independent source needed]

The practitioners of big data analytics processes are generally hostile to slower shared storage,[58] preferring direct-attached storage (DAS) in its various forms from solid state drive (SSD) to high capacity SATA disk buried inside parallel processing nodes. The perception of shared storage architectures—storage area network (SAN) and network-attached storage (NAS)— is that they are relatively slow, complex, and expensive. These qualities are not consistent with big data analytics systems that thrive on system performance, commodity infrastructure, and low cost.

Real or near-real-time information delivery is one of the defining characteristics of big data analytics. Latency is therefore avoided whenever and wherever possible. Data in direct-attached memory or disk is good—data on memory or disk at the other end of an FC SAN connection is not. The cost of an SAN at the scale needed for analytics applications is much higher than other storage techniques.

Applications

[edit]
A graph showing the number of data points used to train notable AI systems from 1950 to 2025.[59]

Big data has increased the demand of information management specialists so much so that Software AG, Oracle Corporation, IBM, Microsoft, SAP, EMC, HP, and Dell have spent more than $15 billion on software firms specializing in data management and analytics. In 2010, this industry was worth more than $100 billion and was growing at almost 10 percent a year, about twice as fast as the software business as a whole.[6]

Developed economies increasingly use data-intensive technologies. There are 4.6 billion mobile-phone subscriptions worldwide, and between 1 billion and 2 billion people accessing the internet.[6] Between 1990 and 2005, more than 1 billion people worldwide entered the middle class, which means more people became more literate, which in turn led to information growth. The world's effective capacity to exchange information through telecommunication networks was 281 petabytes in 1986, 471 petabytes in 1993, 2.2 exabytes in 2000, 65 exabytes in 2007[11] and predictions put the amount of internet traffic at 667 exabytes annually by 2014.[6] According to one estimate, one-third of the globally stored information is in the form of alphanumeric text and still image data,[60] which is the format most useful for most big data applications. This also shows the potential of yet unused data (i.e. in the form of video and audio content).

While many vendors offer off-the-shelf products for big data, experts promote the development of in-house custom-tailored systems if the company has sufficient technical capabilities.[61]

Government

[edit]
See also: Government by algorithm

The use and adoption of big data within governmental processes allows efficiencies in terms of cost, productivity, and innovation,[62] but comes with flaws. Data analysis often requires multiple parts of government (central and local) to work in collaboration and create new and innovative processes to deliver the desired outcome. A common government organization that makes use of big data is the National Security Administration (NSA), which monitors the activities of the Internet constantly in search for potential patterns of suspicious or illegal activities their system may pick up.

Civil registration and vital statistics (CRVS) collects all certificates status from birth to death. CRVS is a source of big data for governments.

International development

[edit]

Research on the effective usage of information and communication technologies for development (also known as "ICT4D") suggests that big data technology can make important contributions but also present unique challenges to international development.[63][64] Advancements in big data analysis offer cost-effective opportunities to improve decision-making in critical development areas such as health care, employment, economic productivity, crime, security, and natural disaster and resource management.[65][page needed][66][67] Additionally, user-generated data offers new opportunities to give the unheard a voice.[68] However, longstanding challenges for developing regions such as inadequate technological infrastructure and economic and human resource scarcity exacerbate existing concerns with big data such as privacy, imperfect methodology, and interoperability issues.[65][page needed] The challenge of "big data for development"[65][page needed] is currently evolving toward the application of this data through machine learning, known as "artificial intelligence for development (AI4D).[69]

Benefits

[edit]

A major practical application of big data for development has been "fighting poverty with data".[70] In 2015, Blumenstock and colleagues estimated predicted poverty and wealth from mobile phone metadata[71] and in 2016 Jean and colleagues combined satellite imagery and machine learning to predict poverty.[72] Using digital trace data to study the labor market and the digital economy in Latin America, Hilbert and colleagues [73][74] argue that digital trace data has several benefits such as:

  • Thematic coverage: including areas that were previously difficult or impossible to measure
  • Geographical coverage: providing sizable and comparable data for almost all countries, including many small countries that usually are not included in international inventories
  • Level of detail: providing fine-grained data with many interrelated variables, and new aspects, like network connections
  • Timeliness and timeseries: graphs can be produced within days of being collected

Challenges

[edit]

At the same time, working with digital trace data instead of traditional survey data does not eliminate the traditional challenges involved when working in the field of international quantitative analysis. Priorities change, but the basic discussions remain the same. Among the main challenges are:

  • Representativeness. While traditional development statistics is mainly concerned with the representativeness of random survey samples, digital trace data is never a random sample.[75]
  • Generalizability. While observational data always represents this source very well, it only represents what it represents, and nothing more. While it is tempting to generalize from specific observations of one platform to broader settings, this is often very deceptive.
  • Harmonization. Digital trace data still requires international harmonization of indicators. It adds the challenge of so-called "data-fusion", the harmonization of different sources.
  • Data overload. Analysts and institutions are not used to effectively deal with a large number of variables, which is efficiently done with interactive dashboards. Practitioners still lack a standard workflow that would allow researchers, users and policymakers to efficiently and effectively deal with data.[73]

Finance

[edit]

Big Data is being rapidly adopted in Finance to 1) speed up processing and 2) deliver better, more informed inferences, both internally and to the clients of the financial institutions.[76] The financial applications of Big Data range from investing decisions and trading (processing volumes of available price data, limit order books, economic data and more, all at the same time), portfolio management (optimizing over an increasingly large array of financial instruments, potentially selected from different asset classes), risk management (credit rating based on extended information), and any other aspect where the data inputs are large.[77] Big Data has also been a typical concept within the field of alternative financial service. Some of the major areas involve crowd-funding platforms and crypto currency exchanges.[78]

Healthcare

[edit]

Big data analytics has been used in healthcare in providing personalized medicine and prescriptive analytics, clinical risk intervention and predictive analytics, waste and care variability reduction, automated external and internal reporting of patient data, standardized medical terms and patient registries.[79][80][81][82] Some areas of improvement are more aspirational than actually implemented. The level of data generated within healthcare systems is not trivial. With the added adoption of mHealth, eHealth and wearable technologies the volume of data will continue to increase. This includes electronic health record data, imaging data, patient generated data, sensor data, and other forms of difficult to process data. There is now an even greater need for such environments to pay greater attention to data and information quality.[83] "Big data very often means 'dirty data' and the fraction of data inaccuracies increases with data volume growth." Human inspection at the big data scale is impossible and there is a desperate need in health service for intelligent tools for accuracy and believability control and handling of information missed.[84] While extensive information in healthcare is now electronic, it fits under the big data umbrella as most is unstructured and difficult to use.[85] The use of big data in healthcare has raised significant ethical challenges ranging from risks for individual rights, privacy and autonomy, to transparency and trust.[86]

Big data in health research is particularly promising in terms of exploratory biomedical research, as data-driven analysis can move forward more quickly than hypothesis-driven research.[87] Then, trends seen in data analysis can be tested in traditional, hypothesis-driven follow up biological research and eventually clinical research.

A related application sub-area, that heavily relies on big data, within the healthcare field is that of computer-aided diagnosis in medicine.[88][page needed] For instance, for epilepsy monitoring it is customary to create 5 to 10 GB of data daily.[89] Similarly, a single uncompressed image of breast tomosynthesis averages 450 MB of data.[90] These are just a few of the many examples where computer-aided diagnosis uses big data. For this reason, big data has been recognized as one of the seven key challenges that computer-aided diagnosis systems need to overcome in order to reach the next level of performance.[91]

Education

[edit]

A McKinsey Global Institute study found a shortage of 1.5 million highly trained data professionals and managers[50] and a number of universities[92][better source needed] including University of Tennessee and UC Berkeley, have created masters programs to meet this demand. Private boot camps have also developed programs to meet that demand, including paid programs like The Data Incubator or General Assembly.[93] In the specific field of marketing, one of the problems stressed by Wedel and Kannan[94] is that marketing has several sub domains (e.g., advertising, promotions, product development, branding) that all use different types of data.

Media

[edit]

To understand how the media uses big data, it is first necessary to provide some context into the mechanism used for media process. It has been suggested by Nick Couldry and Joseph Turow that practitioners in media and advertising approach big data as many actionable points of information about millions of individuals. The industry appears to be moving away from the traditional approach of using specific media environments such as newspapers, magazines, or television shows and instead taps into consumers with technologies that reach targeted people at optimal times in optimal locations. The ultimate aim is to serve or convey, a message or content that is (statistically speaking) in line with the consumer's mindset. For example, publishing environments are increasingly tailoring messages (advertisements) and content (articles) to appeal to consumers that have been exclusively gleaned through various data-mining activities.[95]

  • Targeting of consumers (for advertising by marketers)[96]
  • Data capture
  • Data journalism: publishers and journalists use big data tools to provide unique and innovative insights and infographics.

Channel 4, the British public-service television broadcaster, is a leader in the field of big data and data analysis.[97]

Insurance

[edit]

Health insurance providers are collecting data on social "determinants of health" such as food and TV consumption, marital status, clothing size, and purchasing habits, from which they make predictions on health costs, in order to spot health issues in their clients. It is controversial whether these predictions are currently being used for pricing.[98]

Internet of things (IoT)

[edit]
Main article: Internet of things
Further information: Edge computing

Big data and the IoT work in conjunction. Data extracted from IoT devices provides a mapping of device inter-connectivity. Such mappings have been used by the media industry, companies, and governments to more accurately target their audience and increase media efficiency. The IoT is also increasingly adopted as a means of gathering sensory data, and this sensory data has been used in medical,[99] manufacturing[100] and transportation[101] contexts.

Kevin Ashton, the digital innovation expert who is credited with coining the term,[102] defines the Internet of things in this quote: "If we had computers that knew everything there was to know about things—using data they gathered without any help from us—we would be able to track and count everything, and greatly reduce waste, loss, and cost. We would know when things needed replacing, repairing, or recalling, and whether they were fresh or past their best."

Information technology

[edit]

Especially since 2015, big data has come to prominence within business operations as a tool to help employees work more efficiently and streamline the collection and distribution of information technology (IT). The use of big data to resolve IT and data collection issues within an enterprise is called IT operations analytics (ITOA).[103] By applying big data principles into the concepts of machine intelligence and deep computing, IT departments can predict potential issues and prevent them.[103] ITOA businesses offer platforms for systems management that bring data silos together and generate insights from the whole of the system rather than from isolated pockets of data.

Survey science

[edit]

Compared to survey-based data collection, big data has low cost per data point, applies analysis techniques via machine learning and data mining, and includes diverse and new data sources, e.g., registers, social media, apps, and other forms digital data. Since 2018, survey scientists have started to examine how big data and survey science can complement each other to allow researchers and practitioners to improve the production of statistics and its quality. There have been three Big Data Meets Survey Science (BigSurv) conferences in 2018, 2020 (virtual), 2023, and as of 2023 one conference forthcoming in 2025,[104] a special issue in the Social Science Computer Review,[105] a special issue in Journal of the Royal Statistical Society,[106] and a special issue in EP J Data Science,[107] and a book called Big Data Meets Social Sciences[108] edited by Craig Hill and five other Fellows of the American Statistical Association. In 2021, the founding members of BigSurv received the Warren J. Mitofsky Innovators Award from the American Association for Public Opinion Research.[109]

Marketing

[edit]

Big data is notable in marketing due to the constant "datafication"[110] of everyday consumers of the internet, in which all forms of data are tracked. The datafication of consumers can be defined as quantifying many of or all human behaviors for the purpose of marketing.[110] The increasingly digital world of rapid datafication makes this idea relevant to marketing because the amount of data constantly grows exponentially. It is predicted to increase from 44 to 163 zettabytes within the span of five years.[111] The size of big data can often be difficult to navigate for marketers.[112] As a result, adopters of big data may find themselves at a disadvantage. Algorithmic findings can be difficult to achieve with such large datasets.[113] Big data in marketing is a highly lucrative tool that can be used for large corporations, its value being as a result of the possibility of predicting significant trends, interests, or statistical outcomes in a consumer-based manner.[114]

There are three significant factors in the use of big data in marketing:

  1. Big data provides customer behavior pattern spotting for marketers, since all human actions are being quantified into readable numbers for marketers to analyze and use for their research.[115] In addition, big data can also be seen as a customized product recommendation tool. Specifically, since big data is effective in analyzing customers' purchase behaviors and browsing patterns, this technology can assist companies in promoting specific personalized products to specific customers.[116]
  2. Real-time market responsiveness is important for marketers because of the ability to shift marketing efforts and correct to current trends, which is helpful in maintaining relevance to consumers. This can supply corporations with the information necessary to predict the wants and needs of consumers in advance.[115]
  3. Data-driven market ambidexterity are being highly fueled by big data.[115] New models and algorithms are being developed to make significant predictions about certain economic and social situations.[117]

Case studies

[edit]

Government

[edit]

China

[edit]
  • The Integrated Joint Operations Platform (IJOP, 一体化联合作战平台) is used by the government to monitor the population, particularly Uyghurs.[118] Biometrics, including DNA samples, are gathered through a program of free physicals.[119]
  • By 2020, China plans to give all its citizens a personal "social credit" score based on how they behave.[120] The Social Credit System, now being piloted in a number of Chinese cities, is considered a form of mass surveillance which uses big data analysis technology.[121][122]

India

[edit]
  • Big data analysis was tried out for the BJP to win the 2014 Indian General Election.[123]
  • The Indian government uses numerous techniques to ascertain how the Indian electorate is responding to government action, as well as ideas for policy augmentation.

Israel

[edit]
  • Personalized diabetic treatments can be created through GlucoMe's big data solution.[124]

United Kingdom

[edit]

Examples of uses of big data in public services:

  • Data on prescription drugs: by connecting origin, location and the time of each prescription, a research unit was able to exemplify and examine the considerable delay between the release of any given drug, and a UK-wide adaptation of the National Institute for Health and Care Excellence guidelines. This suggests that new or most up-to-date drugs take some time to filter through to the general patient.[citation needed][125]
  • Joining up data: a local authority blended data about services, such as road gritting rotas, with services for people at risk, such as Meals on Wheels. The connection of data allowed the local authority to avoid any weather-related delay.[126]

United States

[edit]

Retail

[edit]
  • Walmart handles more than 1 million customer transactions every hour, which are imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data—the equivalent of 167 times the information contained in all the books in the US Library of Congress.[6]
  • Windermere Real Estate uses location information from nearly 100 million drivers to help new home buyers determine their typical drive times to and from work throughout various times of the day.[136]
  • FICO Card Detection System protects accounts worldwide.[137]
  • Omnichannel retailing[138] leverages online big data to improve offline experiences.

Science

[edit]
  • The Large Hadron Collider experiments represent about 150 million sensors delivering data 40 million times per second. There are nearly 600 million collisions per second. After filtering and refraining from recording more than 99.99995%[139] of these streams, there are 1,000 collisions of interest per second.[140][141][142]
    • As a result, only working with less than 0.001% of the sensor stream data, the data flow from all four LHC experiments represents 25 petabytes annual rate before replication (as of 2012). This becomes nearly 200 petabytes after replication.
    • If all sensor data were recorded in LHC, the data flow would be extremely hard to work with. The data flow would exceed 150 million petabytes annual rate, or nearly 500 exabytes per day, before replication. To put the number in perspective, this is equivalent to 500 quintillion (5×1020) bytes per day, almost 200 times more than all the other sources combined in the world.
  • The Square Kilometre Array is a radio telescope built of thousands of antennas. It is expected to be operational by 2024. Collectively, these antennas are expected to gather 14 exabytes and store one petabyte per day.[143][144] It is considered one of the most ambitious scientific projects ever undertaken.[145]
  • When the Sloan Digital Sky Survey (SDSS) began to collect astronomical data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy previously. Continuing at a rate of about 200 GB per night, SDSS has amassed more than 140 terabytes of information.[6] When the Large Synoptic Survey Telescope, successor to SDSS, comes online in 2020, its designers expect it to acquire that amount of data every five days.[6]
  • Decoding the human genome originally took 10 years to process; now it can be achieved in less than a day. The DNA sequencers have divided the sequencing cost by 10,000 in the last ten years, which is 100 times less expensive than the reduction in cost predicted by Moore's law.[146]
  • The NASA Center for Climate Simulation (NCCS) stores 32 petabytes of climate observations and simulations on the Discover supercomputing cluster.[147][148]
  • Google's DNAStack compiles and organizes DNA samples of genetic data from around the world to identify diseases and other medical defects. These fast and exact calculations eliminate any "friction points", or human errors that could be made by one of the numerous science and biology experts working with the DNA. DNAStack, a part of Google Genomics, allows scientists to use the vast sample of resources from Google's search server to scale social experiments that would usually take years, instantly.[149][150]
  • 23andme's DNA database contains the genetic information of over 1,000,000 people worldwide.[151] The company explores selling the "anonymous aggregated genetic data" to other researchers and pharmaceutical companies for research purposes if patients give their consent.[152][153][154][155][156] Ahmad Hariri, professor of psychology and neuroscience at Duke University who has been using 23andMe in his research since 2009 states that the most important aspect of the company's new service is that it makes genetic research accessible and relatively cheap for scientists.[152] A study that identified 15 genome sites linked to depression in 23andMe's database lead to a surge in demands to access the repository with 23andMe fielding nearly 20 requests to access the depression data in the two weeks after publication of the paper.[157]
  • Computational fluid dynamics (CFD) and hydrodynamic turbulence research generate massive data sets. The Johns Hopkins Turbulence Databases (JHTDB) contains over 350 terabytes of spatiotemporal fields from Direct Numerical simulations of various turbulent flows. Such data have been difficult to share using traditional methods such as downloading flat simulation output files. The data within JHTDB can be accessed using "virtual sensors" with various access modes ranging from direct web-browser queries, access through Matlab, Python, Fortran and C programs executing on clients' platforms, to cut out services to download raw data. The data have been used in over 150 scientific publications.

Sports

[edit]

Big data can be used to improve training and understanding competitors, using sport sensors. It is also possible to predict winners in a match using big data analytics.[158] Future performance of players could be predicted as well.[159] Thus, players' value and salary is determined by data collected throughout the season.[160]

In Formula One races, race cars with hundreds of sensors generate terabytes of data. These sensors collect data points from tire pressure to fuel burn efficiency.[161] Based on the data, engineers and data analysts decide whether adjustments should be made in order to win a race. Besides, using big data, race teams try to predict the time they will finish the race beforehand, based on simulations using data collected over the season.[162]

Technology

[edit]
  • As of 2013, eBay.com uses two data warehouses at 7.5 petabytes and 40PB as well as a 40PB Hadoop cluster for search, consumer recommendations, and merchandising.[163]
  • Amazon.com handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and as of 2005 they had the world's three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.[164]
  • Facebook handles 50 billion photos from its user base.[165] As of June 2017, Facebook reached 2 billion monthly active users.[166]
  • Google was handling roughly 100 billion searches per month as of August 2012.[167]

COVID-19

[edit]

During the COVID-19 pandemic, big data was raised as a way to minimise the impact of the disease. Significant applications of big data included minimising the spread of the virus, case identification and development of medical treatment.[168]

Governments used big data to track infected people to minimise spread. Early adopters included China, Taiwan, South Korea, and Israel.[169][170][171]

Research activities

[edit]

Encrypted search and cluster formation in big data were demonstrated in March 2014 at the American Society of Engineering Education. Gautam Siwach engaged at Tackling the challenges of Big Data by MIT Computer Science and Artificial Intelligence Laboratory and Amir Esmailpour at the UNH Research Group investigated the key features of big data as the formation of clusters and their interconnections. They focused on the security of big data and the orientation of the term towards the presence of different types of data in an encrypted form at cloud interface by providing the raw definitions and real-time examples within the technology. Moreover, they proposed an approach for identifying the encoding technique to advance towards an expedited search over encrypted text leading to the security enhancements in big data.[172]

In March 2012, The White House announced a national "Big Data Initiative" that consisted of six federal departments and agencies committing more than $200 million to big data research projects.[173]

The initiative included a National Science Foundation "Expeditions in Computing" grant of $10 million over five years to the AMPLab[174] at the University of California, Berkeley.[175] The AMPLab also received funds from DARPA, and over a dozen industrial sponsors and uses big data to attack a wide range of problems from predicting traffic congestion[176] to fighting cancer.[177]

The White House Big Data Initiative also included a commitment by the Department of Energy to provide $25 million in funding over five years to establish the Scalable Data Management, Analysis and Visualization (SDAV) Institute,[178] led by the Energy Department's Lawrence Berkeley National Laboratory. The SDAV Institute aims to bring together the expertise of six national laboratories and seven universities to develop new tools to help scientists manage and visualize data on the department's supercomputers.

The U.S. state of Massachusetts announced the Massachusetts Big Data Initiative in May 2012, which provides funding from the state government and private companies to a variety of research institutions.[179] The Massachusetts Institute of Technology hosts the Intel Science and Technology Center for Big Data in the MIT Computer Science and Artificial Intelligence Laboratory, combining government, corporate, and institutional funding and research efforts.[180]

The European Commission is funding the two-year-long Big Data Public Private Forum through their Seventh Framework Program to engage companies, academics and other stakeholders in discussing big data issues. The project aims to define a strategy in terms of research and innovation to guide supporting actions from the European Commission in the successful implementation of the big data economy. Outcomes of this project will be used as input for Horizon 2020, their next framework program.[181]

The British government announced in March 2014 the founding of the Alan Turing Institute, named after the computer pioneer and code-breaker, which will focus on new ways to collect and analyze large data sets.[182]

At the University of Waterloo Stratford Campus Canadian Open Data Experience (CODE) Inspiration Day, participants demonstrated how using data visualization can increase the understanding and appeal of big data sets and communicate their story to the world.[183]

Computational social sciences – Anyone can use application programming interfaces (APIs) provided by big data holders, such as Google and Twitter, to do research in the social and behavioral sciences.[184] Often these APIs are provided for free.[184] Tobias Preis et al. used Google Trends data to demonstrate that Internet users from countries with a higher per capita gross domestic products (GDPs) are more likely to search for information about the future than information about the past. The findings suggest there may be a link between online behaviors and real-world economic indicators.[185][186][187] The authors of the study examined Google queries logs made by ratio of the volume of searches for the coming year (2011) to the volume of searches for the previous year (2009), which they call the "future orientation index".[188] They compared the future orientation index to the per capita GDP of each country, and found a strong tendency for countries where Google users inquire more about the future to have a higher GDP.

Tobias Preis and his colleagues Helen Susannah Moat and H. Eugene Stanley introduced a method to identify online precursors for stock market moves, using trading strategies based on search volume data provided by Google Trends.[189] Their analysis of Google search volume for 98 terms of varying financial relevance, published in Scientific Reports,[190] suggests that increases in search volume for financially relevant search terms tend to precede large losses in financial markets.[191][192][193][194][195][196][197]

Big data sets come with algorithmic challenges that previously did not exist. Hence, there is seen by some to be a need to fundamentally change the processing ways.[198]

Sampling big data

[edit]

A research question that is asked about big data sets is whether it is necessary to look at the full data to draw certain conclusions about the properties of the data or if is a sample is good enough. The name big data itself contains a term related to size and this is an important characteristic of big data. But sampling enables the selection of right data points from within the larger data set to estimate the characteristics of the whole population. In manufacturing different types of sensory data such as acoustics, vibration, pressure, current, voltage, and controller data are available at short time intervals. To predict downtime it may not be necessary to look at all the data but a sample may be sufficient. Big data can be broken down by various data point categories such as demographic, psychographic, behavioral, and transactional data. With large sets of data points, marketers are able to create and use more customized segments of consumers for more strategic targeting.

Critique

[edit]

Critiques of the big data paradigm come in two flavors: those that question the implications of the approach itself, and those that question the way it is currently done.[199] One approach to this criticism is the field of critical data studies.

Critiques of the big data paradigm

[edit]

"A crucial problem is that we do not know much about the underlying empirical micro-processes that lead to the emergence of the[se] typical network characteristics of Big Data."[23][page needed] In their critique, Snijders, Matzat, and Reips point out that often very strong assumptions are made about mathematical properties that may not at all reflect what is really going on at the level of micro-processes. Mark Graham has leveled broad critiques at Chris Anderson's assertion that big data will spell the end of theory:[200] focusing in particular on the notion that big data must always be contextualized in their social, economic, and political contexts.[201] Even as companies invest eight- and nine-figure sums to derive insight from information streaming in from suppliers and customers, less than 40% of employees have sufficiently mature processes and skills to do so. To overcome this insight deficit, big data, no matter how comprehensive or well analyzed, must be complemented by "big judgment", according to an article in the Harvard Business Review.[202]

Much in the same line, it has been pointed out that the decisions based on the analysis of big data are inevitably "informed by the world as it was in the past, or, at best, as it currently is".[65][page needed] Fed by a large number of data on past experiences, algorithms can predict future development if the future is similar to the past.[203] If the system's dynamics of the future change (if it is not a stationary process), the past can say little about the future. In order to make predictions in changing environments, it would be necessary to have a thorough understanding of the systems dynamic, which requires theory.[203] As a response to this critique Alemany Oliver and Vayre suggest to use "abductive reasoning as a first step in the research process in order to bring context to consumers' digital traces and make new theories emerge".[204] Additionally, it has been suggested to combine big data approaches with computer simulations, such as agent-based models[65][page needed] and complex systems. Agent-based models are increasingly getting better in predicting the outcome of social complexities of even unknown future scenarios through computer simulations that are based on a collection of mutually interdependent algorithms.[205][206] Finally, the use of multivariate methods that probe for the latent structure of the data, such as factor analysis and cluster analysis, have proven useful as analytic approaches that go well beyond the bi-variate approaches (e.g. contingency tables) typically employed with smaller data sets.

In health and biology, conventional scientific approaches are based on experimentation. For these approaches, the limiting factor is the relevant data that can confirm or refute the initial hypothesis.[207] A new postulate is accepted now in biosciences: the information provided by the data in huge volumes (omics) without prior hypothesis is complementary and sometimes necessary to conventional approaches based on experimentation.[208][209] In the massive approaches it is the formulation of a relevant hypothesis to explain the data that is the limiting factor.[210] The search logic is reversed and the limits of induction ("Glory of Science and Philosophy scandal", C. D. Broad, 1926) are to be considered.[citation needed]

Privacy advocates are concerned about the threat to privacy represented by increasing storage and integration of personally identifiable information; expert panels have released various policy recommendations to conform practice to expectations of privacy.[211] The misuse of big data in several cases by media, companies, and even the government has allowed for abolition of trust in almost every fundamental institution holding up society.[212]

Barocas and Nissenbaum argue that one way of protecting individual users is by being informed about the types of information being collected, with whom it is shared, under what constraints and for what purposes.[213]

Critiques of the "V" model

[edit]

The "V" model of big data is concerning as it centers around computational scalability and lacks in a loss around the perceptibility and understandability of information. This led to the framework of cognitive big data, which characterizes big data applications according to:[214]

  • Data completeness: understanding of the non-obvious from data
  • Data correlation, causation, and predictability: causality as not essential requirement to achieve predictability
  • Explainability and interpretability: humans desire to understand and accept what they understand, where algorithms do not cope with this
  • Level of automated decision-making: algorithms that support automated decision making and algorithmic self-learning

Critiques of novelty

[edit]

Large data sets have been analyzed by computing machines for well over a century, including the US census analytics performed by IBM's punch-card machines which computed statistics including means and variances of populations across the whole continent. In more recent decades, science experiments such as CERN have produced data on similar scales to current commercial "big data". However, science experiments have tended to analyze their data using specialized custom-built high-performance computing (super-computing) clusters and grids, rather than clouds of cheap commodity computers as in the current commercial wave, implying a difference in both culture and technology stack.

Critiques of big data execution

[edit]

Ulf-Dietrich Reips and Uwe Matzat wrote in 2014 that big data had become a "fad" in scientific research.[184] Researcher danah boyd has raised concerns about the use of big data in science neglecting principles such as choosing a representative sample by being too concerned about handling the huge amounts of data.[215] This approach may lead to results that have a bias in one way or another.[216] Integration across heterogeneous data resources—some that might be considered big data and others not—presents formidable logistical as well as analytical challenges, but many researchers argue that such integrations are likely to represent the most promising new frontiers in science.[217] In the provocative article "Critical Questions for Big Data",[218] the authors title big data a part of mythology: "large data sets offer a higher form of intelligence and knowledge [...], with the aura of truth, objectivity, and accuracy". Users of big data are often "lost in the sheer volume of numbers", and "working with Big Data is still subjective, and what it quantifies does not necessarily have a closer claim on objective truth".[218] Recent developments in BI domain, such as pro-active reporting especially target improvements in the usability of big data, through automated filtering of non-useful data and correlations.[219] Big structures are full of spurious correlations[220] either because of non-causal coincidences (law of truly large numbers), solely nature of big randomness[221] (Ramsey theory), or existence of non-included factors so the hope, of early experimenters to make large databases of numbers "speak for themselves" and revolutionize scientific method, is questioned.[222] Catherine Tucker has pointed to "hype" around big data, writing "By itself, big data is unlikely to be valuable." The article explains: "The many contexts where data is cheap relative to the cost of retaining talent to process it, suggests that processing skills are more important than data itself in creating value for a firm."[223]

Big data analysis is often shallow compared to analysis of smaller data sets.[224] In many big data projects, there is no large data analysis happening, but the challenge is the extract, transform, load part of data pre-processing.[224]

Big data is a buzzword and a "vague term",[225][226] but at the same time an "obsession"[226] with entrepreneurs, consultants, scientists, and the media. Big data showcases such as Google Flu Trends failed to deliver good predictions in recent years, overstating the flu outbreaks by a factor of two. Similarly, Academy Awards and election predictions solely based on Twitter were more often off than on target. Big data often poses the same challenges as small data; adding more data does not solve problems of bias, but may emphasize other problems. In particular data sources such as Twitter are not representative of the overall population, and results drawn from such sources may then lead to wrong conclusions. Google Translate—which is based on big data statistical analysis of text—does a good job at translating web pages. However, results from specialized domains may be dramatically skewed. On the other hand, big data may also introduce new problems, such as the multiple comparisons problem: simultaneously testing a large set of hypotheses is likely to produce many false results that mistakenly appear significant. Ioannidis argued that "most published research findings are false"[227] due to essentially the same effect: when many scientific teams and researchers each perform many experiments (i.e. process a big amount of scientific data; although not with big data technology), the likelihood of a "significant" result being false grows fast – even more so, when only positive results are published. Furthermore, big data analytics results are only as good as the model on which they are predicated. In an example, big data took part in attempting to predict the results of the 2016 U.S. presidential election[228] with varying degrees of success.

Critiques of big data policing and surveillance

[edit]

Big data has been used in policing and surveillance by institutions like law enforcement and corporations (see: corporate surveillance and surveillance capitalism).[229] Due to the less visible nature of data-based surveillance as compared to traditional methods of policing, objections to big data policing are less likely to arise. According to Sarah Brayne's Big Data Surveillance: The Case of Policing,[230] big data policing can reproduce existing societal inequalities in three ways:

  • Placing people under increased surveillance by using the justification of a mathematical and therefore unbiased algorithm
  • Increasing the scope and number of people that are subject to law enforcement tracking and exacerbating existing racial overrepresentation in the criminal justice system
  • Encouraging members of society to abandon interactions with institutions that would create a digital trace, thus creating obstacles to social inclusion

If these potential problems are not corrected or regulated, the effects of big data policing may continue to shape societal hierarchies. Conscientious usage of big data policing could prevent individual level biases from becoming institutional biases, Brayne also notes.

See also

[edit]
For a list of companies, and tools, see also: Category:Big data

References

[edit]

Bibliography

[edit]

Further reading

[edit]
[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Big data

View on Grokipedia
from Grokipedia
Big data denotes the extensive assemblages of data arising from networked digital systems, sensors, and human activities, which exceed the processing capacities of conventional tools and demand specialized technologies for effective management and analysis.[1] These datasets are primarily defined by three core attributes—volume (immense scale), velocity (rapid generation and flow), and variety (diversity of formats, from structured records to unstructured text and multimedia)—often extended to include veracity (reliability amid noise) and value (potential for meaningful extraction).[2] Originating in the late 1990s amid advances in computing and storage, the concept gained prominence with the proliferation of internet-scale data in the 2000s, enabling breakthroughs in predictive modeling across domains like genomics, finance, and logistics through empirical pattern recognition rather than exhaustive enumeration.[3] Key applications have yielded tangible gains, such as optimized supply chains reducing costs by up to 15% via real-time analytics and accelerated drug discovery shortening development timelines, though causal inference remains constrained by data incompleteness and selection effects.[4] Controversies persist around privacy erosion from pervasive surveillance and algorithmic biases perpetuating inequities when training data reflects historical distortions, underscoring the need for rigorous validation over correlative assumptions.[5][6]

History

Early Foundations and Precursors (Pre-2000)

The foundations of handling large-scale datasets trace back to 18th- and 19th-century efforts in statistics and census processing, where manual and mechanical methods grappled with aggregating population and economic data. In the United States, the first federal census in 1790, overseen by Secretary of State Thomas Jefferson, involved marshals collecting demographic details from all thirteen states, resulting in tabulated reports that highlighted early challenges in manual data compilation and estimation techniques for incomplete records.[7] By the late 19th century, these processes evolved with mechanical innovation: Herman Hollerith developed an electric tabulating machine using punched cards to process the 1890 U.S. Census, reducing tabulation time from years to months by electrically reading holes on cards representing data points, thus enabling faster aggregation of over 60 million cards.[8] Mid-20th-century computing marked a shift toward electronic batch processing for voluminous numerical tasks. The ENIAC, completed in 1945 by John Mauchly and J. Presper Eckert at the University of Pennsylvania, was the first general-purpose electronic computer, capable of executing up to 5,000 additions per second for ballistic calculations, demonstrating programmable handling of complex datasets beyond mechanical limits.[9] This paved the way for systems like the UNIVAC I, delivered to the U.S. Census Bureau in 1951, which processed the 1950 population census and 1954 economic census via magnetic tape storage and automated operations at 1,905 calculations per second, illustrating early electronic scalability for government-scale data volumes.[10] Advancements in data organization culminated in Edgar F. Codd's 1970 relational model, which proposed structuring large shared data banks using n-ary relations and normalization to reduce redundancy and enable declarative querying, addressing inefficiencies in hierarchical and network database models prevalent at IBM.[11] In the 1980s and 1990s, pre-internet data warehousing emerged to integrate disparate sources for analysis; Bill Inmon formalized the concept of a centralized, subject-oriented repository for historical data, emphasizing normalized structures to manage growing volumes from operational systems, as terabyte-scale datasets in telecommunications (e.g., call records) and finance (e.g., transaction logs) strained relational systems with integration and query performance issues.[12] These efforts highlighted causal bottlenecks in storage, retrieval, and scalability, foreshadowing needs for distributed processing without yet invoking volume-velocity-variety paradigms.[12]

Emergence in the Digital Age (2000-2010)

The rapid expansion of the internet in the early 2000s generated unprecedented volumes of data from web crawling, user interactions, and server logs, overwhelming conventional database systems and prompting innovations in distributed storage and processing. Google's Google File System (GFS), detailed in a 2003 research paper, addressed this by providing a scalable, fault-tolerant file system optimized for large files and high-throughput streaming across clusters of commodity machines, supporting applications like web indexing that involved multi-gigabyte to petabyte-scale datasets.[13] Building on GFS, Google introduced MapReduce in 2004, a framework that simplified parallel processing of massive datasets by distributing tasks across thousands of nodes, automatically handling failures and data locality to index the web's burgeoning content.[14] These systems enabled Google to manage the petabyte-scale data required for search relevance amid the web's growth to billions of pages. Yahoo, facing similar challenges in processing search and advertising data, drew from Google's non-proprietary papers to create Hadoop, an open-source platform launched in 2006 that replicated GFS via the Hadoop Distributed File System (HDFS) and MapReduce for distributed computation on inexpensive hardware.[15] Hadoop's release marked a shift toward accessible, scalable big data infrastructure, allowing non-elite organizations to handle terabyte-to-petabyte workloads without proprietary tools. The term "big data" emerged around this period, coined in 2005 by Roger Magoulas of O'Reilly Media to characterize the volume, complexity, and analytical demands of data from web-scale sources like logs and user-generated content, distinct from traditional enterprise data management.[16] Adoption accelerated in industry, with Facebook developing Hive by 2007—initially for internal use and detailed publicly in 2009—as a data warehousing layer atop Hadoop, enabling SQL-like queries on petabyte-scale social data stored in HDFS.[17] E-commerce leaders like Amazon employed custom distributed pipelines throughout the decade to process transaction logs and behavioral data for personalization, prefiguring broader reliance on fault-tolerant, horizontal scaling over vertical hardware upgrades. These developments crystallized big data's practical foundations in volume-driven, web-originating challenges, prioritizing resilience and parallelism over relational consistency.

Expansion and Mainstream Adoption (2011-Present)

The Hadoop ecosystem expanded with the release of Hadoop 2.0 in October 2012, introducing YARN (Yet Another Resource Negotiator) for improved resource management and scheduling beyond MapReduce limitations. This facilitated multi-tenancy and diverse workload support, enabling broader enterprise adoption. Subsequently, Apache Spark emerged as a preferred alternative, with its first stable release in May 2014 offering in-memory processing up to 100 times faster than Hadoop MapReduce for iterative algorithms.[18] Spark's integration with Hadoop ecosystems accelerated its uptake, processing petabyte-scale datasets more efficiently by 2015. Cloud platforms democratized big data access post-2011. Microsoft launched Azure HDInsight in 2013 as a managed Hadoop service, simplifying deployment on its infrastructure.[19] Amazon Web Services' EMR, building on its 2010 debut, saw exponential usage growth, handling billions of objects daily by mid-decade through elastic scaling.[20] These services reduced hardware barriers, with global data volumes surging from 2 zettabytes in 2010 to 64.2 zettabytes created, captured, or consumed by 2020, reaching approximately 149 zettabytes by 2024. Regulatory scrutiny intensified following Edward Snowden's June 2013 disclosures of NSA mass surveillance programs, which relied on big data analytics, prompting global debates on privacy risks and leading to reforms like the EU's strengthened data protection frameworks.[21] The COVID-19 pandemic in 2020 further propelled mainstream integration, with big data enabling real-time epidemiological modeling, mobility tracking via telecom datasets, and resource allocation in over 100 countries' response efforts.[22] By 2023, the big data market was valued at around $185 billion. Projections for the global Big Data market size in 2026 vary by scope and source as forward-looking estimates from early 2026: Big Data and Analytics market at $151.89 billion (up from $134.64 billion in 2025 at 12.8% CAGR); Big Data Analytics market at $447.68 billion (from $394.70 billion in 2025 at 12.8% CAGR); broader Big Data market at $273.4 billion (at 11.0% CAGR). Longer-term estimates project growth to $383 billion by 2030 amid cloud and AI synergies, though estimates vary with inclusions like analytics services.[23][24][25][26][27]

Definition and Characteristics

Core Definition

Big data denotes datasets characterized by such immense scale, diversity, and rapidity of generation that they surpass the storage, management, and analytical capacities of conventional relational database systems and standard on-premises computing infrastructure.[1][28] This limitation stems from the inherent constraints of traditional tools, which rely on centralized processing and structured schemas ill-suited to handle unstructured or semi-structured formats alongside high-velocity streams from sensors, networks, and digital interactions.[2] In practice, big data volumes often commence at terabyte levels but frequently extend to petabyte scales—equivalent to one million gigabytes—where sequential processing becomes computationally prohibitive due to time and resource demands.[29][30] The core challenge lies not solely in sheer size but in the causal necessities of deriving timely, insight-generating operations; conventional systems falter in parallelizing tasks across distributed nodes to process heterogeneous data flows without prohibitive latency.[1][31] This paradigm shift enables progression from mere descriptive aggregation—summarizing historical patterns—to predictive modeling that anticipates outcomes through statistical inference on vast samples, and prescriptive recommendations grounded in simulated causal interventions, all contingent on scalable architectures that mitigate the bottlenecks of legacy methods.[2][32] Such definitions underscore big data's essence as a threshold phenomenon, where exceeding traditional bounds necessitates novel computational strategies to unlock empirical value from otherwise intractable corpora.[33][34]

The "Vs" Framework

The "Vs" framework, initially comprising three dimensions—volume, velocity, and variety—serves as a foundational heuristic for characterizing the challenges posed by big data, originating from analyst Doug Laney's 2001 research note on "3D Data Management: Controlling Data Volume, Velocity, and Variety" while at META Group (later acquired by Gartner).[35] Volume refers to the sheer scale of data, often exceeding petabytes or reaching exabytes in aggregate, as evidenced by projections of global data creation surpassing 181 zettabytes by 2025, driven largely by device proliferation.[36] Velocity encompasses the rapid rate of data generation and the need for real-time or near-real-time processing, such as streaming inputs from sensors that demand sub-second latencies to enable responsive analytics.[37] Variety addresses the heterogeneity of data formats, spanning structured relational records, semi-structured logs, and unstructured multimedia, which complicates uniform ingestion and analysis compared to homogeneous traditional datasets.[35] Subsequent expansions of the framework incorporated additional "Vs" to account for non-technical hurdles, including veracity, which denotes uncertainties in data quality, accuracy, and trustworthiness arising from noise, errors, or biases in sources like crowdsourced inputs.[35] Value emphasizes the extraction of actionable, monetizable insights from raw data, underscoring that scale alone does not confer utility without causal linkages to decision-making outcomes.[38] Other proposed extensions, such as variability (fluctuations in data meaning or flow rates) and visualization (effective rendering for human interpretation), appear in practitioner literature but risk proliferating the model beyond its parsimonious origins.[39] Empirically, the framework highlights tangible pressures, as illustrated by Internet of Things (IoT) ecosystems projected to encompass 55.7 billion connected devices by 2025, collectively generating nearly 80 zettabytes of data annually—a volume-velocity-variety confluence that strains conventional storage and querying paradigms.[40] Laney himself has cautioned against conflating these extensions with the core trio, arguing they represent derivative considerations rather than definitional ones.[37] Critics contend the model functions more as a marketing mnemonic than a rigorous taxonomy, potentially oversimplifying causal complexities like integration dependencies or ethical constraints in data provenance, yet its enduring adoption affirms practical utility in scoping infrastructure requirements and diagnosing processing bottlenecks where traditional methods falter.[41] This heuristic's value lies in prompting first-principles evaluation of whether data regimes necessitate distributed architectures, even as empirical evidence from scaled deployments validates its role in prioritizing interventions over exhaustive enumeration.[37]

Distinctions from Traditional Data Processing

Traditional data processing, exemplified by relational database management systems (RDBMS) and business intelligence (BI) workflows, operates on structured datasets typically ranging from megabytes to gigabytes, emphasizing predefined schemas enforced prior to data ingestion—a paradigm known as schema-on-write.[42] This approach ensures data consistency and enables efficient SQL-based querying for hypothesis-driven analysis, but it constrains handling of diverse or rapidly evolving data formats.[43] In big data contexts, schema-on-read prevails, deferring structure imposition until analysis time, which accommodates unstructured and semi-structured data floods from sources like logs or social feeds, prioritizing ingestion speed over upfront validation.[44] Methodologically, traditional BI relies on batch processing for periodic reporting, where data is aggregated in scheduled intervals against known queries, limiting discovery to anticipated patterns.[45] Big data shifts toward stream or near-real-time processing, facilitating exploratory data mining across petabyte-scale volumes to detect correlations amid noise—such as emergent trends in high-velocity inputs—without rigid hypotheses.[46] Architecturally, legacy systems centralize storage and computation on single nodes, exposing vulnerabilities to failures that halt operations, whereas big data mandates distributed clusters with fault tolerance via replication and dynamic reassignment, ensuring continuity despite node losses at scale.[47][48] These distinctions yield measurable outcomes: firms leveraging big data report average revenue uplifts of 8% and cost reductions of 10%, driven by scalable analytics uncovering actionable insights unattainable in constrained traditional setups.[49][50] Such gains stem from causal enablers like parallel processing over vast datasets, though realization depends on robust implementation to mitigate risks like data silos or analytical overfitting.[51]

Technical Architecture

Data Ingestion and Storage Systems

Apache Kafka serves as a distributed streaming platform for real-time data ingestion, enabling high-throughput handling of event data streams from producers to consumers with durability through log-based storage and partitioning across brokers.[52] Originally developed by LinkedIn in 2011 to address low-latency ingestion challenges, it supports fault-tolerant message delivery via replication factors configurable per topic, typically defaulting to three replicas for availability.[53] Complementing Kafka, Apache Flume provides a reliable service for aggregating and transporting large volumes of log data in streaming fashion, using a channel-based architecture where sources collect events and sinks persist them to destinations like HDFS, with configurable reliability through memory or file channels.[54] For batch ingestion, Apache Sqoop facilitates efficient bulk transfer of structured data from relational databases to Hadoop ecosystems via parallel MapReduce jobs, leveraging JDBC connectors to export/import tables while supporting incremental loads based on timestamps or IDs.[55] This tool optimizes for high-volume imports by splitting large tables into mappers that fetch subsets concurrently, reducing transfer times for terabyte-scale datasets. Data storage in big data architectures emphasizes distributed systems for fault tolerance and scalability. The Hadoop Distributed File System (HDFS) distributes large files as blocks typically sized at 128 MB or 256 MB across clusters of commodity nodes, achieving redundancy via a default replication factor of three, which ensures data availability even with node failures by storing copies across racks.[56] HDFS supports horizontal scalability to petabyte and exabyte levels by adding DataNodes, with block placement policies optimizing for locality and bandwidth. For schema-flexible storage of heterogeneous data, NoSQL databases like Apache Cassandra employ wide-column models with tunable consistency, distributing data via consistent hashing rings for linear scalability and high write throughput without single points of failure.[57] Scalability mechanisms include data partitioning—such as HDFS blocks or Cassandra partitions—and compression codecs like Snappy or Gzip to minimize storage footprints while enabling horizontal expansion. Persistent challenges arise in raw storage paradigms: data lakes aggregate unstructured volumes without enforced schemas, risking quality issues, whereas traditional data warehouses impose structure for query efficiency; Delta Lake addresses this by layering ACID transactions, schema enforcement, and time travel on data lakes using Parquet files and transaction logs, enhancing reliability for petabyte-scale persistence without full warehouse overhead.[58]

Processing Engines and Frameworks

The MapReduce programming model, introduced by Google in a 2004 paper, enables distributed processing of large-scale data sets through a parallel map phase that transforms input data into key-value pairs, followed by a shuffle and reduce phase that aggregates results.[14] This paradigm supports fault tolerance via automatic task reassignment on node failures and scales to thousands of commodity servers, making it suitable for batch-oriented jobs handling terabyte to petabyte volumes.[14] However, MapReduce incurs high I/O overhead by writing intermediate results to disk after each map and reduce operation, limiting efficiency for iterative algorithms or workloads requiring multiple passes over data. Subsequent frameworks evolved beyond MapReduce's rigid two-stage structure to directed acyclic graph (DAG) execution models, allowing optimization of complex workflows. Apache Spark, originating from UC Berkeley research and becoming an Apache project in 2013, introduced resilient distributed datasets (RDDs) for in-memory caching and lazy evaluation, reducing disk I/O for repeated computations.[59] This enables Spark to process data up to 100 times faster than MapReduce for iterative machine learning tasks on clusters of commodity hardware, as intermediate data remains in RAM rather than being persisted to disk.[60] For extract-transform-load (ETL) pipelines, Spark has demonstrated reductions in processing times from hours or days to minutes for multi-terabyte jobs, balancing volume through horizontal scaling and velocity via reduced latency in batch modes.[61] Apache Flink extends DAG-based processing to unified batch and stream workloads, emphasizing low-latency event-time processing with exactly-once semantics and stateful computations.[62] Flink's architecture handles unbounded data streams by maintaining operator state across failures and supports windowed aggregations, making it effective for velocity-intensive scenarios like real-time fraud detection where MapReduce or Spark batch modes fall short.[63] Both Spark and Flink operate on commodity hardware clusters, processing petabyte-scale jobs through fault-tolerant distribution, though they trade some MapReduce simplicity for greater expressiveness in handling diverse data velocities.[59]

Analytics Pipelines and Scalability Mechanisms

Analytics pipelines in big data environments orchestrate end-to-end workflows as directed acyclic graphs (DAGs), enabling the sequencing of data ingestion, transformation, analysis, and output stages across distributed systems. Apache Airflow, an open-source platform released in 2015, facilitates this by allowing programmatic definition, scheduling, and monitoring of such pipelines, supporting fault-tolerant execution through retries and dependency management.[64] Kubeflow extends this for machine learning-specific pipelines on Kubernetes clusters, providing components for data preparation, model training, and serving while ensuring reproducibility via containerized steps.[65] Integration with MLflow, introduced in 2018, adds versioning for models, parameters, and artifacts, tracking experiments to maintain pipeline integrity amid iterative big data analyses.[66] Scalability mechanisms address the volume and velocity of big data by enabling elastic resource allocation, preventing bottlenecks through dynamic adjustment to workload demands. Kubernetes orchestration supports auto-scaling clusters via Horizontal Pod Autoscalers, which adjust the number of pods based on CPU, memory, or custom metrics, achieving sub-minute response times to load changes as of its 1.23 release in December 2021.[67] Data sharding distributes datasets across nodes to parallelize processing, reducing query latency in systems handling petabyte-scale volumes, while indexing structures accelerate retrieval by organizing data for efficient lookups without full scans.[68] Fault-tolerance is embedded via data replication and checkpointing, ensuring continuity during node failures; for instance, triple replication in distributed stores maintains availability even with multiple concurrent outages.[69] These mechanisms demonstrate causal efficacy in real-world elasticity, where auto-scaling clusters dynamically provision resources to absorb traffic surges, averting downtime from overload. E-commerce platforms, for example, leverage such systems to manage Black Friday spikes—often exceeding 10x baseline traffic—by preemptively scaling compute instances, as evidenced by cases reducing infrastructure costs by 85% post-event while sustaining seamless operations.[70] This elasticity directly counters causal chains of failure, such as queue overflows leading to lost data, by matching capacity to instantaneous demand rather than static provisioning.[71]

Key Technologies

Open-Source Foundations (Hadoop Ecosystem)

The Hadoop framework, initiated as an Apache Software Foundation project in April 2006, established the foundational open-source architecture for scalable big data storage and processing on clusters of commodity hardware.[15] Its core components include the Hadoop Distributed File System (HDFS), which provides fault-tolerant, distributed storage optimized for large files by replicating data blocks across nodes, and MapReduce, a programming model for parallel processing that divides tasks into map (data transformation) and reduce (aggregation) phases to handle petabyte-scale datasets efficiently.[72] [73] In 2012, Hadoop 2.0 introduced Yet Another Resource Negotiator (YARN), decoupling resource management from job scheduling to enable multi-tenancy and support diverse workloads beyond MapReduce, thereby enhancing cluster utilization.[74] Complementing the core, higher-level abstractions like Apache Pig and Hive addressed usability gaps in raw MapReduce coding. Pig, a scripting platform launched around 2008, offers a procedural language (Pig Latin) for expressing data flows and transformations, compiling them into MapReduce jobs to simplify ETL processes without requiring Java expertise.[75] Hive, developed starting in 2007 and donated to Apache in 2008, functions as a data warehousing layer atop HDFS, enabling SQL-like querying (HiveQL) for structured data analysis by translating queries into MapReduce or later YARN-managed tasks, thus bridging relational database paradigms with distributed systems.[76] Early adoption propelled Hadoop's influence, with Yahoo deploying its first production cluster in January 2006 and scaling to a 1,000-node setup by 2007 for web indexing and search optimization, validating the framework at massive volumes.[77] [74] Facebook integrated Hadoop extensively from 2008 onward to underpin its data infrastructure, processing billions of events daily for analytics and enabling department-wide self-service data access, which fostered a data-driven operational culture.[74] This open-source model, unencumbered by licensing fees, contrasted with proprietary vendor silos, empowering startups and smaller entities to build competitive big data capabilities on inexpensive hardware rather than relying on costly, closed ecosystems.[78] [79] Despite its breakthroughs, Hadoop's MapReduce paradigm imposed limitations inherent to batch-oriented processing, where jobs incur high latency—often minutes to hours—due to disk I/O for intermediate results and lack of support for real-time or streaming data, rendering it unsuitable for interactive or low-latency applications.[80] [81] Nonetheless, as the dominant infrastructure of the 2010s, Hadoop democratized access to distributed computing, spawning an ecosystem that lowered barriers to entry for big data experimentation and scaled empirical successes across industries.[78][82]

In-Memory and Stream Processing Tools (Spark, Kafka)

Apache Spark, an open-source unified analytics engine, was initially developed as a research project at the University of California, Berkeley's AMPLab in 2009 and open-sourced in 2010, with its first stable release (version 1.0) occurring in May 2014.[83] It enables large-scale data processing through in-memory computation, which caches data in RAM to accelerate iterative algorithms and queries by factors of up to 100 times compared to disk-based alternatives for certain workloads.[84] Spark supports batch processing, real-time stream processing via Spark Streaming, and machine learning through its MLlib library, which provides scalable implementations of algorithms like regression, clustering, and recommendation systems.[85][86] This unified framework allows developers to apply the same APIs across diverse data processing tasks, reducing complexity in handling both static datasets and continuous data flows inherent in big data environments. Apache Kafka, originally created at LinkedIn and open-sourced in early 2011, functions as a distributed event streaming platform that implements a publish-subscribe model for high-throughput messaging.[52] It decouples data producers, which publish events to topics, from consumers, which subscribe to those topics for processing, enabling asynchronous and scalable data pipelines without tight coupling between components.[87] Kafka's architecture supports durable storage of event streams as an ordered, immutable log, allowing for replayability and fault tolerance, while achieving throughput rates of millions of messages per second on commodity hardware.[88] This capability makes it suitable for ingesting and distributing real-time data feeds, such as logs, metrics, or transactions, in environments requiring low-latency continuity. In big data workflows, Spark and Kafka often integrate to form efficient processing pipelines, where Kafka handles ingestion and buffering of streaming events, and Spark performs in-memory analytics on those streams for immediate insights. For instance, financial institutions have deployed such combinations for real-time fraud detection, analyzing transaction patterns as they arrive to flag anomalies; studies indicate that advanced streaming-based systems can reduce fraudulent transactions by up to 35% compared to batch methods.[89] This approach leverages Kafka's high-velocity data routing with Spark's rapid computation, minimizing delays in dynamic scenarios like payment processing where milliseconds matter for loss prevention.

Cloud-Native and Hybrid Solutions

Cloud-native big data architectures utilize public cloud platforms to deliver elastic scalability, managed services, and consumption-based pricing, decoupling users from fixed infrastructure costs. Amazon Web Services (AWS) provides Simple Storage Service (S3) for durable object storage integrated with Elastic MapReduce (EMR) for on-demand Hadoop and Spark clusters, allowing automatic scaling based on workload demands.[90] Google Cloud's BigQuery offers serverless SQL querying over petabyte-scale datasets, eliminating cluster management while supporting real-time analytics through decoupled storage and compute.[91] Microsoft Azure Synapse Analytics combines data integration, warehousing, and machine learning in a unified workspace, enabling independent scaling of compute resources against Azure Data Lake storage.[92] These solutions facilitate infinite horizontal scaling and reduced operational overhead, as providers handle provisioning, patching, and optimization. By 2025, 72% of global workloads, including substantial big data processing tasks, operate in cloud-hosted environments, reflecting a migration from 66% the prior year driven by cost efficiencies and agility.[93] Approximately 95% of new digital workloads, many involving big data pipelines, deploy on cloud-native platforms, prioritizing serverless models for faster iteration.[94] Hybrid cloud approaches integrate on-premises systems with public clouds to address data sovereignty and compliance needs, such as GDPR's requirements for data locality to prevent unauthorized cross-border transfers. In these setups, sensitive datasets remain in private data centers for regulatory adherence, while non-sensitive processing bursts to the cloud during peak demands, using tools like AWS Outposts or Azure Stack for consistent APIs across environments.[95] This model supports compliance by enforcing data residency policies, as seen in hybrid integrations where local storage connects to public services via governed gateways.[96] Providers like AWS, Azure, and Google Cloud offer region-specific deployments certified for GDPR, enabling organizations to process big data volumes without full cloud migration.[97]

Applications and Demonstrated Benefits

Business and Economic Applications

Big data facilitates supply chain optimization by integrating predictive analytics with real-time data streams from sensors, RFID tags, and transaction logs, enabling precise demand forecasting and inventory management. This reduces operational inefficiencies such as overstocking or stockouts, which traditionally account for 5-10% of retail costs. Walmart, for example, utilizes big data platforms to monitor workflow across pharmacies, distribution centers, and stores, allowing for dynamic adjustments that enhance replenishment efficiency and cut delivery times from suppliers to shelves.[98] In marketing, big data drives personalization through recommendation engines that process user interaction histories, purchase patterns, and browsing behaviors to deliver targeted suggestions, thereby boosting conversion rates and customer retention. These engines, often powered by machine learning algorithms analyzing petabytes of data, can increase sales uplift by 10-30% in e-commerce settings by matching products to individual preferences rather than relying on broad segmentation.[99] Such applications shift marketing from mass campaigns to granular, data-informed strategies, amplifying return on ad spend through measurable engagement metrics.[100] Economically, big data adoption correlates with measurable productivity improvements, with McKinsey analysis indicating that data leaders in retail can achieve 5-6% reductions in working capital via optimized merchandising and supply chain decisions. This stems from causal mechanisms like reduced decision latency and error rates, fostering innovation in resource allocation. In competitive markets, big data erodes advantages held by incumbents with physical assets, empowering agile entrants to disrupt through superior informational efficiency and rapid iteration on customer insights, thereby intensifying market contestability.[101]

Sector-Specific Implementations

In healthcare, big data enables predictive epidemiology through integration of diverse datasets such as mobility patterns, electronic health records, and wearable sensor outputs. During the 2020 COVID-19 pandemic, models incorporating these sources forecasted outbreak trajectories; for example, Zhu et al. analyzed large-scale wearable device data segmented by geography to estimate infection trends, achieving alignment with reported cases in multiple regions.[102] Similarly, deep learning frameworks applied to global big data streams, including news and travel records, predicted case surges with reported accuracies exceeding 90% in select national forecasts by mid-2020.[103] The finance sector deploys big data for algorithmic trading via high-frequency processing of tick-level data, which captures every trade, quote update, and order book change. High-frequency trading (HFT) firms analyze petabytes of such granular data daily to execute strategies exploiting microsecond price discrepancies, accounting for over 50% of U.S. equity trading volume as of 2020.[104] Projects leveraging proprietary tick simulators have demonstrated alpha generation through momentum and market-making algorithms on this data scale.[105] Retail applications harness big data for dynamic pricing, adjusting costs in real time based on demand signals, competitor actions, and consumer behavior analytics. Amazon, for instance, updates millions of product prices daily using algorithms that process purchase histories, browsing patterns, and external market feeds to optimize revenue, with reported price changes occurring up to 2.5 million times per day across its platform.[106] Uber employs similar big data-driven surge pricing, factoring in ride requests, driver availability, and traffic data to modulate fares, as seen during peak events where multipliers reached 9x in high-demand areas.[107] In manufacturing and smart cities, Internet of Things (IoT) sensor analytics processes vast streams from connected devices for operational optimization. Factories deploy big data platforms to analyze sensor feeds from machinery, predicting equipment failures via pattern recognition in vibration and temperature data, reducing downtime by up to 50% in implementations reported by industrial adopters.[108] Smart city initiatives integrate IoT big data for traffic management, where aggregated vehicle and infrastructure sensor inputs enable predictive flow modeling; for example, systems in deployed urban networks forecast congestion with 85% accuracy using historical and real-time feeds.[109] Government uses include traffic and crime prediction, drawing on spatiotemporal big data from cameras, GPS, and incident logs. In traffic forecasting, agencies process IoT-derived mobility data to anticipate bottlenecks, as in U.S. Department of Transportation pilots achieving 20-30% improvements in commute predictions via machine learning on multi-source datasets.[110] For crime, predictive policing tools like PredPol, operational since 2011 in cities including Los Angeles, analyze historical offense data to generate daily hot-spot maps, directing patrols to probable incidents with claimed reductions in burglaries by 7-20% in evaluated districts.[111] Global implementations vary, with China's social credit system—outlined in a 2014 State Council document and piloted thereafter—employing big data from financial transactions, surveillance footage, and online activity to score citizen compliance, affecting 1.4 billion individuals through blacklists and incentives by 2020.[112] In contrast, the U.S. emphasizes private-sector leadership in big data efficiency, where firms invest disproportionately in scalable analytics for commercial gains, outpacing state-directed models in sectors like e-commerce and finance through decentralized innovation.[113]

Empirical Evidence of Value

Organizations employing big data analytics have achieved quantifiable financial improvements. A BARC survey of businesses using big data found that those quantifying their analytics outcomes experienced an average 8% revenue increase and 10% cost reduction, attributed to enhanced decision-making and operational efficiencies.[114][115] Big data facilitates accelerated innovation cycles. IDC research indicates that firms with superior enterprise intelligence—including advanced big data processing—innovate at rates 2.5 times faster than peers with deficient capabilities, enabling quicker development and deployment of new products and services.[116] In healthcare, big data combined with AI has driven diagnostic advancements. National Institutes of Health analyses show that these technologies improve diagnostic accuracy and treatment planning by leveraging large-scale patient data for pattern recognition and predictive modeling, yielding superior outcomes over traditional methods.[117] At the macroeconomic level, big data contributes to GDP growth in advanced economies through resource optimization and productivity enhancements. McKinsey Global Institute projections, based on sector-specific analyses, estimate that widespread adoption could add 1-2% to annual GDP via efficiencies in areas like manufacturing and public administration.

Challenges in Implementation

Technical and Operational Difficulties

Managing the heterogeneity and scale of big data introduces significant engineering challenges, particularly in ensuring data quality. Poor data quality undermines analytical outcomes through the "garbage in, garbage out" principle, where erroneous or incomplete inputs propagate inaccuracies across pipelines. Estimates indicate that 60-73% of enterprise data remains unused due to quality deficiencies, while poor data overall costs organizations approximately 12% of annual revenue.[118] Common issues include incomplete datasets, inaccuracies from inconsistent sources, and duplicates arising from heterogeneous formats, exacerbating integration difficulties.[119] Data silos further compound quality problems by isolating information across systems, impeding unified processing and cleansing. These silos, often resulting from legacy architectures or departmental boundaries, hinder schema matching and entity resolution, leading to fragmented views that distort insights. Pre-cloud era storage demands amplified these issues, with exploding volumes driving prohibitive hardware costs—often in the millions for petabyte-scale setups—before distributed file systems like Hadoop mitigated them.[120] Even with modern solutions, velocity challenges persist: high-speed data streams from sources like IoT sensors overload traditional batch processing, causing latency in real-time analytics and potential bottlenecks in ingestion pipelines.[121] Empirical evidence underscores these hurdles, with industry analyses reporting failure rates exceeding 80% for big data projects, frequently attributed to unresolved quality and scalability defects. A 2025 review cites Gartner's longstanding assessment that 85% of such initiatives falter, often from inadequate handling of volume, variety, and velocity. These rates reflect not just technical mismatches but the causal chain where unaddressed data flaws cascade into unreliable models and operational inefficiencies.[122][123]

Human and Organizational Barriers

A persistent challenge in big data implementation is the shortage of skilled personnel, particularly data engineers capable of managing large-scale data pipelines and architectures. According to the World Economic Forum's Future of Jobs Report 2025, skills in AI and big data rank among the fastest-growing in demand, exacerbating a talent gap where supply lags significantly behind needs. Analyses of job applications in Q2 2025 indicate a 12-fold shortfall in data engineering expertise relative to openings, driving up hiring costs and competitive salaries as organizations vie for limited qualified candidates.[124] This disparity, compounded by the need for specialized knowledge in tools like SQL, Python, and distributed systems, hinders scalability and delays project timelines. Cultural resistance further impedes adoption, as entrenched organizational mindsets prioritize intuitive decision-making over empirical data analysis. In established firms, teams often cling to legacy practices rooted in experience-based judgments, viewing data-driven approaches as disruptive or unnecessary despite evidence of superior outcomes in predictive modeling and optimization.[125] This resistance manifests in reluctance to shift workflows, fostering skepticism toward big data's value and slowing cultural transitions toward analytics-centric operations.[126] Organizational structures exacerbate these issues through data silos and fragmented governance, where departments maintain isolated repositories that prevent holistic data utilization. Such silos, prevalent in large enterprises, obstruct cross-functional collaboration and comprehensive analytics, as data remains trapped within business units without standardized access protocols.[127] In the public sector, this contributes to high failure rates, with estimates indicating over 50% of big data initiatives falter due to inadequate business cases and unproven ROI, often from misaligned metrics that undervalue long-term gains against upfront investments.[128] Gartner analyses similarly report that up to 85% of big data projects overall fail to deliver expected returns, underscoring the need for integrated governance to align data strategies with measurable objectives.[122]

Controversies and Critiques

Privacy, Security, and Surveillance Concerns

The aggregation and analysis of vast datasets in big data systems have amplified privacy risks, as demonstrated by high-profile incidents of unauthorized access and misuse. In 2017, Equifax suffered a breach that exposed sensitive personal information, including Social Security numbers and birth dates, of approximately 147 million individuals due to unpatched software vulnerabilities in its big data infrastructure.[129] Similarly, the 2018 Cambridge Analytica scandal involved the harvesting of profile data from up to 87 million Facebook users without explicit consent, enabling psychographic targeting for political campaigns through app-based data collection and inference techniques.[130] These cases highlight how centralized big data repositories, often reliant on third-party integrations, create single points of failure for identity theft, profiling, and manipulation, though such breaches frequently trace to implementation flaws rather than inherent data scale.[131] Surveillance concerns arise from state actors leveraging big data for monitoring, as seen in the post-9/11 expansion of NSA programs collecting metadata and communications en masse to detect threats. This approach, involving petabyte-scale analysis, contributed to foiling specific plots by correlating patterns across global datasets, underscoring big data's role in preempting terrorism through probabilistic modeling.[132] On the law enforcement front, predictive policing algorithms like PredPol have empirically reduced targeted crimes by 7.4% to 19.8% in controlled deployments, such as in Los Angeles and other U.S. jurisdictions, by forecasting hotspots from historical incident data and optimizing patrols.[133][134] These security gains illustrate causal links where big data analytics enhance deterrence and response efficiency, often outweighing privacy costs in high-stakes domains when calibrated against baseline crime rates. Private-sector innovations address these tensions more effectively than prescriptive rules, with techniques like federated learning enabling model training across distributed datasets without transferring raw data, thus preserving privacy in big data workflows—data remains localized while aggregated insights improve accuracy.[135] Empirical assessments indicate that stringent privacy mandates can impede such advancements by raising compliance burdens, correlating with reduced innovation in data-driven firms, particularly smaller entities reliant on agile experimentation.[136] While alarmism over big data surveillance risks systemic overreach, evidence from breaches and applications alike reveals that targeted security practices yield measurable benefits, tempering the narrative of unmitigated harm with instances of causal efficacy in threat mitigation.[137]

Bias, Accuracy, and Overreliance Issues

Big data analyses frequently amplify inherent biases in source datasets, particularly when algorithms are trained on historically skewed samples, leading to discriminatory outcomes in decision-making tools. For example, AI-driven hiring systems have been observed to favor candidates from overrepresented demographics, as training data reflecting past hiring patterns—often male-dominated in tech—penalizes resumes with terms like "women's" or names associated with underrepresented groups.[138] This algorithmic amplification occurs because machine learning models optimize for patterns in available data without inherent causal understanding, perpetuating inequities unless explicitly corrected.[139] A related statistical pitfall is the conflation of correlation with causation, where vast datasets uncover spurious associations—such as ice cream sales correlating with drownings due to seasonal confounders—mistaken for direct effects, undermining causal realism in inferences.[140] Accuracy challenges arise from the "big data fallacy," the misconception that data volume alone ensures validity, overlooking that small, carefully curated datasets often yield superior, less noisy results for hypothesis testing.[141] In large samples, even low error rates produce numerous false positives; for instance, genomic studies in the 2010s, including genome-wide association analyses, generated thousands of illusory variant-disease links due to unadjusted multiple testing across millions of data points, prompting retractions and methodological reforms.[142] These overclaims stemmed from overreliance on p-value thresholds without accounting for dataset scale, highlighting how empirical overconfidence ignores base rates and selection effects. Critiques of big data often emphasize equity risks from biased inputs, a perspective prominent in academia and media sources exhibiting systemic left-wing institutional biases that prioritize narrative over falsifiable evidence. However, rigorous studies demonstrate that diversifying training data—incorporating varied demographic and contextual samples—significantly reduces model bias while preserving predictive accuracy, as validated in machine learning applications across domains.[143] Overreliance fears, including exaggerated job displacement, lack empirical support; analyses of AI and big data adoption show negative correlations with unemployment, driven by productivity boosts creating net new roles in analytics and tech, with displacement limited to routine tasks offset by demand for skilled oversight.[144]

Regulatory and Ethical Debates

The European Union's General Data Protection Regulation (GDPR), effective May 25, 2018, mandates stringent requirements for data processing, consent, and breach notifications, resulting in compliance costs for companies averaging €1-3 million annually for mid-sized firms handling big data.[145] Critics argue these burdens disproportionately hinder innovation by restricting data flows essential for machine learning models, particularly disadvantaging startups reliant on aggregated datasets. Empirical analyses indicate GDPR has shifted firm focus from novel product development to compliance, contributing to Europe's lag behind the United States in big data-driven AI advancements, where U.S. private investment in AI reached $67 billion in 2023 compared to Europe's $6 billion.[146][147] Similarly, California's Consumer Privacy Act (CCPA), effective January 1, 2020, imposes opt-out rights and disclosure obligations on data brokers, with enforcement actions yielding fines up to $7,500 per intentional violation, amplifying operational overhead for big data analytics firms.[148][149] Ethical controversies in big data often center on consent and autonomy, exemplified by Facebook's 2012 experiment, published in 2014, which altered news feeds for 689,003 users to study emotional contagion without explicit informed consent, prompting accusations of violating human subjects research standards.[150][151] Researchers contended this breached institutional review board protocols, as users' terms-of-service agreement did not suffice for psychological manipulation at scale.[152] Pushback against framing merit-based algorithmic outcomes as inherent "discrimination" emphasizes that such critiques overlook causal evidence of performance differentials rooted in verifiable inputs rather than systemic exclusion.[5] Policy debates reflect ideological divides, with advocates for treating personal data as individual property rights arguing this enables voluntary markets for data exchange, fostering efficient allocation without coercive mandates.[153] In contrast, equity-focused perspectives, often from academic and advocacy circles, demand regulatory interventions to enforce proportional representation in datasets, prioritizing distributive fairness over utility maximization.[5] Empirical observations favor lighter regulatory touch, as U.S. market-driven approaches have accelerated big data synergies with AI—evidenced by 90% of leading AI models originating from U.S. firms—yielding broader societal gains in productivity and discovery compared to Europe's precautionary frameworks.[147][154] This supports policy preferences for targeted safeguards and innovation sandboxes over blanket rules, preserving competitive dynamism.[155]

Future Trends and Developments

AI and Machine Learning Synergies

The convergence of big data and artificial intelligence (AI) in the 2020s has revolutionized pattern recognition by supplying voluminous, diverse datasets essential for training complex machine learning models. Large language models (LLMs), such as OpenAI's GPT-3, were trained on approximately 45 terabytes of filtered text data sourced from the internet, books, and other repositories, enabling emergent capabilities in language understanding and generation.[156] Successor models like GPT-4 expanded this scale to petabytes of data, incorporating multimodal inputs to improve contextual reasoning and predictive performance across tasks.[157] This integration underscores how big data's volume and variety directly fuel AI's ability to discern intricate correlations unattainable with smaller datasets. Automated insights derived from AI processing of big data have become ubiquitous in enterprise analytics by 2025, propelled by generative AI's efficiency in extracting actionable intelligence from petabyte-scale repositories.[158] Predictive analytics has advanced markedly, with machine learning algorithms applied to big data enabling real-time forecasting of outcomes in domains like supply chain management and customer behavior, often surpassing traditional statistical methods in accuracy.[159] These hybrids facilitate causal inference and scenario simulation, transforming raw data volumes into probabilistic models that inform strategic decisions. Synthetic data generation represents a pivotal advance in this synergy, addressing data scarcity and privacy constraints by algorithmically creating datasets that replicate the statistical properties of real big data without exposing sensitive information. Techniques such as generative adversarial networks produce high-fidelity synthetic samples, augmenting training sets for AI models while complying with regulations like GDPR.[160] Empirical trends from 2024-2025 demonstrate that big data-AI integrations yield substantial firm-level gains, including productivity uplifts valued in trillions globally through optimized operations and innovation.[161]

Emerging Paradigms (Edge, Real-Time, Quantum)

Edge computing represents a paradigm shift in big data handling by decentralizing processing to the data generation site, particularly within IoT networks, thereby bypassing centralized cloud dependencies for latency-sensitive applications. This approach processes voluminous sensor data locally, reducing transmission overhead and enabling sub-millisecond response times in prototypes deployed in industrial IoT settings as of 2025. For instance, edge gateways in manufacturing have achieved latency drops from tens of milliseconds to under one millisecond, facilitating predictive maintenance on petabyte-scale equipment data streams without compromising accuracy.[162][163] Real-time big data paradigms prioritize streaming analytics to address velocity challenges, ingesting and querying high-throughput data flows continuously rather than in batches. Frameworks like Apache Flink and Kafka Streams support this by applying complex event processing to terabytes-per-second inputs from sources such as financial transactions or traffic sensors, yielding actionable insights within seconds. Early 2020s prototypes demonstrated scalability to millions of events per second, optimizing for low-latency anomaly detection in datasets exceeding classical batch limits.[164][165] Quantum computing paradigms are emerging to tackle big data optimization problems beyond classical feasibility, leveraging qubits for parallel exploration of vast search spaces in areas like clustering and recommendation systems. Experiments from the early 2020s, including IBM's quantum approximate optimization algorithm applications, have prototyped speedups for logistics datasets with billions of variables, though noise-limited coherence restricts scale to hundreds of qubits as of 2025. These efforts foreshadow post-2025 hybrids where quantum processors augment classical big data pipelines for exponential gains in simulation-based analytics.[166][167] Collectively, these paradigms project handling a global datasphere swelling to 394 zettabytes by 2028, driven by IoT proliferation and AI demands.[168] While fostering innovations in secure, decentralized analytics—such as edge-encrypted federated learning—they heighten risks of fragmented governance, potentially amplifying surveillance vulnerabilities or unmitigated biases in unregulated quantum-accelerated models.[169]

References

  1. [PDF] NIST Big Data Interoperability Framework: Volume 1, Definitions
  2. What is your definition of Big Data? Researchers' understanding of ...
  3. [2008.05835] "Big Data" and its Origins - arXiv
  4. Strategic business value from big data analytics: An empirical ...
  5. Ethical Challenges Posed by Big Data - PMC - NIH
  6. Privacy and Big Data | Stanford Law Review
  7. Who Conducted the First Census in 1790?
  8. The Hollerith Machine - U.S. Census Bureau
  9. https://www.britannica.com/technology/ENIAC
  10. UNIVAC I - U.S. Census Bureau
  11. [PDF] A Relational Model of Data for Large Shared Data Banks
  12. A Short History of Data Warehousing - Dataversity
  13. [PDF] The Google File System
  14. [PDF] MapReduce: Simplified Data Processing on Large Clusters
  15. A Brief History of the Hadoop Ecosystem - Dataversity
  16. The history of big data | LightsOnData
  17. Hive - A Petabyte Scale Data Warehouse using Hadoop
  18. Downloads | Apache Spark
  19. Azure HDInsight announcements: Significant price reduction and ...
  20. Amazon EMR archive of release notes
  21. Surveillance, Snowden, and Big Data: Capacities, consequences ...
  22. Applications of Big Data Analytics to Control COVID-19 Pandemic
  23. Global Market to Reach $383.4 Billion by 2030 - Explosion of IoT Big ...
  24. Big Data Market Size To Reach $862.31 Billion By 2030
  25. What Is Big Data? - Oracle
  26. How big is Big Data? A comprehensive survey of data production ...
  27. Big data tools: A guide for scalable data operations - RudderStack
  28. Components and Development in Big Data System: A Survey
  29. NIST Big Data Interoperability Framework: Volume 1, Big Data ...
  30. [PDF] NIST Big Data Interoperability Framework: Volume 1, Definitions
  31. Scientific Research and Big Data
  32. What Are the 3 V's of Big Data? | Definition from TechTarget
  33. Big data statistics: How much data is there in the world? - Rivery
  34. Gartner's Original "Volume-Velocity-Variety" Definition of Big Data
  35. The 7 Vs of Big Data - Integrate.io
  36. Big Data characteristics (3V, 5V, 10V, 14V) - Artera
  37. Future of Industry Ecosystems: Shared Data and Insights - IDC Blog
  38. Gartner's Big Data Definition Consists of Three Parts, Not to Be ...
  39. Data Management: Schema-on-Write Vs. Schema-on-Read
  40. Schema-on-Read vs. Schema-on-Write - CelerData
  41. Data Management: Schema-on-Write Vs. Schema-on-Read | Upsolver
  42. Real-Time Vs. Batch Analytics: How Modern BI Platforms Handle Both
  43. Batch Processing vs Stream Processing: Key Differences & Use Cases
  44. What Is a Distributed Database? - Oracle
  45. The Power of Distributed Systems for Data-Driven Innovation
  46. Percentage of Companies Investing in Big Data - Edge Delta
  47. 5 Stats That Show How Data-Driven Organizations Outperform Their ...
  48. Full article: BIG data – BIG gains? Understanding the link between ...
  49. Introduction - Apache Kafka
  50. What is Kafka? - Apache Kafka Explained - AWS - Updated 2025
  51. Welcome to Apache Flume — Apache Flume
  52. Sqoop User Guide (v1.4.6)
  53. What is Hadoop Distributed File System (HDFS)? - IBM
  54. Apache Cassandra | Apache Cassandra Documentation
  55. Delta Lake vs Data Lake - What's the Difference?
  56. [PDF] MapReduce vs. Spark for Large Scale Data Analytics
  57. Hadoop MapReduce vs. Apache Spark Who Wins the Battle?
  58. Spark vs Hadoop MapReduce: 5 Key Differences | Integrate.io
  59. Apache Flink® — Stateful Computations over Data Streams ...
  60. Apache Flink: Stream Processing for All Real-Time Use Cases
  61. Orchestrating ML Workflows with Airflow and Kubeflow
  62. A Brief Comparison of Kubeflow vs Airflow - JFrog
  63. A Guide to MLOps with Airflow and MLflow - Medium
  64. Horizontal Pod Autoscaling - Kubernetes
  65. Scaling Databases: A Comprehensive Guide to Database Indexes ...
  66. Designing Scalable Architectures for Cloud-Native Applications
  67. Case Study: Autoscaling for Black Friday Traffic Surges - Inventive HQ
  68. Understanding Elasticity and Scalability in Cloud Computing
  69. An introduction to Apache Hadoop for big data - Opensource.com
  70. What Is Hadoop? Components of Hadoop and How Does It Work
  71. Evolution of Hadoop from MapReduce to YARN | Qubole
  72. Introduction to Apache Pig - GeeksforGeeks
  73. What is Hive? - Apache Hive Explained - AWS
  74. Apache Hadoop turns 10: The Rise and Glory of Hadoop - ProjectPro
  75. Apache Hadoop: What is it and how can you use it? - Databricks
  76. Apache Hadoop. In the dynamic realm of data mining and… - Medium
  77. 13 Big Limitations of Hadoop & Solution To Hadoop Drawbacks
  78. Limitations of Hadoop – How to overcome Hadoop drawbacks
  79. Harness the Colossal Power of Big Data with Apache Hadoop
  80. Apache Spark History
  81. What is Spark? - Introduction to Apache Spark and Analytics - AWS
  82. Overview - Spark 4.0.1 Documentation - Apache Spark
  83. Spark Streaming Programming Guide
  84. Apache Kafka documentation
  85. Powered By - Apache Kafka
  86. [PDF] Real-Time Fraud Detection: Leveraging Apache Kafka and Spark for ...
  87. 15 Best Big Data Analytics Tools for Smarter Decisions in 2025
  88. Top 8 Big Data Platforms and Tools in 2025 - Turing
  89. Top 6 Cloud Data Warehouse Solutions in 2025 [Compared]
  90. Cloud Adoption Statistics 2025: Growth, Migration Drivers, ROI
  91. 300+ Cloud Computing Statistics (October- 2025) - Brightlio
  92. What Is Hybrid Cloud? Use Cases, Pros and Cons - Oracle
  93. Hybrid Cloud Solutions Can Make Your Organization GDPR ...
  94. Top GDPR Cloud Storage Solutions for Data Protection in 2025
  95. 5 Ways Walmart Uses Big Data to Help Customers
  96. The Power of Recommendation Engines in E-commerce
  97. Unlocking the next frontier of personalized marketing - McKinsey
  98. Why Big Data is the new competitive advantage
  99. Digital epidemiology: harnessing big data for early detection and ...
  100. Forecasting the Spread of COVID-19 Using Deep Learning and Big ...
  101. The World of High-Frequency Algorithmic Trading - Investopedia
  102. [PDF] MS&E 448: Big Financial Data for Algorithmic Trading High ...
  103. How these 8 brands drove massive success from Dynamic Pricing
  104. Harnessing AI For Dynamic Pricing For Your Business - Forbes
  105. IoT Data Analytics: Turning Insights into Revenue Opportunities
  106. IoT Smart City Applications (2025) - Digi International
  107. IoT Analytics for Smart Cities - CARTO
  108. The Role of Data Analytics in Predictive Policing
  109. [PDF] China's Social Credit System: Data, Algorithms and Implications By
  110. Charted: U.S. is the private sector AI leader - Axios
  111. Benefits of Big Data Analytics: Increased Revenues and ... - BARC
  112. How Companies Are Using Big Data to Boost Sales, and How You ...
  113. Worldwide Future of Digital Innovation 2023 Predictions | IDC Blog
  114. Impact of AI and big data analytics on healthcare outcomes - NIH
  115. Data Analytics Statistics 2025 – Market Insights and Industry Trends
  116. Data Quality Problems? 8 Ways to Fix Them in 2025 - Atlan
  117. Top 7 Big Data Challenges - Datamation
  118. The 3 V's of Big Data: Velocity Remains A Challenge for Many
  119. 50 Statistics Every Technology Leader Should Know in 2025
  120. Why Big Data Science & Data Analytics Projects Fail
  121. Data Engineering skill-gap analysis : r/dataengineering - Reddit
  122. Why Most Big Data Projects Fail - Proactive Strategies for Success
  123. Enabling a Data Driven Culture: Strategies to Overcoming ...
  124. What are Data Silos? | IBM
  125. [PDF] Unveiling the Roots of Big Data Project Failure: a Critical Analysis of ...
  126. Equifax to Pay $575 Million as Part of Settlement with FTC, CFPB ...
  127. Revealed: 50 million Facebook profiles harvested for Cambridge ...
  128. Cambridge Analytica and Facebook: The Scandal and the Fallout ...
  129. 9/11 and the reinvention of the US intelligence community | Brookings
  130. Predictive policing test substantially reduces crime
  131. Full article: The Effectiveness of Big Data-Driven Predictive Policing
  132. How Federated Learning Protects Privacy - People + AI Research
  133. Does regulation hurt innovation? This study says yes - MIT Sloan
  134. Frontiers: The Intended and Unintended Consequences of Privacy ...
  135. Ethics and discrimination in artificial intelligence-enabled ... - Nature
  136. [PDF] ALGORITHMIC BIAS - The Greenlining Institute
  137. Big Data's Causation and Correlation Issue | The TIBCO Blog
  138. [PDF] Causal Models
  139. Exaggerated false positives by popular differential expression ...
  140. Bias in machine learning models can be significantly mitigated ... - NIH
  141. The relationship between artificial intelligence, big data, and ...
  142. The impact of the EU General data protection regulation on product ...
  143. Is GDPR the Right Model for the U.S.? | Regulatory Studies Center
  144. Catch-up with the US or prosper below the tech frontier? An EU ...
  145. The Hidden Costs of Data Privacy Laws for Small Businesses
  146. Compliance in Numbers: The Cost of GDPR/CCPA Violations
  147. Experimental evidence of massive-scale emotional contagion ...
  148. Facebook emotion study breached ethical guidelines, researchers say
  149. Facebook's Emotion Experiment: Implications for Research Ethics
  150. [PDF] ON THE PROPERTIZATION OF DATA AND THE HARMONIZATION ...
  151. US vs EU AI Playbooks – Deregulation vs Trustworthy‑by‑Design
  152. Artificial Intelligence Regulation in 2024: Examining the US's Market ...
  153. OpenAI GPT-3: Everything You Need to Know [Updated] - Springboard
  154. Caution: ChatGPT Doesn't Know What You Are Asking and ... - NIH
  155. The 10 Most Powerful Data Trends That Will Transform Business In ...
  156. Unleashing the Potential of Big Data Predictive Analytics | Pecan AI
  157. 3 Questions: The pros and cons of synthetic data in AI | MIT News
  158. AI in the workplace: A report for 2025 - McKinsey
  159. Edge Computing for IoT - IBM
  160. Edge Computing and IoT: Key Benefits & Use Cases - TierPoint
  161. Big Data Defined: Examples and Benefits | Google Cloud
  162. Streaming Analytics: Intro, Tools & Use Cases - Confluent
  163. 2020s are the decade of commercial quantum computing, says IBM
  164. What is quantum computing? - McKinsey
  165. [PDF] Infographic: The AI Data Cycle - Western Digital
  166. Worldwide IDC Global DataSphere Forecast, 2024–2028
  167. Big Data and Analytics Global Market Report
  168. Big Data Analytics Market Size, Value & Share Analysis
  169. Big Data Market Report by MarketsandMarkets
User Avatar
No comments yet.