Community hub
Recent from talks
Contribute something
Nothing was collected or created yet.
Search engine indexing
View on WikipediaSearch engine indexing is the collecting, parsing, and storing of data to facilitate fast and accurate information retrieval. Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, and computer science. An alternate name for the process, in the context of search engines designed to find web pages on the Internet, is web indexing.
Popular search engines focus on the full-text indexing of online, natural language documents.[1] Media types such as pictures, video, audio,[2] and graphics[3] are also searchable.
Meta search engines reuse the indices of other services and do not store a local index whereas cache-based search engines permanently store the index along with the corpus. Unlike full-text indices, partial-text services restrict the depth indexed to reduce index size. Larger services typically perform indexing at a predetermined time interval due to the required time and processing costs, while agent-based search engines index in real time.
Indexing
[edit]The purpose of storing an index is to optimize speed and performance in finding relevant documents for a search query. Without an index, the search engine would scan every document in the corpus, which would require considerable time and computing power. For example, while an index of 10,000 documents can be queried within milliseconds, a sequential scan of every word in 10,000 large documents could take hours. The additional computer storage required to store the index, as well as the considerable increase in the time required for an update to take place, are traded off for the time saved during information retrieval.
Index design factors
[edit]Major factors in designing a search engine's architecture include:
- Merge factors
- How data enters the index, or how words or subject features are added to the index during text corpus traversal, and whether multiple indexers can work asynchronously. The indexer must first check whether it is updating old content or adding new content. Traversal typically correlates to the data collection policy. Search engine index merging is similar in concept to the SQL Merge command and other merge algorithms.[4]
- Storage techniques
- How to store the index data, that is, whether information should be data compressed or filtered.
- Index size
- How much computer storage is required to support the index.
- Lookup speed
- How quickly a word can be found in the inverted index. The speed of finding an entry in a data structure, compared with how quickly it can be updated or removed, is a central focus of computer science.
- Maintenance
- How the index is maintained over time.[5]
- Fault tolerance
- How important it is for the service to be reliable. Issues include dealing with index corruption, determining whether bad data can be treated in isolation, dealing with bad hardware, partitioning, and schemes such as hash-based or composite partitioning,[6] as well as replication.
Index data structures
[edit]Search engine architectures vary in the way indexing is performed and in methods of index storage to meet the various design factors.
- Suffix tree
- Figuratively structured like a tree, supports linear time lookup. Built by storing the suffixes of words. The suffix tree is a type of trie. Tries support extendible hashing, which is important for search engine indexing.[7] Used for searching for patterns in DNA sequences and clustering. A major drawback is that storing a word in the tree may require space beyond that required to store the word itself.[8] An alternate representation is a suffix array, which is considered to require less virtual memory and supports data compression such as the BWT algorithm.
- Inverted index
- Stores a list of occurrences of each atomic search criterion,[9] typically in the form of a hash table or binary tree.[10][11]
- Citation index
- Stores citations or hyperlinks between documents to support citation analysis, a subject of bibliometrics.
- n-gram index
- Stores sequences of length of data to support other types of retrieval or text mining.[12]
- Document-term matrix
- Used in latent semantic analysis, stores the occurrences of words in documents in a two-dimensional sparse matrix.
Challenges in parallelism
[edit]A major challenge in the design of search engines is the management of serial computing processes. There are many opportunities for race conditions and coherent faults. For example, a new document is added to the corpus and the index must be updated, but the index simultaneously needs to continue responding to search queries. This is a collision between two competing tasks. Consider that authors are producers of information, and a web crawler is the consumer of this information, grabbing the text and storing it in a cache (or corpus). The forward index is the consumer of the information produced by the corpus, and the inverted index is the consumer of information produced by the forward index. This is commonly referred to as a producer-consumer model. The indexer is the producer of searchable information and users are the consumers that need to search. The challenge is magnified when working with distributed storage and distributed processing. In an effort to scale with larger amounts of indexed information, the search engine's architecture may involve distributed computing, where the search engine consists of several machines operating in unison. This increases the possibilities for incoherency and makes it more difficult to maintain a fully synchronized, distributed, parallel architecture.[13]
Inverted indices
[edit]Many search engines incorporate an inverted index when evaluating a search query to quickly locate documents containing the words in a query and then rank these documents by relevance. Because the inverted index stores a list of the documents containing each word, the search engine can use direct access to find the documents associated with each word in the query in order to retrieve the matching documents quickly. The following is a simplified illustration of an inverted index:
| Word | Documents |
|---|---|
| the | Document 1, Document 3, Document 4, Document 5, Document 7 |
| cow | Document 2, Document 3, Document 4 |
| says | Document 5 |
| moo | Document 7 |
This index can only determine whether a word exists within a particular document, since it stores no information regarding the frequency and position of the word; it is therefore considered to be a Boolean index. Such an index determines which documents match a query but does not rank matched documents. In some designs the index includes additional information such as the frequency of each word in each document or the positions of a word in each document.[14] Position information enables the search algorithm to identify word proximity to support searching for phrases; frequency can be used to help in ranking the relevance of documents to the query. Such topics are the central research focus of information retrieval.
The inverted index is a sparse matrix, since not all words are present in each document. To reduce computer storage memory requirements, it is stored differently from a two dimensional array. The index is similar to the term document matrices employed by latent semantic analysis. The inverted index can be considered a form of a hash table. In some cases the index is a form of a binary tree, which requires additional storage but may reduce the lookup time. In larger indices the architecture is typically a distributed hash table.[15]
Implementation of Phrase Search Using an Inverted Index
[edit]For phrase searching, a specialized form of an inverted index called a positional index is used. A positional index not only stores the ID of the document containing the token but also the exact position(s) of the token within the document in the postings list. The occurrences of the phrase specified in the query are retrieved by navigating these postings list and identifying the indexes at which the desired terms occur in the expected order (the same as the order in the phrase). So if we are searching for occurrence of the phrase "First Witch", we would:
- Retrieve the postings list for "first" and "witch"
- Identify the first time that "witch" occurs after "first"
- Check that this occurrence is immediately after the occurrence of "first".
- If not, continue to the next occurrence of "first".
The postings lists can be navigated using a binary search in order to minimize the time complexity of this procedure.[16]
Index merging
[edit]The inverted index is filled via a merge or rebuild. A rebuild is similar to a merge but first deletes the contents of the inverted index. The architecture may be designed to support incremental indexing,[17] where a merge identifies the document or documents to be added or updated and then parses each document into words. For technical accuracy, a merge conflates newly indexed documents, typically residing in virtual memory, with the index cache residing on one or more computer hard drives.
After parsing, the indexer adds the referenced document to the document list for the appropriate words. In a larger search engine, the process of finding each word in the inverted index (in order to report that it occurred within a document) may be too time consuming, and so this process is commonly split up into two parts, the development of a forward index and a process which sorts the contents of the forward index into the inverted index. The inverted index is so named because it is an inversion of the forward index.
The forward index
[edit]The forward index stores a list of words for each document. The following is a simplified form of the forward index:
| Document | Words |
|---|---|
| Document 1 | the,cow,says,moo |
| Document 2 | the,cat,and,the,hat |
| Document 3 | the,dish,ran,away,with,the,spoon |
The rationale behind developing a forward index is that as documents are parsed, it is better to intermediately store the words per document. The delineation enables asynchronous system processing, which partially circumvents the inverted index update bottleneck.[18] The forward index is sorted to transform it to an inverted index. The forward index is essentially a list of pairs consisting of a document and a word, collated by the document. Converting the forward index to an inverted index is only a matter of sorting the pairs by the words. In this regard, the inverted index is a word-sorted forward index.
Compression
[edit]Generating or maintaining a large-scale search engine index represents a significant storage and processing challenge. Many search engines utilize a form of compression to reduce the size of the indices on disk.[19] Consider the following scenario for a full text, Internet search engine.
- It takes 8 bits (or 1 byte) to store a single character. Some encodings use 2 bytes per character[20][21]
- The average number of characters in any given word on a page may be estimated at 5 (Wikipedia:Size comparisons)
Given this scenario, an uncompressed index (assuming a non-conflated, simple, index) for 2 billion web pages would need to store 500 billion word entries. At 1 byte per character, or 5 bytes per word, this would require 2500 gigabytes of storage space alone.[citation needed] This space requirement may be even larger for a fault-tolerant distributed storage architecture. Depending on the compression technique chosen, the index can be reduced to a fraction of this size. The tradeoff is the time and processing power required to perform compression and decompression.[citation needed]
Notably, large scale search engine designs incorporate the cost of storage as well as the costs of electricity to power the storage. Thus compression is a measure of cost.[citation needed]
Document parsing
[edit]Document parsing breaks apart the components (words) of a document or other form of media for insertion into the forward and inverted indices. The words found are called tokens, and so, in the context of search engine indexing and natural language processing, parsing is more commonly referred to as tokenization. It is also sometimes called word boundary disambiguation, tagging, text segmentation, content analysis, text analysis, text mining, concordance generation, speech segmentation, lexing, or lexical analysis. The terms 'indexing', 'parsing', and 'tokenization' are used interchangeably in corporate slang.
Natural language processing is the subject of continuous research and technological improvement. Tokenization presents many challenges in extracting the necessary information from documents for indexing to support quality searching. Tokenization for indexing involves multiple technologies, the implementation of which are commonly kept as corporate secrets.[citation needed]
Challenges in natural language processing
[edit]- Word boundary ambiguity
- Native English speakers may at first consider tokenization to be a straightforward task, but this is not the case with designing a multilingual indexer. In digital form, the texts of other languages such as Chinese or Japanese represent a greater challenge, as words are not clearly delineated by whitespace. The goal during tokenization is to identify words for which users will search. Language-specific logic is employed to properly identify the boundaries of words, which is often the rationale for designing a parser for each language supported (or for groups of languages with similar boundary markers and syntax).
- Language ambiguity
- To assist with properly ranking matching documents, many search engines collect additional information about each word, such as its language or lexical category (part of speech). These techniques are language-dependent, as the syntax varies among languages. Documents do not always clearly identify the language of the document or represent it accurately. In tokenizing the document, some search engines attempt to automatically identify the language of the document.
- Diverse file formats
- In order to correctly identify which bytes of a document represent characters, the file format must be correctly handled. Search engines that support multiple file formats must be able to correctly open and access the document and be able to tokenize the characters of the document.
- Faulty storage
- The quality of the natural language data may not always be perfect. An unspecified number of documents, particularly on the Internet, do not closely obey proper file protocol. Binary characters may be mistakenly encoded into various parts of a document. Without recognition of these characters and appropriate handling, the index quality or indexer performance could degrade.
Tokenization
[edit]Unlike literate humans, computers do not understand the structure of a natural language document and cannot automatically recognize words and sentences. To a computer, a document is only a sequence of bytes. Computers do not 'know' that a space character separates words in a document. Instead, humans must program the computer to identify what constitutes an individual or distinct word referred to as a token. Such a program is commonly called a tokenizer or parser or lexer. Many search engines, as well as other natural language processing software, incorporate specialized programs for parsing, such as YACC or Lex.
During tokenization, the parser identifies sequences of characters that represent words and other elements, such as punctuation, which are represented by numeric codes, some of which are non-printing control characters. The parser can also identify entities such as email addresses, phone numbers, and URLs. When identifying each token, several characteristics may be stored, such as the token's case (upper, lower, mixed, proper), language or encoding, lexical category (part of speech, like 'noun' or 'verb'), position, sentence number, sentence position, length, and line number.
Language recognition
[edit]If the search engine supports multiple languages, a common initial step during tokenization is to identify each document's language; many of the subsequent steps are language dependent (such as stemming and part of speech tagging). Language recognition is the process by which a computer program attempts to automatically identify, or categorize, the language of a document. Other names for language recognition include language classification, language analysis, language identification, and language tagging. Automated language recognition is the subject of ongoing research in natural language processing. Finding which language the words belongs to may involve the use of a language recognition chart.
Format analysis
[edit]If the search engine supports multiple document formats, documents must be prepared for tokenization. The challenge is that many document formats contain formatting information in addition to textual content. For example, HTML documents contain HTML tags, which specify formatting information such as new line starts, bold emphasis, and font size or style. If the search engine were to ignore the difference between content and 'markup', extraneous information would be included in the index, leading to poor search results. Format analysis is the identification and handling of the formatting content embedded within documents which controls the way the document is rendered on a computer screen or interpreted by a software program. Format analysis is also referred to as structure analysis, format parsing, tag stripping, format stripping, text normalization, text cleaning and text preparation. The challenge of format analysis is further complicated by the intricacies of various file formats. Certain file formats are proprietary with very little information disclosed, while others are well documented. Common, well-documented file formats that many search engines support include:
- HTML
- ASCII text files (a text document without specific computer readable formatting)
- Adobe's Portable Document Format (PDF)
- PostScript (PS)
- LaTeX
- UseNet netnews server formats
- XML and derivatives like RSS
- SGML
- Multimedia meta data formats like ID3
- Microsoft Word
- Microsoft Excel
- Microsoft PowerPoint
- IBM Lotus Notes
Options for dealing with various formats include using a publicly available commercial parsing tool that is offered by the organization which developed, maintains, or owns the format, and writing a custom parser.
Some search engines support inspection of files that are stored in a compressed or encrypted file format. When working with a compressed format, the indexer first decompresses the document; this step may result in one or more files, each of which must be indexed separately. Commonly supported compressed file formats include:
- ZIP - Zip archive file
- RAR - Roshal ARchive file
- CAB - Microsoft Windows Cabinet File
- Gzip - File compressed with gzip
- BZIP - File compressed using bzip2
- Tape ARchive (TAR), Unix archive file, not (itself) compressed
- TAR.Z, TAR.GZ or TAR.BZ2 - Unix archive files compressed with Compress, GZIP or BZIP2
Format analysis can involve quality improvement methods to avoid including 'bad information' in the index. Content can manipulate the formatting information to include additional content. Examples of abusing document formatting for spamdexing:
- Including hundreds or thousands of words in a section that is hidden from view on the computer screen, but visible to the indexer, by use of formatting (e.g. hidden "div" tag in HTML, which may incorporate the use of CSS or JavaScript to do so).
- Setting the foreground font color of words to the same as the background color, making words hidden on the computer screen to a person viewing the document, but not hidden to the indexer.
Section recognition
[edit]Some search engines incorporate section recognition, the identification of major parts of a document, prior to tokenization. Not all the documents in a corpus read like a well-written book, divided into organized chapters and pages. Many documents on the web, such as newsletters and corporate reports, contain erroneous content and side-sections that do not contain primary material (that which the document is about). For example, articles on the Wikipedia website display a side menu with links to other web pages. Some file formats, like HTML or PDF, allow for content to be displayed in columns. Even though the content is displayed, or rendered, in different areas of the view, the raw markup content may store this information sequentially. Words that appear sequentially in the raw source content are indexed sequentially, even though these sentences and paragraphs are rendered in different parts of the computer screen. If search engines index this content as if it were normal content, the quality of the index and search quality may be degraded due to the mixed content and improper word proximity. Two primary problems are noted:
- Content in different sections is treated as related in the index when in reality it is not
- Organizational side bar content is included in the index, but the side bar content does not contribute to the meaning of the document, and the index is filled with a poor representation of its documents.
Section analysis may require the search engine to implement the rendering logic of each document, essentially an abstract representation of the actual document, and then index the representation instead. For example, some content on the Internet is rendered via JavaScript. If the search engine does not render the page and evaluate the JavaScript within the page, it would not 'see' this content in the same way and would index the document incorrectly. Given that some search engines do not bother with rendering issues, many web page designers avoid displaying content via JavaScript or use the Noscript tag to ensure that the web page is indexed properly. At the same time, this fact can also be exploited to cause the search engine indexer to 'see' different content than the viewer.
HTML priority system
[edit] | This section possibly contains original research. (November 2013) |
Indexing often has to recognize the HTML tags to organize priority. Indexing low priority to high margin to labels like strong and link to optimize the order of priority if those labels are at the beginning of the text could not prove to be relevant. Some indexers like Google and Bing ensure that the search engine does not take the large texts as relevant source due to strong type system compatibility.[22]
Meta tag indexing
[edit]Meta tag indexing plays an important role in organizing and categorizing web content. Specific documents often contain embedded meta information such as author, keywords, description, and language. For HTML pages, the meta tag contains keywords which are also included in the index. Earlier Internet search engine technology would only index the keywords in the meta tags for the forward index; the full document would not be parsed. At that time full-text indexing was not as well established, nor was computer hardware able to support such technology. The design of the HTML markup language initially included support for meta tags for the very purpose of being properly and easily indexed, without requiring tokenization.[23]
As the Internet grew through the 1990s, many brick-and-mortar corporations went 'online' and established corporate websites. The keywords used to describe webpages (many of which were corporate-oriented webpages similar to product brochures) changed from descriptive to marketing-oriented keywords designed to drive sales by placing the webpage high in the search results for specific search queries. The fact that these keywords were subjectively specified was leading to spamdexing, which drove many search engines to adopt full-text indexing technologies in the 1990s. Search engine designers and companies could only place so many 'marketing keywords' into the content of a webpage before draining it of all interesting and useful information. Given that conflict of interest with the business goal of designing user-oriented websites which were 'sticky', the customer lifetime value equation was changed to incorporate more useful content into the website in hopes of retaining the visitor. In this sense, full-text indexing was more objective and increased the quality of search engine results, as it was one more step away from subjective control of search engine result placement, which in turn furthered research of full-text indexing technologies.[citation needed]
In desktop search, many solutions incorporate meta tags to provide a way for authors to further customize how the search engine will index content from various files that is not evident from the file content. Desktop search is more under the control of the user, while Internet search engines must focus more on the full text index.[citation needed]
See also
[edit]References
[edit]- ^ Clarke, C., Cormack, G.: Dynamic Inverted Indexes for a Distributed Full-Text Retrieval System. TechRep MT-95-01, University of Waterloo, February 1995.
- ^ "An Industrial-Strength Audio Search Algorithm" (PDF). Archived from the original (PDF) on 2006-05-12.
- ^ Charles E. Jacobs, Adam Finkelstein, David H. Salesin. Fast Multiresolution Image Querying. Department of Computer Science and Engineering, University of Washington. 1995. Verified Dec 2006
- ^ Brown, E.W.: Execution Performance Issues in Full-Text Information Retrieval. Computer Science Department, University of Massachusetts Amherst, Technical Report 95-81, October 1995.
- ^ Cutting, D., Pedersen, J.: Optimizations for dynamic inverted index maintenance. Proceedings of SIGIR, 405-411, 1990.
- ^ Linear Hash Partitioning. MySQL 5.1 Reference Manual. Verified Dec 2006
- ^ trie, Dictionary of Algorithms and Data Structures, U.S. National Institute of Standards and Technology.
- ^ Gusfield, Dan (1999) [1997]. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. US: Cambridge University Press. ISBN 0-521-58519-8..
- ^ Black, Paul E., inverted index, Dictionary of Algorithms and Data Structures, U.S. National Institute of Standards and Technology Oct 2006. Verified Dec 2006.
- ^ C. C. Foster, Information retrieval: information storage and retrieval using AVL trees, Proceedings of the 1965 20th national conference, p.192-205, August 24–26, 1965, Cleveland, Ohio, United States
- ^ Landauer, W. I.: The balanced tree and its utilization in information retrieval. IEEE Trans. on Electronic Computers, Vol. EC-12, No. 6, December 1963.
- ^ Google Ngram Datasets Archived 2013-09-29 at the Wayback Machine for sale at LDC Catalog
- ^ Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Google, Inc. OSDI. 2004.
- ^ Grossman, Frieder, Goharian. IR Basics of Inverted Index. 2002. Verified Aug 2011.
- ^ Tang, Hunqiang. Dwarkadas, Sandhya. "Hybrid Global Local Indexing for Efficient Peer to Peer Information Retrieval". University of Rochester. Pg 1. http://www.cs.rochester.edu/u/sandhya/papers/nsdi04.ps
- ^ Büttcher, Stefan; Clarke, Charles L. A.; Cormack, Gordon V. (2016). Information retrieval: implementing and evaluating search engines (First MIT Press paperback ed.). Cambridge, Massachusetts London, England: The MIT Press. ISBN 978-0-262-52887-0.
- ^ Tomasic, A., et al.: Incremental Updates of Inverted Lists for Text Document Retrieval. Short Version of Stanford University Computer Science Technical Note STAN-CS-TN-93-1, December, 1993.
- ^ Sergey Brin and Lawrence Page. The Anatomy of a Large-Scale Hypertextual Web Search Engine. Stanford University. 1998. Verified Dec 2006.
- ^ H.S. Heaps. Storage analysis of a compression coding for a document database. 1NFOR, I0(i):47-61, February 1972.
- ^ The Unicode Standard - Frequently Asked Questions. Verified Dec 2006.
- ^ Storage estimates. Verified Dec 2006.
- ^ Google Webmaster Tools, "Hypertext Markup Language 5", Conference for SEO January 2012.
- ^ Berners-Lee, T., "Hypertext Markup Language - 2.0", RFC 1866, Network Working Group, November 1995.
Further reading
[edit]- R. Bayer and E. McCreight. Organization and maintenance of large ordered indices. Acta Informatica, 173-189, 1972.
- Donald E. Knuth. The Art of Computer Programming, volume 1 (3rd ed.): fundamental algorithms, Addison Wesley Longman Publishing Co. Redwood City, CA, 1997.
- Donald E. Knuth. The art of computer programming, volume 3: (2nd ed.) sorting and searching, Addison Wesley Longman Publishing Co. Redwood City, CA, 1998.
- Gerald Salton. Automatic text processing, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1988.
- Gerard Salton. Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986.
- Gerard Salton. Lesk, M.E.: Computer evaluation of indexing and text processing. Journal of the ACM. January 1968.
- Gerard Salton. The SMART Retrieval System - Experiments in Automatic Document Processing. Prentice Hall Inc., Englewood Cliffs, 1971.
- Gerard Salton. The Transformation, Analysis, and Retrieval of Information by Computer, Addison-Wesley, Reading, Mass., 1989.
- Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Chapter 8. ACM Press 1999.
- G. K. Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949.
- Adelson-Velskii, G.M., Landis, E. M.: An information organization algorithm. DANSSSR, 146, 263-266 (1962).
- Edward H. Sussenguth Jr., Use of tree structures for processing files, Communications of the ACM, v.6 n.5, p. 272-279, May 1963
- Harman, D.K., et al.: Inverted files. In Information Retrieval: Data Structures and Algorithms, Prentice-Hall, pp 28–43, 1992.
- Lim, L., et al.: Characterizing Web Document Change, LNCS 2118, 133–146, 2001.
- Lim, L., et al.: Dynamic Maintenance of Web Indexes Using Landmarks. Proc. of the 12th W3 Conference, 2003.
- Moffat, A., Zobel, J.: Self-Indexing Inverted Files for Fast Text Retrieval. ACM TIS, 349–379, October 1996, Volume 14, Number 4.
- Mehlhorn, K.: Data Structures and Efficient Algorithms, Springer Verlag, EATCS Monographs, 1984.
- Mehlhorn, K., Overmars, M.H.: Optimal Dynamization of Decomposable Searching Problems. IPL 12, 93–98, 1981.
- Mehlhorn, K.: Lower Bounds on the Efficiency of Transforming Static Data Structures into Dynamic Data Structures. Math. Systems Theory 15, 1–16, 1981.
- Koster, M.: ALIWEB: Archie-Like indexing in the Web. Computer Networks and ISDN Systems, Vol. 27, No. 2 (1994) 175-182 (also see Proc. First Int'l World Wide Web Conf., Elsevier Science, Amsterdam, 1994, pp. 175–182)
- Serge Abiteboul and Victor Vianu. Queries and Computation on the Web. Proceedings of the International Conference on Database Theory. Delphi, Greece 1997.
- Ian H Witten, Alistair Moffat, and Timothy C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. New York: Van Nostrand Reinhold, 1994.
- A. Emtage and P. Deutsch, "Archie--An Electronic Directory Service for the Internet." Proc. Usenix Winter 1992 Tech. Conf., Usenix Assoc., Berkeley, Calif., 1992, pp. 93–110.
- M. Gray, World Wide Web Wanderer.
- D. Cutting and J. Pedersen. "Optimizations for Dynamic Inverted Index Maintenance." Proceedings of the 13th International Conference on Research and Development in Information Retrieval, pp. 405–411, September 1990.
- Stefan Büttcher, Charles L. A. Clarke, and Gordon V. Cormack. Information Retrieval: Implementing and Evaluating Search Engines Archived 2020-10-05 at the Wayback Machine. MIT Press, Cambridge, Mass., 2010.
Search engine indexing
View on GrokipediaIntroduction
Definition and Process Overview
Search engine indexing is the process of collecting, parsing, and storing document content in a structured format to enable fast and accurate information retrieval in response to user queries. This foundational step transforms unstructured web pages and other documents into an organized repository that supports efficient searching across vast datasets. By preprocessing and indexing content, search engines can quickly match queries to relevant documents without scanning the entire corpus each time.[5] The high-level process begins with document acquisition, often through web crawling, where automated programs systematically fetch pages from the internet to form the input for indexing. Subsequent steps involve parsing the raw documents to extract meaningful content, such as text, followed by tokenization to break content into searchable units like words or terms. The core indexing phase then builds data structures—such as inverted indices as the primary output—that map terms to their locations in documents, culminating in storage of the index for rapid access. This pipeline ensures scalability, allowing search engines to handle billions of documents efficiently.[5][7] Indexing is crucial for modern search engines, enabling relevance ranking algorithms to evaluate and order results based on query-document matches, while supporting advanced features like full-text search across diverse languages and formats. It addresses the web's scale and heterogeneity, where duplicate content can account for up to 40% of pages, by incorporating deduplication and normalization during processing. Key performance metrics include index size, which measures storage demands (e.g., over 100 GB for early large-scale systems indexing tens of millions of pages), query latency, which evolved from 1-10 seconds in early systems to sub-second response times in modern search engines, and trade-offs in recall (completeness of retrieved relevant documents) versus precision (accuracy of those retrieved). These factors underscore indexing's role in balancing speed, comprehensiveness, and resource efficiency.[5][7]Historical Evolution
The origins of search engine indexing trace back to 1990 with the development of Archie, created by Alan Emtage, Bill Heelan, and J. Peter Deutsch at McGill University. Archie was designed to index file names and descriptions from FTP archives, maintaining a centralized database that periodically queried remote servers to update its index and enable keyword-based file searches across the early Internet. [8] This approach represented the first automated effort to organize and retrieve distributed digital content, addressing the limitations of manual directory listings prevalent in the pre-web era. [4] By the mid-1990s, indexing evolved to handle the burgeoning World Wide Web, with AltaVista launching in 1995 as a pioneering full-text search engine. AltaVista employed inverted indices to map terms to their occurrences within crawled web documents, supporting natural language queries and indexing millions of pages on single, high-performance machines. [4] This shift from filename-based to content-based indexing dramatically improved retrieval precision and scale, though it was constrained by hardware limitations that required frequent recrawls to keep indices current. [5] A key milestone occurred in 1998 when Google integrated PageRank into its indexing framework, enhancing index quality by incorporating hyperlink analysis to assess document authority during crawling and ranking. [9] This innovation addressed spam and relevance issues in early web indices. Further scalability advancements arrived in 2004 with Google's introduction of MapReduce, a distributed processing model that parallelized index construction across commodity clusters, enabling the handling of web-scale data volumes. [10] The 2010s saw the rise of real-time indexing to support dynamic social search environments, where systems like Twitter's integrated streaming data into indices for near-instantaneous retrieval of user-generated content. [11] Post-2020 developments emphasized multimodal indexing, exemplified by Google's Multitask Unified Model (MUM) announced in 2021, which processes and indexes text alongside images and other media types to enable cross-modal queries. [12] Subsequent advancements from 2023 onward incorporated large language models like Gemini for improved semantic understanding and indexing of complex, AI-generated content, alongside core updates in 2024 and 2025 that prioritized high-quality, original material to refine index composition amid rising duplicates from generative AI.[13] Throughout this evolution, indexing challenges have scaled dramatically, from the single-machine constraints of 1990s systems managing gigabytes of data to today's petabyte-scale distributed indices supporting hundreds of billions of webpages. [14]Document Processing
Parsing and Tokenization
Parsing in search engine indexing involves converting unstructured or semi-structured documents, such as HTML webpages or PDF files, into a sequential stream of meaningful textual units by identifying and extracting content while discarding irrelevant elements like markup tags or boilerplate.[15] This process typically begins after language detection, which identifies the document's primary language to apply appropriate parsing rules.[15] For instance, HTML parsing uses tools to strip tags and extract body text, while PDF parsing employs layout analysis to reconstruct reading order from visual elements.[15] Tokenization follows parsing by breaking the extracted text into discrete tokens, usually words or subword units, to facilitate indexing and retrieval.[15] Word-based splitting relies on delimiters like spaces and punctuation to segment text, often treating sequences of alphanumeric characters as tokens while removing or isolating punctuation marks such as commas, periods, or hyphens.[15] Stopwords, common function words like "the" or "and" that carry little semantic value, are typically filtered out during this stage to reduce index size and improve query matching efficiency.[15] To normalize tokens for better recall in searches, stemming and lemmatization are applied to reduce variant word forms to a base representation.[16] Stemming, exemplified by the Porter stemmer algorithm, uses heuristic rules to iteratively strip suffixes in five phases, conflating inflected forms like "connecting" and "connection" to "connect" without relying on a dictionary, thereby enhancing information retrieval performance by reducing vocabulary size.[17] Lemmatization, in contrast, employs morphological analysis and part-of-speech context to map words to their dictionary lemma, such as reducing "better" to "good," offering higher accuracy than stemming but at greater computational cost.[16] Challenges in tokenization arise particularly with non-English languages, where ambiguous delimiters like compound words in German or scriptio continua in languages without spaces, such as Thai or Chinese, complicate boundary detection and lead to inefficient token splits.[18] Handling numbers and dates as tokens requires special rules to preserve their integrity, treating them as single units (e.g., "2023-11-10" as one token) rather than splitting on hyphens, to support numeric queries in retrieval.[15] Multilingual tokenization can also introduce biases, with morphologically rich languages requiring more tokens per word, inflating processing costs up to 15 times compared to English.[18] Efficient tokenization algorithms often leverage finite-state transducers (FSTs), which model token boundaries and normalization rules as compact automata for linear-time processing of large document streams in information retrieval pipelines.[19] A basic tokenizer can be implemented with simple pseudocode that iterates over characters, accumulating alphabetic sequences while skipping or handling delimiters:function tokenize(text):
tokens = []
current_token = ""
for char in text:
if char.isalnum():
current_token += char.lower()
elif current_token:
tokens.append(current_token)
current_token = ""
if current_token:
tokens.append(current_token)
return [t for t in tokens if t not in stopwords] # Filter stopwords
function tokenize(text):
tokens = []
current_token = ""
for char in text:
if char.isalnum():
current_token += char.lower()
elif current_token:
tokens.append(current_token)
current_token = ""
if current_token:
tokens.append(current_token)
return [t for t in tokens if t not in stopwords] # Filter stopwords
Language Detection and Format Analysis
Language detection is a critical initial step in search engine indexing, enabling the system to identify the primary language of a document for tailored processing. Traditional methods rely on n-gram models, which analyze sequences of characters or words to compute perplexity scores and match against language-specific profiles, offering simplicity and efficiency for short texts. For instance, weighted n-gram approaches, using byte-level sequences of length 3 to 12, achieve high accuracy (up to 99.2%) across over 1,100 languages by employing cosine similarity and discriminative weighting to filter redundant features. Statistical classifiers, such as the langdetect library, extend this by porting Google's n-gram-based categorization, supporting over 50 languages through frequency-based profiling without requiring extensive training data. More advanced neural approaches, like FastText introduced in 2017, leverage subword n-grams and averaged word embeddings in a linear classifier, attaining approximately 97% accuracy on 176 languages while processing thousands of documents per second.[20] These methods collectively ensure robust identification even for brief or noisy content encountered in web crawling. Format analysis complements language detection by examining document structure and metadata to guide extraction. Search engines detect MIME types primarily through HTTP Content-Type headers or file extensions, distinguishing formats like HTML (text/html) from plain text (text/plain) to apply appropriate parsing rules, as standardized in media type specifications. Encoding handling is equally vital, with UTF-8 serving as the predominant scheme for its compatibility with Unicode, though mismatches—such as undetected legacy encodings like ISO-8859—can lead to garbled text; processors mitigate this by scanning byte patterns or meta declarations to normalize to UTF-8. Boilerplate removal further refines analysis by identifying and stripping non-essential elements like navigation or ads, using techniques such as convolutional neural networks on DOM features to label text blocks, improving content purity and retrieval metrics (e.g., F1 score of 0.90 on benchmarks). Challenges in this phase arise particularly with multilingual documents and format quirks. Multilingual content often mixes languages within a single page, complicating detection; script identification, such as distinguishing Cyrillic from Latin alphabets, requires additional heuristics like character set analysis to avoid misclassification in non-Latin scripts (e.g., CJK or Arabic). JavaScript-rendered content poses format-specific issues, as dynamic loading delays visibility to crawlers, extending indexing times up to ninefold and risking incomplete capture of essential text. These hurdles demand integrated pipelines that prioritize server-side rendering or pre-processing to ensure comprehensive analysis. The outputs of language detection and format analysis directly inform subsequent steps, such as tokenization, by selecting language-specific stopwords—common words like "the" in English or "le" in French—to filter noise during segmentation. For example, tools like NLTK or spaCy use detected languages to apply tailored stopword lists across 20+ languages, enhancing indexing efficiency without overgeneralizing across scripts.Content Extraction and Normalization
Content extraction is a critical step in search engine indexing, where raw documents from diverse formats are processed to isolate the primary textual and structured information relevant for retrieval. For HTML-based web pages, extraction typically begins with DOM parsing, which builds a tree representation of the document's structure to identify and retrieve main content nodes while discarding irrelevant elements such as navigation menus, advertisements, and metadata. This approach allows precise navigation through the hierarchy, enabling the isolation of article bodies or key sections via recursive traversal from the root HTML node.[21] Libraries like Beautiful Soup exemplify practical implementations of DOM parsing in extraction pipelines, offering fault-tolerant handling of malformed HTML common in web crawling. For non-HTML formats, such as PDFs, search engines apply specialized text extraction techniques that decode the document's internal layout and font mappings to reconstruct readable text streams, often preserving spatial relationships for better context. PyMuPDF serves as an efficient library for this purpose, supporting high-fidelity extraction from complex layouts including tables and multi-column text. When documents contain non-text elements like scanned images, optical character recognition (OCR) is employed to convert embedded visuals into searchable text; Google Cloud Vision API, for instance, leverages machine learning to detect and extract printed or handwritten content from images within PDFs or standalone files. Following extraction, normalization standardizes the isolated content to ensure consistency across the index, mitigating variations that could fragment retrieval. Case folding, a foundational technique in information retrieval, converts all text to lowercase to treat variants like "Search Engine" and "search engine" as equivalent, thereby enhancing query matching without introducing undue ambiguity in most cases.[22] Entity normalization extends this by canonicalizing specific elements, such as resolving URLs to their preferred forms (e.g., stripping trailing slashes or parameters) to consolidate duplicates, as guided by rel="canonical" directives in web standards.[23] Duplicate detection further refines the process using probabilistic methods like MinHash, which estimates Jaccard similarity between document sets via locality-sensitive hashing, allowing efficient identification and removal of near-duplicates in large-scale web corpora.[24] Structured data within documents is handled separately to enrich indexing with semantic metadata. Search engines parse formats like meta tags, JSON-LD scripts, and schema.org markup to extract entities such as product details or author information, enabling enhanced features like rich snippets. Google's Programmable Search Engine, for example, directly utilizes schema.org vocabularies to interpret and index this embedded data for improved result presentation.[25] A key challenge in content extraction arises from noisy elements, including advertisements, sidebars, and navigation links, which can dilute the relevance of indexed text. To address this, readability algorithms apply heuristic scoring to DOM nodes based on factors like text density, link prevalence, and tag semantics, prioritizing central content blocks. Mozilla's Readability.js, originating from Arc90's 2007 efforts and refined in the 2010s for Firefox's Reader View, exemplifies such solutions by stripping boilerplate and reconstructing a clean, article-focused view, significantly reducing visual clutter and improving extraction accuracy.[26] Empirical studies confirm that these algorithms boost reading speed by 5% in controlled tests by focusing on core narrative elements.[27]Index Construction
Inverted Index Building
The process of building an inverted index begins with scanning a stream of tokenized documents, where each token is paired with its corresponding document identifier (docID). These pairs, typically in the form (term, docID), are collected from the entire document collection. The pairs are then sorted lexicographically by term (as the primary key) and by docID (as the secondary key) to group occurrences of the same term together. From these sorted pairs, term-document mappings are constructed, where each unique term points to a list of docIDs in which it appears. This mapping forms the core of the inverted index, enabling efficient retrieval of documents containing specific terms.[28] In the resulting data structure, each term is associated with a postings list that records not only the docIDs but also additional metadata such as term frequency (the number of times the term appears in the document) and positional information (the offsets or positions of the term within the document). This allows for advanced query processing, such as phrase or proximity searches. For example, the postings list for the term "cat" might be represented as:"cat": [(doc1, 2, [5, 12]), (doc3, 1, [8])]
"cat": [(doc1, 2, [5, 12]), (doc3, 1, [8])]
Forward Index Construction
Forward index construction occurs during the initial stages of document processing, following parsing and tokenization, where each document is analyzed to extract terms and their positions within the text. As documents are crawled and ingested, the system builds term lists sequentially for each document identifier (docID), capturing the bag-of-words representation including term frequencies and positional offsets to preserve word order. This process involves linguistic preprocessing such as stemming and normalization to standardize terms before storage, ensuring the index reflects the document's core content without redundancy.[31] The resulting data structure is a document-centric mapping, typically formatted as{ docID: [(term1, pos1), (term2, pos2), ...] }, where each entry lists terms paired with their byte or token positions in the document. This structure enables efficient sequential access to a full document's terms, contrasting with the term-centric focus of the inverted index, and supports reconstruction of document content when needed. Its advantages lie in low-overhead updates for individual documents and rapid traversal for tasks requiring complete per-document views, making it suitable for dynamic environments.[31]
Forward indices serve key roles beyond core querying, including support for web crawling through efficient link extraction from stored document structures and snippet generation via positional data to highlight query terms in result previews. For example, in Twitter's search engine (rebranded as X in 2023), forward indices allow efficient access to tweet content by ID.[31][32][33]