Hash array mapped trie
View on WikipediaThis article may be too technical for most readers to understand. (October 2019) |
A hash array mapped trie[1] (HAMT, /ˈhæmt/) is an implementation of an associative array that combines the characteristics of a hash table and an array mapped trie.[1] It is a refined version of the more general notion of a hash tree.
Operation
[edit]A HAMT is an array mapped trie where the keys are first hashed to ensure an even distribution of keys and a constant key length.
In a typical implementation of HAMT's array mapped trie, each node contains a table with some fixed number N of slots with each slot containing either a nil pointer or a pointer to another node. N is commonly 32. As allocating space for N pointers for each node would be expensive, each node instead contains a bitmap which is N bits long where each bit indicates the presence of a non-nil pointer. This is followed by an array of pointers equal in length to the number of ones in the bitmap (its Hamming weight).
Advantages of HAMTs
[edit]The hash array mapped trie achieves almost hash table-like speed while using memory much more economically.[citation needed] Also, a hash table may have to be periodically resized, an expensive operation, whereas HAMTs grow dynamically. Generally, HAMT performance is improved by a larger root table with some multiple of N slots; some HAMT variants allow the root to grow lazily[1] with negligible impact on performance.
Implementation details
[edit]Implementation of a HAMT involves the use of the population count function, which counts the number of ones in the binary representation of a number. This operation is available in many instruction set architectures, but it is available in only some high-level languages. Although population count can be implemented in software in O(1) time using a series of shift and add instructions, doing so may perform the operation an order of magnitude slower.[citation needed]
Implementations
[edit]The programming languages Clojure,[2] Scala, and Frege[3] use a persistent variant of hash array mapped tries for their native hash map type. The Haskell library "unordered-containers" uses the same to implement persistent map and set data structures.[4] Another Haskell library "stm-containers" adapts the algorithm for use in the context of software transactional memory.[5] A Javascript HAMT library [6] based on the Clojure implementation is also available. The Rubinius[7] implementation of Ruby includes a HAMT, mostly written in Ruby but with 3[8] primitives. Large maps in Erlang use a persistent HAMT representation internally since release 18.0.[9] The Pony programming language uses a HAMT for the hash map in its persistent collections package.[10] The im and im-rc crates, which provide persistent collection types for the Rust programming language, use a HAMT for their persistent hash tables and hash sets. [11]
The concurrent lock-free version[12] of the hash trie called Ctrie is a mutable thread-safe implementation which ensures progress. The data-structure has been proven to be correct[13] - Ctrie operations have been shown to have the atomicity, linearizability and lock-freedom properties.
See also
[edit]References
[edit]- ^ a b c Phil Bagwell (2000). Ideal Hash Trees (PDF) (Report). Infoscience Department, École Polytechnique Fédérale de Lausanne.
- ^ "clojure/clojure". GitHub. 8 December 2022.
- ^ "Frege/frege". GitHub. 7 December 2022.
- ^ Johan Tibell, A. Announcing unordered-containers 0.2
- ^ Nikita Volkov, Announcing the "stm-containers" library, 2014
- ^ "mattbierner/hamt". GitHub. 27 November 2022.
- ^ "Ruby source file of Rubinius's HAMT". GitHub.
- ^ https://github.com/rubinius/rubinius/blob/master/machine/builtin/system.cpp#L1724-L1802 [dead link]
- ^ "Erlang Programming Language".
- ^ "horse: Pony is an open-source, actor-model, capabilities-secure, high performance programming language: Ponylang/ponyc". GitHub. 2018-11-26.
- ^ "API docs for Rust im-rc crate".
- ^ Prokopec, A. Implementation of Concurrent Hash Tries on GitHub
- ^ Prokopec, A. et al. (2011) Cache-Aware Lock-Free Concurrent Hash Tries. Technical Report, 2011.
Hash array mapped trie
View on GrokipediaOverview
Definition
A hash array mapped trie (HAMT) is a persistent data structure that implements associative arrays, such as maps or sets, by combining the indexing efficiency of hash tables with the path compression of tries. It organizes key-value pairs in a tree where the path to each entry is determined by the bits of a hash value computed from the key, enabling O(1) average-case operations with bounded worst-case performance independent of the number of elements. This structure avoids the resizing issues of traditional hash tables by using a fixed-width branching factor (typically 32) and sparse node representations, making it particularly suitable for immutable collections in functional programming languages.[5][6] Key features of a HAMT include its use of key hashes to define key-independent traversal paths, which handle collisions by extending the hash with additional bits as needed during insertion. To save space in sparse scenarios, internal nodes employ bitmap-indexed arrays for child pointers: a 32-bit bitmap indicates which of the 32 possible slots are occupied, followed by a variable-length array containing only the present sub-nodes or leaf values. Immutability is supported through path copying, where updates create new versions of modified nodes along the access path while sharing unchanged subtrees, facilitating structural sharing across multiple collection versions.[5][6] A basic HAMT node can be conceptually represented as follows, where the bitmap encodes occupied slots and the array holds references:Node {
bitmap: uint32 // Bit i set if slot i is occupied
children: [array](/page/Array)[Node or KeyValue] // Length up to popcount([bitmap](/page/Bitmap)), up to 32 entries
}
For example, with a bitmap of 0b00000000000000000000000000000101 (binary 5), slots 0 and 2 are occupied, so the children array has two entries pointing to substructures derived from the corresponding hash bits.[5]
Unlike an array-mapped trie (AMT), which indexes paths directly using bits from the keys themselves for ordered or dense data like sequences, a HAMT relies on hash codes to create unordered, collision-resistant paths suitable for arbitrary keys in associative arrays. This hashing decouples the structure from key ordering, improving efficiency for dynamic, sparse sets.[6]
History
The hash array mapped trie (HAMT) was invented by Phil Bagwell in 2001 at the École Polytechnique Fédérale de Lausanne (EPFL)'s Programming Methods Laboratory (LAMP), as detailed in his technical report "Ideal Hash Trees."[1] Bagwell's design addressed limitations in traditional hash tables and tries by combining hashing with array-mapped branching to achieve near-ideal performance characteristics, such as O(1) average-case operations independent of table size.[1] Bagwell's motivation stemmed from the demands of functional programming for efficient, persistent data structures that support immutability and structural sharing without performance penalties, particularly in emerging languages like Scala developed at the same institution.[1] The HAMT enabled cheap creation of new versions of the structure upon updates, making it suitable for concurrent and pure functional paradigms. The structure gained prominence through its adoption in Clojure's PersistentHashMap, implemented by Rich Hickey for the language's 1.0 release in May 2009, providing immutable hash maps with efficient persistence. It also influenced Scala's collections library, where a concurrent variant, TrieMap, was introduced in Scala 2.10 in 2012 as a lock-free hash array mapped trie for thread-safe operations. Later extensions, such as those in Michael Steindorfer's 2015 work on optimizing HAMTs for immutable data stores, refined collision handling and space efficiency for broader applications.[7] Key milestones include Bagwell's foundational 2001 report, followed by practical implementations in functional languages by the late 2000s, and widespread integration into production systems throughout the 2010s and 2020s, including refinements for concurrent access and reduced memory overhead in libraries like Scala's, as well as 2021 research on compiler and runtime support for HAMTs.[8]Data Structure
Node Types
In a hash array mapped trie (HAMT), nodes are the fundamental building blocks that enable efficient hashing and traversal while supporting persistence. The structure primarily features internal nodes for branching, leaf nodes for data storage, collision nodes for handling hash conflicts, and mechanisms for expansion to manage growth. Internal nodes act as the core branching elements, each consisting of a fixed-size bitmap—typically 32 bits long, where each bit represents whether a child slot (0 to 31) is occupied—and a dynamic array of pointers to child nodes. The bitmap's bits are set based on 5-bit segments of a key's hash value, allowing quick determination of occupied positions via bit operations like population count (popcount) to index into the child array, which holds exactly as many entries as set bits in the bitmap for space efficiency. This design avoids allocating a full 32-entry array, using only the necessary pointers to sub-nodes.[1] Leaf nodes serve as terminals in the trie, storing the actual key-value pairs once the hash-based path resolution concludes. They contain the full key and associated value, with hash information often retained for verification; during traversal, full key equality checks occur only at this leaf level to confirm matches.[1] Collision nodes address cases where multiple keys produce identical full hash values, preventing overwrite or loss. In the original design, these take the form of a sub-trie that applies additional hash bits to further differentiate keys.[1] Some implementations use a linear structure like a list or array of key-value entries, where resolution relies on complete key comparisons rather than hashing.[2] When an internal node's bitmap fills completely (all 32 bits set), expansion occurs by introducing a new internal node level; existing children are redistributed using the next hash bits, creating a deeper path while preserving the maximum branching factor of 32 per node. This process ensures the trie remains balanced without unbounded node sizes.[1] The following pseudocode illustrates a basic node layout in a C-like syntax, highlighting the bitmap and dynamic array for internal nodes, with simplified variants for others:typedef struct Node Node;
struct InternalNode {
uint32_t bitmap; // 32-bit bitmap for occupied slots
Node* children[32]; // Dynamic array, actual size = __builtin_popcount(bitmap)
};
struct LeafNode {
uint32_t hash; // Full hash for verification
Key key; // Actual key
Value value; // Associated value
};
struct CollisionNode {
uint32_t hash; // Shared collision hash
Entry<Key, Value> entries[]; // Array or list of colliding pairs; size >1
// Alternatively: sub-trie root for further hashing
};
Internal and collision nodes derive from a common Node base, enabling uniform traversal.[1]
Hashing and Traversal
In a hash array mapped trie (HAMT), the hashing process begins with computing a fixed-width hash value for each key, typically a 32-bit or 64-bit integer, to enable path indexing through the trie structure.[1] Common hash functions include non-cryptographic algorithms such as Universal Hash or those producing uniform distributions like MurmurHash and FNV, which generate bit sequences that guide traversal without requiring key string comparisons at higher levels.[1][7] In implementations like Clojure's PersistentHashMap, hash codes are computed using utility functions that may invoke the JavahashCode() method for objects. In contrast, Scala's HAMT applies bit-spreading to the hash codes to enhance distribution for the 5-bit indexing scheme.[7]
The traversal algorithm extracts successive bit slices from the hash to navigate the sparse array nodes. For a branching factor of 32 (common in 32-bit hash setups), 5 bits are taken at a time starting from the most significant bits; these bits form an index that is used to probe a node's bitmap, which marks occupied slots.[1] The bitmap, a 32-bit integer where each bit corresponds to a potential child, allows efficient location of the child offset via a population count (CTPOP) operation on the bits up to the index position.[1] The shift value advances by 5 bits for each level (e.g., shift = level * 5), enabling descent into sub-tries until the leaf or a collision is reached.[7]
Collisions occur when multiple keys produce identical hash prefixes, leading the traversal to a shared path that terminates in a collision node.[1] In such cases, resolution proceeds by descending into the collision node and using the remaining hash bits or direct key equality checks to distinguish entries, avoiding full rehashing unless necessary.[7] This approach maintains the trie's efficiency by localizing collisions at the leaves.
The following pseudocode illustrates a basic traversal function for lookup, assuming a 32-bit hash and 5-bit slices:
function traverse(node, key, hash, shift):
if node is leaf:
return node if key equals stored key else null
index = (hash >> shift) & 31 // Extract 5 bits
bit = 1 << index
if (node.bitmap & bit) == 0:
return null // No child at this index
child_index = popcount(node.bitmap & (bit - 1)) // Offset via CTPOP
child = node.children[child_index]
if child is collision node:
return find_in_collision(child, key) // Use key equality
else:
return traverse(child, key, hash, shift + 5)
This algorithm, rooted in the original HAMT design, iterates through levels until resolution.[1][7]
Operations
Insertion
Insertion in a hash array mapped trie (HAMT) begins by computing a fixed-width hash value for the input key, typically 32 bits using a universal hashing function to minimize collisions.[1] This hash guides traversal from the root node, consuming 5 bits at each level to determine the branch index (0 to 31), enabling a branching factor of up to 32.[7] The process follows the search path until reaching a leaf node or an empty slot; if the key already exists at the leaf, the value is updated by creating a new leaf node with the same hash but the new value, without altering the structure.[7] If the slot is empty, a new leaf node containing the key-value pair is inserted.[1] To maintain persistence, insertion employs path copying: a new version of each node along the traversal path is created, while unchanged subtrees are shared with the original structure.[7] This results in O(log_{32} n) new nodes, approximately 5 to 7 for typical 32-bit hashes, ensuring that the original trie remains unmodified and multiple versions can coexist efficiently.[7] Internal nodes use a 32-bit bitmap to sparsely represent present children, where each bit corresponds to a potential hash-derived index.[1] During insertion, if the target bit in the bitmap is unset, it is set (via bitwise OR), and the child array is expanded by inserting the new child at the position determined by the population count of bits lower than the target bit.[7] Expansions occur when a collision is detected at a leaf level, such as differing keys sharing the same hash prefix up to that depth. In this case, the existing leaf is replaced by a new internal node, with the old key-value pair and the new one placed as children in a sub-trie using the next 5 hash bits for indexing; this may deepen the trie beyond the initial hash width if necessary, recomputing hashes as needed.[1] The child array in internal nodes remains fixed at size up to 32 but is sparsely populated, avoiding full resizes—instead, full bitmaps (all 32 bits set) simply recurse deeper without splitting, as further bits distinguish keys.[1] The insertion is typically implemented as a recursive function that returns a new root node, copying nodes only on the mutation path:function insert(node, key, value, shift = 0):
if node is Leaf:
if node.key == key:
return new Leaf(key, value) // overwrite value
else:
// create collision sub-trie
bit = (hash(key) >>> shift) & 31
new_internal = new Internal(bitmap = 0, children = [])
// add old leaf as child for its bit
old_bit = (hash(node.key) >>> shift) & 31
insert_child(new_internal, old_bit, node, shift + 5)
// add new leaf
insert_child(new_internal, bit, new Leaf(key, value), shift + 5)
return new_internal
else: // Internal node
bit = (hash(key) >>> shift) & 31
index = popcount(node.bitmap & ((1 << bit) - 1))
if (node.bitmap & (1 << bit)) != 0:
// bit set, recurse to existing child
new_child = insert(node.children[index], key, value, shift + 5)
if new_child == node.children[index]:
return node // no change
else:
// copy path: new node with updated child
new_children = copy(node.children)
new_children[index] = new_child
return new Internal(node.bitmap, new_children)
else:
// bit unset, create new leaf
new_leaf = new Leaf(key, value)
new_bitmap = node.bitmap | (1 << bit)
new_index = popcount(new_bitmap & ((1 << bit) - 1))
new_children = copy(node.children)
insert_into_array(new_children, new_index, new_leaf) // shift existing
return new Internal(new_bitmap, new_children)
This recursive approach ensures structural sharing and immutability, with array copies limited to the path length.[7] Duplicates are handled solely by value replacement at the leaf, incurring no additional node creation beyond the path copy.[7]
Lookup
The lookup operation in a hash array mapped trie (HAMT) retrieves the value associated with a given key by traversing the structure based on the key's hash, without modifying the trie. This process begins by computing a fixed-width hash (typically 32 or 64 bits) of the key to determine the path through the trie levels.[1] The traversal proceeds level by level, using successive bit chunks (commonly 5 bits) from the hash to index into sparse array-mapped nodes, which are compressed via bitmaps to save space.[7] At each level, the selected bit chunk identifies a position in the node's bitmap; if the corresponding bit is unset, indicating no child at that index, the lookup fails immediately and returns null or undefined. If a child exists, the process advances to that sub-node (potentially an array entry, sub-trie, or leaf) by calculating the offset using the population count of set bits below the target position in the bitmap. This continues until reaching a leaf node or collision node at the trie's depth. Node traversal details, such as bitmap indexing, are covered in the Hashing and Traversal section.[1][7] Upon reaching the endpoint, if the node is a leaf, the full key is compared for equality; a match yields the associated value, while a mismatch returns not found. In cases of hash collisions—where multiple keys share the same hash prefix leading to a collision node—the lookup performs a linear search through a list or sub-trie of key-value pairs, again comparing full keys until a match is found or all are exhausted. The operation concludes by returning the value if present, or null/undefined otherwise, ensuring no structural changes.[1][7] The following pseudocode illustrates an iterative lookup in a typical 32-bit HAMT with 5-bit chunks and bitmap-indexed nodes:function lookup(root, key):
if root is null:
return null // Empty trie
hash = hashFunction(key) // 32-bit hash
node = root
shift = 0
while shift < 32:
bitIndex = (hash >>> shift) & 31 // Extract 5 bits (0-31)
if not bitmapHasBit(node.bitmap, bitIndex):
return null // No child at this index
childIndex = popcount(node.bitmap & ((1 << bitIndex) - 1)) // Offset via popcount
node = node.children[childIndex]
if node is leaf:
if node.key == key:
return node.value
else:
return null
elif node is collision:
for pair in node.pairs:
if pair.key == key:
return pair.value
return null
shift += 5
return null // Should not reach if properly structured
This algorithm detects misses early during traversal, contributing to average O(1) time complexity.[1]
Edge cases include an empty trie, where the root is null and lookup immediately returns null, and non-existent keys, which fail either mid-traversal (missing child) or at the endpoint (key mismatch). In collision nodes, exhaustive search ensures correctness even with hash collisions.[1][7]
Removal
The removal operation in a hash array mapped trie (HAMT) deletes a specified key-value pair while maintaining the structure's immutability and persistence in functional implementations. The process begins by traversing the trie from the root to the location of the key, using the key's hash value to index into bitmap-indexed nodes along the path, similar to the lookup operation but with subsequent modifications for deletion. This traversal identifies the leaf node or collision node containing the key.[7] At the target node, if it is a leaf and the key matches, an empty node is returned to indicate removal. If it is a collision node, a new collision node is created with the matching key-value pair excised; if the collision becomes empty, an empty node is returned instead. Upon removal, the algorithm backtracks up the path, potentially contracting the structure to maintain efficiency. If a child is now empty, the corresponding bit in the bitmap is cleared, and the child array is compacted by removing the empty slot. If a node has only one child after removal (singleton path), it may be collapsed by inlining the child directly into the parent, reducing depth; this contraction is optional in basic implementations but common in optimized variants to avoid unnecessary levels. Empty subtrees are pruned by returning an empty node sentinel. Path copying ensures persistence: new nodes are allocated only along the modified path, sharing unchanged subtrees with the original trie to minimize space overhead. If the root becomes empty after contraction, a new empty root is returned. For nodes with full bitmaps (all 32 children present), removal recurses deeper using further hash bits without splitting.[7][1] The removal is typically implemented recursively, with each call returning a new node representing the updated subtree. The following pseudocode outlines a simplified recursive remove function for a persistent HAMT with 5-bit hash chunks, bitmap-indexed array nodes, and leaf/collision children:function remove(node, key, shift = 0):
if node is Empty:
return Empty
if node is Leaf:
if node.key == key:
return Empty // Remove matching leaf
else:
return node // No match
if node is Collision:
new_pairs = []
for pair in node.pairs:
if pair.key != key:
new_pairs.append(pair)
if new_pairs.empty():
return Empty
else:
return new Collision(new_pairs)
// Internal node
bit = (hash(key) >>> shift) & 31
index = popcount(node.bitmap & ((1 << bit) - 1))
if (node.bitmap & (1 << bit)) == 0:
return node // No child at bit
child = node.children[index]
new_child = remove(child, key, shift + 5)
if new_child == child:
return node // Unchanged
if new_child is Empty:
// Prune empty child: clear bit, compact array
new_bitmap = node.bitmap & ~(1 << bit)
new_children = copy(node.children)
remove_from_array(new_children, index) // Shift subsequent
if new_bitmap == 0:
return Empty // Node now empty
// Optional contraction: if only one child left, inline it
if popcount(new_bitmap) == 1:
remaining_bit = lowest_set_bit(new_bitmap)
remaining_index = 0 // Since only one
return new_child_of_remaining // Inline logic simplified
return new Internal(new_bitmap, new_children)
else:
// Update with new child; no contraction needed unless singleton
new_children = copy(node.children)
new_children[index] = new_child
// Optional contraction if singleton
if node.children.length == 1 and not changed before:
return new_child // Inline
return new Internal(node.bitmap, new_children)
This recursive approach handles removal from leaves and collisions, propagates prunings upward, and optionally contracts singletons, ensuring O(log_{32} n) time complexity in practice due to bounded depth.[7]