Community hub
0 subscribers8 pages, 0 posts
Recent from talks
Be the first to start a discussion here.
Be the first to start a discussion here.
Be the first to start a discussion here.
Be the first to start a discussion here.
Contribute something
Welcome to the community hub built to collect knowledge and have discussions related to HTTrack.
Nothing was collected or created yet.
HTTrack
View on Wikipediafrom Wikipedia
| HTTrack | |
|---|---|
 | |
 | |
| Developer | Xavier Roche[1] |
| Initial release | May 1998[2] |
| Stable release | 3.49.5[3]
/ 27 January 2024 |
| Repository | |
| Written in | C |
| Operating system | Microsoft Windows, macOS, Linux, FreeBSD and Android[4] |
| Type | Offline browser and Web crawler |
| License | GNU General Public License Version 3 |
| Website | www |
HTTrack is a free and open-source Web crawler and offline browser, developed by Xavier Roche and licensed under the GNU General Public License Version 3.
HTTrack allows users to download World Wide Web sites from the Internet to a local computer.[5][6] By default, HTTrack arranges the downloaded site by the original site's relative link-structure. The downloaded (or "mirrored") website can be browsed by opening a page of the site in a browser.
HTTrack can also update an existing mirrored site and resume interrupted downloads. HTTrack is configurable by options and by filters (include/exclude), and has an integrated help system. There is a basic command line version and two GUI versions (WinHTTrack and WebHTTrack); the former can be part of scripts and cron jobs.
HTTrack uses a Web crawler to download a website. Some parts of the website may not be downloaded by default due to the robots exclusion protocol unless disabled during the program. HTTrack can follow links that are generated with basic JavaScript and inside Applets or Flash, but not complex links (generated using functions or expressions) or server-side image maps.
See also
[edit]
References
[edit]- ^ Credits: Greetings & authors
- ^ Roche, Xavier (February 8, 2014). "Re: Full History of HTTrack". HTTrack Forum. Retrieved November 21, 2016.
The first release was in May 1998, but only as binaries.
- ^ "Release 3.49.5".
- ^ HTTrack on Google Play
- ^ Engebretson, Patrick (2011). The Basics of Hacking and Penetration Testing. Elsevier. pp. 19–22. ISBN 9781597496568.
- ^ Beaver, Kevin (2012). Hacking For Dummies. John Wiley & Sons. pp. 278, 280–281. ISBN 9781118380963.
External links
[edit] | This Web-software-related article is a stub. You can help Wikipedia by adding missing information. |
HTTrack
View on Grokipediafrom Grokipedia
HTTrack is a free and open-source offline browser utility that enables users to download entire World Wide Web sites from the Internet to a local directory on their computer, recursively building all directories and retrieving HTML pages, images, and other files while preserving the original site's relative link structure for seamless offline navigation.[1]
Developed by French software engineer Xavier Roche, HTTrack was first released in May 1998, with its codebase transitioning to free software under the GNU General Public License starting with version 2.00 in 2000, and to version 3 under the GNU General Public License version 3 since version 3.45 in 2007.[2][3] The project has evolved through multiple major updates, including enhanced parsing and ETag support in version 3.00 (2002), HTTPS/SSL capabilities in 3.20-2 (2002), and a web-based graphical interface (WebHTTrack) in 3.30 (2004), with the most recent stable release being 3.49-6 in 2025, which addressed security fixes like CVE-2017-14062.[3][4]
Key features of HTTrack include its ability to update existing mirrored sites, resume interrupted downloads, and handle configurable options such as filters for specific file types or depth limits, all supported by an integrated help system and extensive documentation.[1] Written primarily in C, the software is cross-platform, offering command-line versions for Linux, Unix, and BSD systems, as well as a Windows-specific GUI variant called WinHTTrack compatible with Windows 2000 through 10 and later.[1][3] HTTrack's design emphasizes ease of use for mirroring websites without requiring advanced technical knowledge, making it a popular tool for archiving and offline access in various professional and personal contexts.[1]
History
Origins and initial development
HTTrack originated in 1998 as a project led by Xavier Roche, with contributions from Vann Philippot, both affiliated with the Institut Supérieur d'Électronique de Physique et de Chimie Industrielles de l'Université de Caen (ISMRA-ENSI) in Caen, France. The tool was conceived as a straightforward solution for mirroring websites, allowing users to retrieve webpages, graphics, and files from remote sites for offline browsing via a standard web browser. This development addressed the growing need for accessible offline web content during an era when internet connectivity was less reliable and bandwidth limited, enabling recursive downloading without the intricacies of more advanced web crawling tools.[5] The primary motivations behind HTTrack's creation were to provide a user-friendly utility for archiving entire websites locally, preserving their structure and functionality for disconnected use. By supporting features like URL filtering, scheduling, and multithreading to enhance download speed, the developers aimed to simplify the process of website replication, making it suitable for personal, educational, or archival purposes without requiring extensive technical expertise. As a personal and academic endeavor, it emerged from the practical challenges of accessing web resources in resource-constrained environments.[5] The initial public release, version 1.00, took place in May 1998 and was distributed exclusively as precompiled binaries for platforms including Windows 95/98 and Unix-like systems such as Solaris, AIX, and Linux. Early versions, like the beta 1.2 for Windows and 1.16b for Unix, highlighted its cross-platform ambitions from the outset, with the software offered as a free tool—requesting only optional donations such as a PC mouse in lieu of payment. This binary-only distribution reflected the project's nascent stage, focusing on usability over immediate source code availability.[5][6] By 2000, with the release of version 2.00, HTTrack transitioned to an open-source model under the GNU General Public License (GPL), solidifying its status as free software and encouraging community contributions. This shift marked a pivotal moment, aligning the project with broader open-source principles and facilitating wider adoption and modification by users worldwide.[1]Major releases and evolution
HTTrack's evolution has been characterized by periodic major releases that enhanced its core mirroring engine, user interface, and compatibility with evolving web standards. Version 2.00, released in 2000, marked the first GPL-licensed release of the software, establishing a free software model under the GNU General Public License and introducing foundational recursive mirroring capabilities that allowed users to download entire website structures locally.[3] A significant milestone came with version 3.00 in 2002, which underwent a comprehensive overhaul including refined HTML parsing for better structure preservation, initial JavaScript support to handle dynamic elements, and ETag implementation for optimized incremental updates by detecting unchanged resources.[3] In 2004, version 3.30 expanded accessibility with the addition of the WebHTTrack graphical user interface, enabling web-based configuration and monitoring, alongside advanced URL filtering enhancements that permitted more precise control over crawl depth, types of links followed, and content selection criteria.[3] The most recent stable release, version 3.49.6 from March 2025, addressed critical security vulnerabilities such as CVE-2017-14062 related to gzip handling, while incorporating improvements in stability.[3][4] Maintenance of HTTrack continues actively through its official GitHub repository, prioritizing bug fixes, security patches, and compatibility enhancements with modern web protocols and operating systems well into 2025.[2]Features
Core mirroring capabilities
HTTrack's core mirroring process begins with recursive downloading, which systematically retrieves an entire website from the internet to a local directory on the user's computer. This involves fetching HTML pages, images, CSS stylesheets, and other associated files such as JavaScript and multimedia content, while simultaneously constructing a mirrored directory structure that replicates the original site's organization. By default, HTTrack scans linked resources breadth-first, ensuring comprehensive coverage without duplicating files unnecessarily, and applies recursion limits to avoid infinite loops on sites with excessive internal linking or dynamic content. However, users can configure for comprehensive full-site mirroring by adjusting recursion depth, restricting to the same domain, and using appropriate file filters as detailed in the configuration and advanced options section, ensuring the mirror captures an entire static site for offline access.[1][7] A key aspect of this mirroring is the preservation of the original site's relative link structure, allowing users to browse the downloaded content offline as if accessing the live site. HTTrack rewrites absolute URLs to relative paths during the download, enabling standard web browsers to navigate between pages seamlessly without requiring an internet connection or additional server setup. For instance, a link pointing to "/images/logo.png" remains intact relative to its HTML file, maintaining the site's navigational integrity for offline viewing. This feature ensures that the mirror functions as a self-contained archive, ideal for archival purposes or areas with limited connectivity.[1][7] HTTrack supports resuming interrupted downloads and updating existing mirrors to keep local copies current. If a download is paused due to network issues or user intervention, the tool can pick up from the last successful point, avoiding redundant transfers and saving time on large sites. For updates, HTTrack checks for modifications in source files by comparing timestamps or content hashes, downloading only changed elements to refresh the mirror efficiently while preserving the directory structure.[1][7] By default, HTTrack adheres to the robots.txt exclusion protocol, respecting directives that site owners use to prevent automated crawling of specific paths or files. This compliance helps avoid overloading servers and ensures ethical mirroring practices, as the tool skips disallowed sections unless explicitly overridden by the user.[7][8] While HTTrack excels at static content, its handling of dynamic elements is limited to basic detection rather than full execution. It can follow simple JavaScript-generated links or embed Flash and Applet files for download, but it does not render or execute these elements offline, potentially leading to incomplete functionality on interactive sites. For example, forms or animations reliant on server-side processing or runtime JavaScript will not operate in the mirror, emphasizing HTTrack's focus on structural replication over behavioral simulation.[9][7]Configuration and advanced options
HTTrack provides an integrated help system accessible via thehttrack --help command, which details all available options for configuring mirroring tasks, including specifications for output directories, limits on depth and file counts, and file type inclusions or exclusions using pattern rules such as +*.jpg to include JPEG images or -*.exe to exclude executable files.[10] These command-line options allow users to tailor the mirroring process precisely, for instance by setting an output path with -O /path/to/mirror or limiting recursion depth to prevent excessive crawling.[10]
URL filtering rules enable fine-grained control over the download scope, with options like -rN to set a maximum depth (e.g., -r6 for six levels of recursion from the starting URL).[10] Blacklist and whitelist mechanisms use + and - prefixes for domain-specific rules, such as +www.example.com -www.example.com/badsection/* to include a site while excluding problematic subpaths, ensuring targeted captures without unnecessary data. For a complete mirror of an entire website, a high recursion depth (e.g., -r99999), restriction to the same domain using -s1, and acceptance of all files via +*example.com/* are commonly used.[10] Note that while the GUI interface allows setting a maximum number of links to process, the command-line version does not have a direct equivalent for capping total files.[10] Additional limits include maximum transfer rates via -AN (e.g., -A1000 for 1 KB/s) to respect server loads and avoid detection as aggressive scraping.[10]
To download a full website with HTTrack for offline hosting or archiving, use the "Mirror Website(s)" option (default in GUI) or the equivalent command-line mode. This creates a complete local copy with original directory structure and relative links adjusted for offline use. Optionally override robots.txt compliance with --robots=0, but respecting robots.txt is ethically recommended by default. The resulting folder can be hosted via a local web server (e.g., Python http.server module via python -m http.server) for viewing. Republishing requires explicit permissions from the content owner to comply with copyright laws and terms of service.[7]
Proxy support is configured through -P proxy:port (e.g., -P proxy.example.com:8080), with authentication integrated by appending credentials like -P user:pass@proxy:port, facilitating operation behind firewalls or in restricted networks.[10] Rate limiting extends to connection controls, such as -%cN for maximum connections per second (e.g., -%c10), further mitigating overload on remote servers.[10]
Advanced features include MMS streaming capture, introduced in version 3.40 to handle mms:// protocols for media downloads, though this was later removed due to protocol obsolescence.[11] ETag-based incremental updates leverage HTTP ETag headers stored during initial downloads; on subsequent runs with the --update option, HTTrack sends the ETag to the server, receiving a 304 Not Modified response for unchanged files or downloading only modifications, optimizing bandwidth for repeated mirrors.[12] This mechanism depends on server compliance with ETag standards for effective caching.[12]
Configuration files support saving and reusing settings, with system-wide defaults in /etc/httrack.conf and project-specific .hts files generated via --save project_name for loading in future sessions with --load project_name, streamlining complex setups across multiple runs.[10]
Platforms and interfaces
Supported operating systems
HTTrack, an offline browser utility, is designed for cross-platform compatibility, with its core implemented in the C programming language to facilitate portability across various systems using standard libraries without extensive dependencies.[2] The software provides primary support for Microsoft Windows through WinHTTrack, which includes native installers for versions from Windows 2000 to Windows 10 and later, along with shell integration for seamless right-click context menu access to mirroring functions.[1][13] For Linux and other Unix-like systems, including FreeBSD, HTTrack is available as command-line binaries and through package managers such as apt for Debian-based distributions and yum for RPM-based ones, enabling straightforward installation and execution on these environments.[14][15][16] macOS users can achieve compatibility by compiling HTTrack from source or installing it via Homebrew, a popular package manager that provides pre-built binaries for Apple Silicon and Intel-based systems.[14][17] Additionally, an Android port has been available since 2013, offered as a dedicated app on the Google Play Store that supports mobile website mirroring with a touch-friendly interface.[18][19]Available user interfaces
HTTrack provides multiple user interfaces to accommodate different user preferences and environments, ranging from command-line tools for automation to graphical options for easier interaction. The core command-line interface, known as httrack, serves as the foundational tool for interacting with the software, particularly suited for scripting, automation, and advanced users who require precise control over mirroring operations. It supports a wide array of options to customize downloads, such as specifying URLs, file filters, and exclusions; for instance, the commandhttrack "site.com" +*.html -adults initiates a mirror of a website while including HTML files and excluding adult content. This interface is cross-platform and integral to all HTTrack installations, enabling integration into batch scripts or remote server tasks without a graphical environment.[7]
For Windows users, WinHTTrack offers a dedicated graphical user interface (GUI) that simplifies the mirroring process through a wizard-based setup. It allows users to create projects by selecting destination folders, entering website URLs, and configuring options like update modes or filters via intuitive dialogs, while also supporting action scheduling and real-time progress monitoring through visual logs and status indicators. This interface enhances accessibility for non-technical users by abstracting complex command-line parameters into point-and-click interactions.[14]
WebHTTrack provides a browser-based GUI alternative, accessible via a local web server launched by the software, making it ideal for Linux, Unix, or server-based environments where desktop GUIs may not be available. Users can configure mirrors, set filters, and monitor downloads through a web interface that mimics the WinHTTrack wizard, supporting features like project management and option previews directly in a browser window. This interface facilitates non-desktop use cases, such as remote administration.[7]
An official Android application extends HTTrack's functionality to mobile devices with a touch-optimized interface for on-device website mirroring and file management. The app allows users to input URLs, apply filters, and manage downloaded content via a smartphone-friendly layout, enabling portable offline browsing without relying on a computer. It is developed by the HTTrack team and available through standard app stores.[20][19]
HTTrack does not offer an official native GUI for macOS; instead, users on this platform typically rely on the command-line interface or third-party wrappers for graphical access, with installation available via package managers like Homebrew or MacPorts.[17][21]
Technical details
Operational mechanism
HTTrack initiates its mirroring process by accepting one or more seed URLs provided by the user, which serve as the starting points for the crawl. It employs a recursive algorithm to traverse the website, parsing HTML content using an internal parser to identify and extract hyperlinks, including both relative and absolute links. This parser scans for elements such as<a href>, <img src>, and other resource references, adding discovered URLs to the processing queue while respecting user-defined filters for inclusion or exclusion based on patterns, domains, or types.[7]
The tool handles file downloads through a multi-threaded approach, where multiple simultaneous connections—configurable via the --sockets option, typically up to 20 or more—enable parallel retrieval of resources like HTML pages, images, CSS, and JavaScript files. To prevent naming conflicts in the local directory structure, HTTrack renames files as necessary while preserving the original site's hierarchical organization, and it updates embedded links within downloaded content to reference local paths instead of remote URLs, ensuring the mirror remains functional offline. Larger files may be prioritized in the download sequence to optimize throughput, with smaller ones queued subsequently.[7][22]
URL queue management operates on a first-in, first-out (FIFO) basis, maintaining an ordered list of links to fetch while employing hashing techniques on URLs to detect and eliminate duplicates, thereby preventing redundant downloads and mitigating risks of infinite loops from cyclical links. The queue supports pausing and resuming operations, allowing interrupted sessions to continue from the last processed point without restarting the entire crawl.[7][2]
HTTrack primarily supports HTTP, HTTPS, and FTP protocols for fetching content, with built-in handling for common web mechanics including HTTP redirects (e.g., 301 and 302 status codes), basic cookie management for session persistence, and simple authentication mechanisms like HTTP Basic Auth. It does not natively support more advanced protocols without extensions.[7]
Following the download phase, HTTrack performs post-processing on the mirrored files to adapt them for local use, systematically rewriting internal hyperlinks, image sources, and other embeds to point to the replicated directory structure. This step includes generating an optional index file for navigation and applying any user-specified transformations, such as HTML tidying, to create a cohesive, standalone replica of the site.[7]