Skip to content

# Command-Line Guide

This is a task-oriented guide to the `httrack` command line: how to do the things people actually ask for, and the handful of defaults that surprise newcomers. It sits on top of the generated manual page, which lists every option in full. When you want the exhaustive detail for a flag, that page is the reference; this one is the map.

Two habits before anything else. First, HTTrack has its own options: they are not wget or curl flags, so reach for the tables here rather than guessing. Second, when a mirror does something you did not expect, the answer is almost always in the log. Every project writes `hts-log.txt` (and `hts-err.txt`) into its output directory, and those files name every URL that was refused, redirected, or filtered out. Read them first.

### 1. Quick start

A mirror is one command: a start URL and an output directory.

`httrack https://example.com/ --path mydir`

With no other options HTTrack mirrors that site, stays on the same host, follows links to any depth, rebuilds them to browse offline, and stores everything under `mydir`. The same directory also holds the log files and the `hts-cache/` folder that makes a later update or resume possible.

Two defaults are worth knowing up front, because both catch people out:

- HTTrack throttles itself to about 100 KB/s even when you pass no rate option. If a mirror feels slow, that is why. See Limits for how to lift it.
- The download proceeds as a well-behaved robot: it identifies itself as `HTTrack`, obeys `robots.txt`, and sends a Referer with each request. A site that blocks that behavior needs the levers in Identity, not brute force.

### 2. Scope: how far the crawl reaches

Scope decides which links HTTrack is even willing to follow, before any filter you write. Get this right and most "it downloaded too much" or "it only grabbed the index" problems disappear.

Option | What it controls
`--depth (-r)` | Maximum link depth. The start page is level 1, so one level of links out is `-r2`, not `-r1`.
`--stay-on-same-address (-a), --stay-on-same-domain (-d), --stay-on-same-tld (-l), --go-everywhere (-e)` | How far off the starting host the crawl may travel: same address (host), same principal domain, same top-level domain (for example .com), or everywhere. The default keeps you on the starting host.
`--can-go-down (-D), --can-go-up (-U), --stay-on-same-dir (-S), --can-go-up-and-down (-B)` | Directory travel: down into subdirectories only, up to parent directories only, stay in the same directory, or both up and down.
`--near (-n)` | Also fetch non-HTML files "near" a followed link, such as an image linked from a page you kept but hosted elsewhere.
`--ext-depth (-%e)` | How many levels of external links to follow once the crawl leaves your scope (default 0).
`--test (-t)` | Also HEAD-test links that fall outside the scope, which are normally refused, without downloading them: a way to see what scope is excluding.
`--sitemap (-%m), --sitemap-url URL (-%mu)` | Also take start URLs from the site's sitemap, for pages nothing links to. Off by default.

Link-following only finds what something links to. Anything a site publishes solely in its sitemap is invisible to HTTrack unless you ask for it. `--sitemap` reads the start host's `robots.txt` for `Sitemap:` lines and falls back to `/sitemap.xml`; `--sitemap-url` names one directly. Nested `sitemapindex` files and gzipped `.xml.gz` sitemaps are followed. The URLs found become start URLs with the full depth budget, but they still go through your filters and scope rules, so a sitemap cannot widen a crawl you deliberately narrowed. It is off by default because a sitemap can list thousands of pages nothing links to.

One surprise worth knowing: a sitemap you name with `--sitemap-url`, and one the site itself declares in `robots.txt`, are fetched even when `robots.txt` disallows that path, because naming or declaring a sitemap is an invitation to read it. Only the guessed `/sitemap.xml` obeys a `Disallow`. The URLs listed inside are gated normally either way.

The single most common surprise is "only the home page came down." That is usually not a scope option at all: it is an off-host redirect. A start URL of `http://example.com/` that redirects to `https://www.example.com/` lands you on a different host, and same-host scope stops the crawl there. Start from the final URL, or add a filter that re-admits the real host (see Filters). The log will show the redirect.

`-n` is the fix for pages that render locally without their images or stylesheets: it lets HTTrack pull in requisites that sit just outside scope. Note that its embedded-asset handling (following `img`, `link`, `script`, `style` and HTML5 `source`/`track` targets past the normal depth and filter limits) applies only when `-n` is on; it is not automatic. It can also over-fetch by dragging in a whole external host from a single link, in which case name the assets you want with a filter instead.

### 3. Filters and scan rules

Filters are the number-one source of confusion, and also the tool that solves most scope problems once you understand them. A filter is a rule that accepts (`+`) or rejects (`-`) URLs by pattern. The sign is mandatory: `+pattern` adds, `-pattern` removes, and a bare pattern is an error.

The rules that matter:

- Last match wins. Rules are applied in order and the last one that matches a URL decides its fate. Order your rules from general to specific.
- Wildcards. `*` matches any run of characters; `*[a-z]`, `*[0-9]` and similar classes match sets. So `+*.pdf` means "any URL ending in .pdf".
- Whitelisting. To keep one site and nothing else, deny everything then re-admit the host: `"-*" "+example.com/*"`. A lone `+` rule only adds to the default scope; it never restricts.
- Size rules. `*[>100000]` and `*[<1000]` filter by byte size. Because size is only known once the transfer starts, an oversize file is fetched partway and then aborted, not skipped for free.
- mime: rules. A rule like `-mime:video/*` matches the `Content-Type`. That type is only known after the response headers arrive, so a mime rule cannot stop a request; it can only abort the body. Use a URL pattern when you want to avoid the fetch entirely.

Quote your filters. Shells treat `*`, `[` and sometimes `+` specially, so wrap each rule in quotes as shown above. The full pattern language, with tables for wildcards, size and mime, is in the filters page, and the FAQ has a worked tutorial.

robots.txt. By default HTTrack obeys `robots.txt` (`-s2`). `-s0` ignores it entirely, `-s1` obeys it but lets one of your `+` filters override a disallow for a URL you explicitly asked for. Note that a `403 Forbidden` is a server refusal, not a robots rule: robots options will not help there. That is an identity problem.

#### Filter wildcards

Inside a filter pattern, `*` matches any run of characters; a few bracket forms match narrower sets. The full table, with size and mime rules, is on the filters page.

Wildcard | Matches | Example
`*` | any run of characters | `+*.pdf` — any URL ending `.pdf`
`*[file]`, `*[name]` | one path segment (any char but `/` and `?`) | `example.com/*[file]/` — a directory-index page
`*[path]` | a path, slashes allowed (any char but `?`) | `example.com/*[path].zip`
`*[param]` | an optional query string | `page.html*[param]` matches with or without `?...`
`*[a,b,c]` | any one character in the set | `*[a,b,c].txt`
`*[a-z]` | any one character in the range | `img*[0-9].gif`
`*[\x]` | the literal character x (escapes `* [ ] \`) | `*[\*]` matches a real `*`
`*[<NN]`, `*[>NN]` | file size in KB below / above NN | `-*.gif*[<5]` skips GIFs under 5 KB
`*[]` | end anchor: nothing may follow | `*.html*[]` rejects `i.html?p=1`

### 4. Limits and politeness

HTTrack ships cautious on purpose: it is easy to hammer a small site by accident, and the abuse page is worth a read. The limits below let you go faster when you own the target, and slower when you do not.

Option | What it controls
`--max-rate (-A)` | Maximum transfer rate in bytes/sec. The default is about 100 KB/s even without this flag. Raise it to go faster.
`--sockets (-c)` | Number of parallel connections (default 4). `--tiny`, `--wide` and `--ultrawide` are presets.
`--connection-per-second (-%c)` | New connections opened per second (default 5).
`--max-size (-M)` | Stop after N bytes received from the network across the whole mirror (this counts what was transferred, not what was saved).
`--max-time (-E)` | Stop after N seconds of wall-clock time.
`--max-files (-m)` | Per-file size caps.
`--timeout (-T), --retries (-R), --min-rate (-J), --host-control (-H)` | Idle timeout, retry count, minimum acceptable rate, and host-ban behavior for slow or dead hosts.
`--max-pause (-G), --pause (-%G)` | Pause the mirror at N bytes, or pause between files, to spread the load.

The security clamps. To keep an accidental typo from turning into a flood, HTTrack silently caps a few values: at most 8 connections (`-c`), at most 10 MB/s (`-A`), and at most 5 new connections per second (`-%c`). Ask for more and you get the ceiling, quietly. The single flag `--disable-security-limits` lifts all three (the short form `-%!` also works, but the bare `!` is awkward to type safely in a shell). Use it only against infrastructure you are allowed to load that hard.

### 5. File names and types

Where local files land, and what they are called, is controlled by the naming options. This is the second-biggest source of "why did it do that" questions, usually about a URL like `/article?id=42` or a `.php` page that is really HTML.

Option | What it controls
`--structure (-N)` | The local path and name layout. Presets are numeric, and you can also give a template such as `--structure "%h%p/%n%q.%t"`.
`--long-names (-L)` | Long names, 8.3 names, or ISO9660 for CD masters.
`--assume (-%A)` | Assume a MIME type for an extension, for example `--assume php=text/html`. This also skips the extra HEAD probe HTTrack would otherwise send to learn the type.
`--delayed-type-check (-%N), --cached-delayed-type-check (-%D), --check-type (-u), -%t` | When and how the content type is checked, and whether the original extension is kept.
`--include-query-string (-%q), --strip-query (-%g)` | Whether the query string appears in the local filename, and whether query keys are stripped when deciding if two URLs are the same file.

The `-N` presets are built from modular arithmetic on the name fields, so undocumented number combinations often "work" by accident. If you care about the exact layout, use an explicit template (the `%h %p %n %q %t` placeholders) rather than a magic number, and check the result on a small crawl first.

A dynamic page served as `.php` or `.asp` that is actually HTML is the classic case: without help it can be saved with an extension a browser will not open locally. `--assume php=text/html` fixes both the extension and the naming.

### 6. Links and page building

After a page is downloaded, HTTrack parses it for more links and rewrites the ones it kept so the local copy browses offline. These options tune both halves.

Option | What it controls
`--keep-links (-K)` | How links are rewritten in saved pages. The numbering is inverted from what you might guess: bare `-K` keeps absolute URLs, and `-K0` is the relative default. `-K3` keeps absolute URIs, `-K4` keeps the original links.
`--replace-external (-x), --generate-errors (-o)` | Replace external links with an error page, and generate an error page for links that failed.
`--preserve (-%p), --disable-passwords (-%x)` | Leave HTML untouched (no rewriting), and strip passwords out of saved links.
`--extended-parsing (-%P), --parse-java (-j)` | Aggressive link discovery, and how much script content is parsed for links.
`--mime-html (-%M)` | Save the whole mirror as a single MIME-encapsulated `.mht` archive (`index.mht`).
`--single-file (-%Z), --single-file-max-size N` | Once the mirror is finished, rewrite every saved page with its stylesheets, scripts, images and fonts embedded as `data:` URIs. Assets over the cap (10 MB by default) keep their link, as do audio, video, and the links from one page to another. A sibling of `-%M`, not a replacement: see the recipe below for which to pick.
`--index (-I), --build-top-index (-%i), --search-index (-%I)` | Build a per-mirror index, a top index across projects, and a searchable keyword index.

HTTrack finds links by parsing HTML and CSS. It does not run JavaScript, so any URL a page builds at runtime in script (a lazy-loaded image, a JavaScript-assembled path) is invisible to the crawler and will be missing from the mirror. There is no flag that fixes this; the asset has to appear in the static HTML or CSS to be found. `-%P` widens discovery for links that are present but awkwardly formatted, not for links that do not exist until script runs.

### 7. Identity, cookies and login

By default HTTrack is an honest robot: it sends a `User-Agent` of `HTTrack`, a Referer with each link (which reveals the crawl path to the server), and obeys robots. Plenty of sites filter exactly that profile. These options control what HTTrack says about itself.

Option | What it controls
`--user-agent (-F)` | The `User-Agent`. Set a browser string to get past crawler blocks; `--user-agent ""` sends none.
`--referer (-%R), --from (-%E), --language (-%l), --accept (-%a)` | Referer, From, Accept-Language and Accept headers.
`--headers (-%X)` | Add raw header lines to every request.
`--footer (-%F)` | A footer written into saved pages (on disk, not a network header). See Footer fields below.
`--cookies (-b), --cookies-file (-%K)` | Accept cookies, and preload a Netscape `cookies.txt`.

Footer fields. A footer with no `%s` may reference named fields: `{addr}`, `{path}`, `{url}`, `{date}` (mirror time), `{lastmodified}` (the page's Last-Modified), `{version}`, `{mime}`, `{charset}`, `{status}` and `{size}`; write `{{` or `}}` for a literal brace. A footer that contains `%s` keeps the older positional form (host, path, date in that order). Example: `-%F "<!-- Mirrored from {url} on {date} -->"`.

Login. For HTTP Basic auth, put the credentials in the URL: `http://user:pass@host/`. An `@` inside the username must be written `%40`. Only Basic is supported, not Digest.

For cookie or form logins, the simplest path is to log in with a browser, export its `cookies.txt`, and drop that file in the project directory so HTTrack sends the session cookie. For a form that needs a POST, `--catchurl` can capture the exact request your browser sends and replay it. A few cookie caveats to know: expiry is ignored, there is a silent cap of about 8 cookies sent per request, and `-b0` disables cookies and the reuse of Basic credentials across links at the same time.

### 8. Proxy and network

Option | What it controls
`--proxy (-P)` | Route through a proxy. HTTP, SOCKS5 and CONNECT are supported: `-P host:8080`, `-P socks5://host:1080`, `-P connect://host:443`, with optional `user:pass@`.
`--httpproxy-ftp (-%f)` | Send FTP requests through the HTTP proxy.
`--protocol (-@i)` | Prefer IPv4 or IPv6.
`--http-10 (-%h), --keep-alive (-%k), --disable-compression (-%z)` | Force HTTP/1.0 (drops keep-alive and compression, useful for fragile CGI), toggle keep-alive, and toggle compression.
`--bind (-%b), --tolerant (-%B)` | Bind to a local address, and accept technically-bogus responses some servers send.

Two network facts worth stating plainly. Over SOCKS5, HTTrack always resolves host names at the proxy (remote DNS) for both `socks5://` and `socks5h://`, so your local resolver is never consulted. And HTTrack does not verify TLS certificates: HTTPS gives you an encrypted transport, but not an authenticated one. That is a deliberate choice for a mirroring tool, not a bug, but it is worth knowing if you are relying on it for trust.

### 9. Update and cache

Every project keeps a cache under `hts-cache/`. It records every URL that was fetched, together with the options you used, and it is what makes resuming and updating possible. It is not a size-limited scratch area you can delete: throw it away and you lose the ability to continue or update the mirror.

Option | What it controls
`--continue` | Carry on an interrupted mirror, trusting the cache: it does not re-check pages already stored.
`--update` | Re-run the mirror, revalidating each page with the server (If-Modified-Since / If-None-Match) and downloading only what changed.
`--purge-old=0 (-X0)` | Do not purge. By default an update deletes local files that are no longer part of the mirror; `--purge-old=0` keeps them.
`--cache (-C)` | Cache mode. The default already does the right thing and switches to update-checking when it detects an existing mirror.
`--urlhack (-%u), --keep-www-prefix (-%j), --keep-double-slashes (-%o), --keep-query-order (-%y), --do-not-recatch (-%n), --updatehack (-%s), --store-all-in-cache (-k)` | URL-deduplication behavior and cache storage details.
`--debug-cache (-#C), --repair-cache (-#R), --clean` | Inspect the cache, repair its ZIP, and erase cache plus logs.

The purge trap. An `--update` run rebuilds the list of files the mirror should contain, then deletes any previously-mirrored file that is not on the new list. This is what keeps a mirror in sync with a shrinking site, but it means a partial or interrupted update can delete files you meant to keep. If an update might not complete cleanly, add `-X0` to protect the existing tree, and expect dynamic pages to look "changed" on every run. See the cache page for the details.

### 10. Experts and scripting

HTTrack is also a scriptable fetch-and-scan tool. These options turn off the mirror behavior and expose the engine.

Option | What it controls
`--get URL` | Fetch a single file and stop. Cache, index, depth, cookies and robots are all off for this mode.
`--spider --testlinks --skeleton` | Scan without saving, test links at depth 1, or keep HTML only. Handy for checking a site before a real crawl.
`--userdef-cmd (-V)` | Run a shell command on each downloaded file; `$0` is the file path. Good for on-the-fly processing.
`--callback (-%W)` | Load an external callback module to hook the engine.
`--do-not-log (-Q), --quiet (-q), --verbose (-v), --file-log (-f), --extra-log (-z), --debug-log (-Z)` | Logging: quiet, no questions, verbose on screen, and the various log-to-file levels.

The developer page covers the callback API and batch use in more depth.

### 11. Recipes

Copy-ready command lines for the tasks people ask about most. Each has the one gotcha that trips it up.

#### Mirror one site, nothing off-host

`httrack https://example.com/ --path mydir`
Same-host is already the default. The usual failure is a `www.`-to-apex or `http`-to-`https` redirect that moves you off host and stops the crawl; start from the final URL, or add `"+finalhost/*"`.

#### One level of links only

`httrack https://example.com/ --depth=2 --path mydir`
Depth counts the start page as level 1, so one level out is `--depth=2`.

#### This site only, deny everything else

`httrack https://example.com/ "-*" "+example.com/*" --path mydir`
You must deny-all first; a lone `+` only adds, and the last matching rule wins.

#### Download the PDFs on a site

`httrack https://example.com/ "-*" "+https://example.com/*.html" "+https://example.com/*[path]/" "+https://example.com/*.pdf" --path mydir`
HTTrack finds PDFs by parsing the site's HTML, so a plain `"-*" "+example.com/*.pdf"` is wrong: it prunes the pages that carry the links and keeps only PDFs reachable from the front page. Instead admit the HTML as scaffolding (`*.html` and `*[path]/` for directory-index pages at any depth, e.g. `docs/` or `a/b/deep/`; `*[file]/` would stop at one level), keep the PDFs, and let `-*` drop everything else (images, archives, off-site assets). PDFs on another host (a CDN or docs subdomain) are not included by default; allow that host too, e.g. `"+docs.example.com/*.pdf"`, or widen to `"+*.pdf"` for PDFs anywhere.

#### Keep page requisites, including off-host images

`httrack https://example.com/blog/ --near --path mydir`
`--near` can over-fetch by pulling in an entire external host from one link; if it does, name the assets with `"+cdn.example.com/*"` instead.

#### Resume an interrupted crawl

`httrack --continue --path mydir`
Needs an intact `hts-cache/`. Deleting it loses the URL list and your options.

#### Update a mirror, downloading only what changed

`httrack --update --path mydir`
Purge deletes any previously-mirrored file not seen this run; add `--purge-old=0` to protect the tree against a partial update.

#### Reach a section behind a login cookie

Export your browser's `cookies.txt` into the project directory, then:
`httrack https://example.com/members/ --path mydir`
The file must be Netscape `cookies.txt` in the project folder; `--cookies=0` would disable cookies and Basic-auth reuse together.

#### Get past a crawler-UA block

`httrack https://example.com/ --user-agent "Mozilla/5.0 (Windows NT 10.0)" --path mydir`
A 403 is server-side, not robots: change the User-Agent, and add `--http-10` for fragile CGI. Do not reach for `--robots=0`.

#### Full-speed mirror on your own server

`httrack https://example.com/ --max-rate=2000000 --sockets=8 --disable-security-limits --path mydir`
The default throttles to about 100 KB/s. `--disable-security-limits` lifts the built-in caps; use it only against infrastructure you are allowed to load.

#### Save a WARC archive of the crawl

`httrack https://example.com/ --warc --path mydir`
Writes a standard WARC/1.1 file (`httrack-<timestamp>.warc.gz`) in the project folder alongside the browsable mirror, not instead of it. Set the name with `--warc-file NAME` and split a large crawl with `--warc-max-size N`; add `--warc-cdx` for a sorted CDXJ index, or `--wacz` to bundle the archive, index and pages into one WACZ for replay tools such as replayweb.page.

#### See what a re-crawl changed

`httrack https://example.com/ --update --changes --path mydir`
Writes `hts-changes.json` in the project folder, listing every mirrored file as new, changed, unchanged or gone, plus a one-line summary in the log. "Changed" means the bytes really differ: a server that answers 200 with the same content it served last time lands in `unchanged`. Deletions are reported whether or not `--purge-old` is deleting them. The format is documented in the change report specification.

#### Pages you can hand to someone as one file

`httrack https://example.com/ --single-file --path mydir`
Rewrites each saved page after the crawl so its stylesheets, scripts, images and fonts are embedded as `data:` URIs. The mirror stays a normal browsable tree, with links between pages relative and the assets still on disk, but any single `.html` file can now be mailed or archived on its own. Raise or lower the 10 MB per-asset limit with `--single-file-max-size N`; anything over it, plus audio and video, keeps its link. One caveat if you raise it: an inlined stylesheet becomes a `data:` URL, whose path is opaque, so an asset it referenced that stayed over the cap no longer resolves from inside it. Raising the cap past that asset embeds it too and the question goes away.
Reach for this when the file has to open for someone you cannot make assumptions about: it is plain HTML and needs no add-on. Reach for `--mime-html` instead when a Chromium-family browser is a given and the mirror is large: MIME carries text parts without the base64 tax, keeps each resource's original URL, and stores a shared asset once rather than re-embedding it in every page that uses it.

#### HTTrack as a fetch tool

`httrack --get https://host/file.bin --path tmp`
`--get` fetches one file with cache, index, depth, cookies and robots all disabled.

For the complete, always-current option list, see the manual page. For the filter language, see filters; for the cache and updates, see cache; for the change report, see changes.