Picture a small firm whose web developer stopped answering email months ago. The hosting renews next month, nobody remembers the password to the old content system, and the only complete copy of the site is the one being served to the public. Wanting it on a disk of your own is reasonable. Doing it well, and politely, takes more thought than pointing a tool at the home page.
Why anyone wants an offline copy
The reasons are mostly unglamorous, which is a good sign.
- Archives. A record of what the site said on a given day: prices, terms, notices to clients. If anyone later asks what you told them, a dated copy answers faster than memory.
- Audits. Checking a few hundred pages for broken links, stale content or missing alt text is easier offline, where you can search every file at once without touching the live server again.
- Migrations. Moving off an old content system or a site builder with a poor export. The copy lists every page that existed, so nothing quietly disappears in the move.
- Learning. Seeing how a well-made site structures its pages and assets teaches more than most tutorials. Studying is fine. Republishing is another matter.
What "the whole site" really means
The HTML pages are the easy part. A page is really a list of other things to fetch, and each of those can point somewhere else again.
- Stylesheets, which pull in files of their own: fonts, background images, sometimes more stylesheets through
@import. None of those appear in the HTML, so a downloader has to read the CSS too. - Scripts, and whatever they fetch after the page loads. If a product list arrives by JavaScript, it is not in the HTML at all, and a copy taken without running the scripts has an empty box where the list should be.
- Images in every size. Pages often offer several versions of one picture through
srcset, and the copy needs each one the page might ask for. - Links, rewritten. This is where most copies break. A link to
https://www.example.com/about/fails offline, so every reference has to become a relative path on disk, and an address like/about/has to become a real file, usuallyabout/index.html.
Then come the edges: search filters that generate endless addresses so a careless crawler never finishes, and fonts or libraries on another domain, where you decide whether they belong in the copy. A good copy is a set of decisions as much as a set of files.
The manners
Every request you send is answered by someone else's server, and for a small site on shared hosting, someone else pays for it. A crawler fetching dozens of files a second can slow the site for its real visitors.
- Respect robots.txt. It is a request, not a lock, which is exactly why it matters: honouring it is a choice. Read the site's terms too. Some forbid automated copying outright.
- Rate-limit. One request at a time, with a pause between them. A copy that takes an hour instead of five minutes costs you nothing; you are not sitting and watching it.
- Identify your tool. Send a user agent that names the tool and gives a way to reach you. A crawler disguised as a browser leaves the owner nobody to ask, so their only option is a block.
- Do not copy what you have no right to. Pages behind a login you were not given are not part of the job, and having the files is not permission to use them. Your own site, a client's site with their written agreement, a public site you are studying for yourself: fine. Republishing someone else's text, photos or design: not fine, however easy the tool made it.
With wget, a polite full copy looks something like this:
wget --mirror --page-requisites --convert-links --adjust-extension \
--no-parent --wait=2 --random-wait --limit-rate=200k \
--user-agent="SiteCopy/1.0 (contact: you@example.com)" \
https://www.example.com/--page-requisites fetches what each page needs, --convert-links rewrites links for offline use, and the wait and rate options keep the load small. wget honours robots.txt by default; leave it that way. It does not run JavaScript, so content loaded by scripts needs another approach.
The tool I built for it
I wrote a tool for this job, Website Downloader, in JavaScript on Node.js. It downloads entire websites, HTML, CSS, JavaScript and images included, for offline backups, website migration, or learning from an existing site by examining its code structure and assets. Demo access is on request through my contact page, and it sits with the rest of my work on the projects page.
A tool can make good manners easy. It cannot supply them. The pause between requests, the honest user agent and the question of whether you should be copying at all stay with the person who presses the button.
Sometimes the politest crawler is an email
Before any of this, look for a simpler way in. For your own site, the hosting account can usually produce a full backup, database included, more complete than any crawl. For someone else's, a short note saying what you want and why often gets a better copy than a crawler could, and settles the question of permission first. Crawl when asking is not possible, and crawl as if the owner were watching the logs. Sometimes they are.
