Web Archive Search Tips: Search Feature & CDX Server

[Gene Worm]

Hello, everyone. I just wanted to provide an overview for features in archive.org that I found useful and didn't think was common knowledge.

My favorite trick is being able to view every page under a specific prefix. Instead of using the sitemap which I personally find rather painful or trying to haphazardly navigate through the website archive itself, you can put an asterisk on the front and end of the URL you want to search for. The first asterisk needs to sandwiched between two forward-slashes. Note that this function is limited up to 10,000 results.

For example, here's the link for the list of every archived page of the author Nobuyuki's old website:
https://web.archive.org/web/*/http://getterth.hp.infoseek.co.jp:80/mugen*

I do want to mention that, unfortunately, Web Archive does not prioritize download links due to the sheer size associated with preserving every file uploaded to the internet. This means that, while amazing things can be and have been found on Archive.org, there's a lot of luck involved.



Furthermore, there's a more advanced function to archive.org: The Wayback CDX Server.
https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server 

This is basically what I described above but turned to the extreme. Among other things, this allows you to call for a page that lists archived URLs under many specific conditions. I recommend learning regex to get the best use of its filter function.

https://web.archive.org/cdx/search/cdx?url=*.big.freett.com&fl=original&collapse=urlkey&filter=statuscode:200&filter=original:.*\.(zip|rar)$
Here URL crawls every zip and rar file uploaded to the old site "big.freett" (a Japanese hosting site). I'll dissect it a little bit:

- With url=, asterisks can be used to denote prefix or suffix (prefix, domain, etc. can be specified explicitly with matchType as well).
- fl= is Field Order. Determines if you get additional info like time archived and whatnot.
- collapse=urlkey does what you probably think. Instead of getting this massive line of information, this narrows it down to just the link you want.
- filter=statuscode:200 is an important filter that should hopefully mean you only get working links and not some redirect.
- filter= is where you can add all the various regex and name conditions. All this does in my link above is look for URLs ending in zip or rar.

This can bypass the 10,000 results search limit, but a browser can still only handle a certain extent. The "pages" feature can be used to help with that, but that still can't capture the full scope of truly enormous domains.

I hope this comes in handy to fellow preservationists and anyone interested in the endeavor.