During a site rebuild, URLs are often changed for simplicity or to reflect a necessary structure or nesting change. (For example, products moving to a different product line, or changing names entirely) Web developers and marketers will typically spider the new site to find any 404s that result from this new structure.
While correcting those pre-launch links is certainly important, what about the URLs that are already in place for the production site? That is, if /title-of-product (old site structure) is changing to /product/title-of-product (new site structure), we need to account for those redirects.
List Mode To The Rescue
Screaming Frog's List Mode has one key difference from its default Spider mode. With List Mode, you upload a list of URLs; Screaming Frog will spider that list specifically, rather than traversing the on-page links as it does in Spider mode.
Using List Mode, Screaming Frog allows us to test against links that are not present on the pre-launch version of your site. It's a simulation of what would occur if obsolete links were reached on the new site.
Here are the steps to check for these 404 urls:
1. Spider the current PRODUCTION website
We want to get a list of all current website links as-is. This is why we spider the current production site in this step, and not the staging or pre-launch site.
2. Export the full URL list from the "Page Titles" tab
3. Load the above URL list in a text editor or spreadsheet. Rename the domain name as needed
(e.g. find "current-domain.com" and replace with "staging-domain.com"). Save out the modified URL list for the next step.
4. Change Screaming Frog to List Mode
5.Upload your URL list from step 3
6.Spider your uploaded list and monitor the 404s
7.In your Content Management System, or using nginx or htaccess directs, redirect each 404 until you are satisfied
If you don't know how to adjust URLs or fix 404s on your own, ask your friendly web developer.
What's the benefit?
On launch day, your website is going to be much better prepared to handle obsolete links, whether they are on-site or in a search engine's index. It'll be far less likely that your client marketing department is going to furnish you with a list of 404's that should have been caught before launch.