Simple Page Inventory Crawl with Screamingfrog

Last updated on August 23rd, 2021 at 02:31 pm

This crawl is perfect for when you want to get response codes, meta data and other high level information for a typical website. Sometimes the most simple things do not seem as obvious as maybe they should.

Here are the configuration settings you need to run a quick crawl on a typical website for the purposes of gathering internal link page inventory. 

Page Inventory Crawl Settings for Screamingfrog

  1. Reset to the default configuration

    Go to menu: File > Configuration > Clear default Configuration

  2. Set to spider mode

    Go to menu: Mode > Spider

  3. Open crawl configuration settings

    Go to menu: Configuration > Spider

  4. Update crawl configuration settings

    uncheck all of the crawl boxes in the “Resource Links” section then uncheck all of the crawl boxes except for “Internal Links” in the “Page links” section then click “OK”.page inventory configuration in screamingfrog

  5. Run the crawl

    Put the domain root address in the address bar and click “Start” and watch the page data roll in.
    run the screaming frog crawl

Why isn’t screaming frog crawling all of my pages?

If you are starting the crawl but you are not seeing the pages you expect to show up then the robots.txt may be disallowing sections of your site. If that is the case you can set the crawler to ignore the robots.txt in menu: configuration > robots.txt > settings and changing the drop down to “Ignore robots.txt” then try the crawl again.
ignore robots in screamingfrog

I set Screamingfrog to ignore my robots.txt but it is still not crawling all of my pages

If robots.txt was not the issue, the website may be using JavaScript for navigation or hyperlinks. Try enabling JavaScript rendering in Menu: Configuration > Spider > Rendering tab. Change the dropdown to “JavaScript” then uncheck “Enable Rendered Page Screenshots” then click “OK”
enable javascript rendering in screamingfrog
enable javascript rendering in screamingfrog

Where can I find additional resources for Screamingfrog?

Screamingfrog YouTube channel
Screamingfrog User Guide
Screamingfrog Tutorials

Published
Categorized as Articles

By Eric Mandell

Eric Mandell is a an SEO professional since 1999 and is currently a Director of SEO at Catalyst Performance Marketing Services out of Boston.

Leave a comment

Your email address will not be published.