It’s Time to Dump CoSchedule: 9 Better Alternatives - February 25, 2022.If you have a big site you should increase the SEO spider’s memory. Under ‘configuration’ you can also choose to include or exclude crawling of URLs or files (such as images, JS, CSS etc). This helps to avoid information overload first of all, but also speeds up the process and uses less memory. Intelligent Spidering – If you have a site with thousands of pages (you’ll need the premium version ideally!), it’s best to spider the site in sections using the above method.If you have a more complicated set up like subdomains and subfolders you can specify both. Hence, if you only want to crawl the blog of your website which is in its own subfolder/sub directory, simply spider the URI with file path for example –. The tool spiders from directory path onwards.The tool spiders (only) the subdomain you enter and treats other subdomains it encounters as external links.XML Sitemap Generator – You can create a basic XML sitemap using the SEO spider.(Please note – This is not a data extraction or scraping feature yet.) Custom Source Code Search – The spider allows you to find anything you want in the source code of a website! Whether that’s analytics code, specific text, or code etc.User-Agent Switcher – Crawl as Googlebot, Bingbot, or Yahoo! Slurp.Images over 100kb, missing alt text, alt text over 100 characters Images – All URIs with the image link & all images from a given page.Follow & Nofollow – At link level (true/false).Outlinks – All pages a URI links out to.Canonical link element & canonical HTTP headers.

Meta Refresh – Including target page and time delay.Meta Robots – Index, noindex, follow, nofollow, noarchive, nosnippet, noodp, noydir etc.H2 – Missing, duplicate, over 70 characters, multiple.H1 – Missing, duplicate, over 70 characters, multiple.Meta Keywords – Mainly for reference as it’s only (barely) used by Yahoo.Meta Description – Missing, duplicate, over 156 characters, multiple.Page Title – Missing, duplicate, over 70 characters, same as h1, multiple.Duplicate Pages – Hash value / MD5checksums lookup for pages with duplicate content.URI Issues – Non ASCII characters, underscores, uppercase characters, dynamic uris, long over 115 characters.External Links – All followed links and their subsequent status codes.Redirects – (3XX, permanent or temporary).Errors – Client & server errors (No responses, 4XX, 5XX).A quick summary of some of the data collected.

Screaming frog seo spider licence