Web crawlers, spiders, and bots, oh my!
Though it may sound like the lyrics from a geeky horror musical gone terribly wrong, the world of search engine optimization (SEO) is actually seriously reliant on and invested in the reality of web crawlers.
Web crawlers (also known as search engine bots or website spiders) are automated digital software applications that are programmed to find and index websites for search engines. They move from page to page, scrutinizing and reporting on on-page and technical aspects. Ultimately, their findings inform web page rankings.
But wait. Wouldn't it be great if you could The Future Of SEO & How To Stay Ahead Of The Curve">stay one step ahead of search engine bots by engaging a crawling tool of your own? A tool that could give you a snapshot of your site's strengths and weaknesses so that you could optimize it for future web crawls and boost your search engine rankings?
Enter this Screaming Frog review.
Screaming Frog is an SEO spider that crawls your site in search of on-page and technical SEO issues. It is incredibly thorough, straightforward, and built with digital marketing professionals in mind (consultants, agencies). I’m personally a huge fan of Screaming Frog. In fact, I'm not sure how I'd get through my day without it!
Though no software is perfect, I stand solidly by my 5/5 star overall rating for the product.
Screaming Frog is fully loaded with features. From internal reviews to external, and from integrations to reporting, its intuitive interface and clearly organized tab and filtering systems keep everything in plain view. To my mind, Screaming Frog's features are refreshingly comprehensive and accessible. A winning combination.
Across the board, Screaming Frog has a consistent reputation for reliability and accuracy. While no software can ever be 100% accurate all the time, Screaming Frog is about as thorough and trustworthy as it gets. Also, the software isn't glitchy, which is a huge plus in my books.
Screaming Frog offers a free version that may be adequate for the needs of small businesses. In my opinion, though, the paid version is well worth the approximately $200 annual fee (price is in British pounds, so fluctuates slightly with exchange rates). For the amount of benefit the program provides, I think its pricing is very reasonable.
Screaming Frog is a spider tool that crawls websites to perform on-page and technical SEO audits. It identifies problems (missing meta descriptions, broken links, mismatched titles and tags, plus so much more) and shows you exactly what repairs are required and where. Using Screaming Frog to regularly crawl your website can keep it in tip-top technical condition and help it to attain and maintain higher rankings.
Screaming Frog was founded in 2010 by a team of four experienced SEO professionals with proven personal track records. Why? Because they felt the digital marketing landscape lacked a trustworthy tool for collecting data, identifying technical issues, and providing a base for onsite SEO recommendations. Screaming Frog has certainly filled that void—and its sustained popularity is evidence of its success.
Today, Screaming Frog has a more than 40-person team and is a staple tool for SEO professionals worldwide. Located in Henley-on-Thames, Oxfordshire, U.K., it's been showered with prestigious tech and marketing awards.
As if all that weren't enough, I'll personally back Screaming Frog. As an experienced SEO professional for a leading agency, I can honestly say that I use Screaming Frog every single day. It’s one of my fundamental go-to tools and I’d be hard pressed to do my job without the efficient and relevant data it provides.
That being said, Screaming Frog will not necessarily be the right tool for everyone. In this review, I'm going to walk you through the platform's pros, cons, and features. I'll finish off by looking at some case studies and answering a few commonly asked questions. If you stick with me, by the end you should have a good sense of what Screaming Frog has to offer and whether or not you want to invest in it.
Let's get started!
Before I jump into my overview of Screaming Frog's features (of which there are many), let's just pause for a moment to talk about how they're laid out within the software. Essentially, Screaming Frog's main dashboard is divided into four panes.
The top left pane shows crawl data organized in a format that resembles an Excel spreadsheet. At the top of the pane, all of the data is organized into separate tabs (Page Titles, Meta Description, Security, etc.) that can each be filtered and segmented to your heart's content. When you click on a tab, you'll then see its specific metrics reflected in the columns of the spreadsheet for each separate URL crawled.
The top right overview pane updates a list of data in real time. You can scroll down through it to see which tabs are populated for any given crawl (rather than opening each tab individually). If you like, you can select the Issues Tab to see only a list of data that Screaming Frog believes to be problematic. (Be careful, though. Not everything in the Issues Tab is necessarily a problem you need to address. In some cases, intentional actions—such as no-index pages you've purposely created—may also get flagged.)
The bottom left pane provides further information related to the specific URL and tab you've selected (URL details, Inlinks, Outlinks, Image Details, etc.). And, finally, the bottom right pane displays visual data graphs.
At any point during a crawl, you can pause Screaming Frog, export information (narrowed down by tab and filter, if you wish), and then resume the crawl. You can also access historical crawls to note the progress being made as you make changes to a website (known as a comparison crawl).
In case you require extra support, Screaming Frog offers a thorough selection of self-guided resources as well as technical support via email or a ticketing system.
Now that I've provided a snapshot of Screaming Frog's architecture to put its features in context, let's get to them. Without further ado, here are some of Screaming Frog's core features (most of which are positioned as tabs):
Screaming Frog's Internal Tab is the perfect starting point for analyzing the details of pages within your own website. Reported data includes Page Title, Meta Description, H1 and H2 Contents, Word Count, and much more. Available filters include HTML, JavaScript, CSS, Images, PDF Flash, Other, and Unknown. Worth noting is that, if you don't select a specific tab, the Internal Tab is Screaming Frog’s default tab.
Not surprisingly, Screaming Frog's External Tab provides information about external URLs (URLs on a different subdomain than the page you started your crawl on). The External Tab provides less in-depth information than the Internal tab, but there's still plenty to work with. Reported data includes Address, Content, Status Code, Crawl Depth, and more. Filter options are the same as for the Internal Tab.
The Security Tab displays data related to security. It only displays security information for internal URLs. Reported data includes Address, Content, Status Code, Indexability, and more. Filters include HTTP URLs, HTTPS URLs, Mixed Content, Form URL Insecure, and more.
The Screaming Frog Response Codes Tab will alert you in the event that any of your URLs are returning bad codes such as 404 (Not Found), 3XX (Redirect), or 5XX (Server Error). Reported data includes Address, Content, Status Code, Indexability, Response Time, Redirect URL, and more. Filters include Blocked by Robots.txt, Blocked Resource, No Response, and more.
This tab reveals common URL issues directly related to the specific URLs in your crawl. Reported data includes Address, Content, Status Code, Indexability, and more. Filters include Non ASCII Characters, Underscores, Uppercase, Multiple Slashes, and more.
Page titles (also known as title tags or SEO titles) explain the purpose of a web page to users and search engines alike. They’re an important on-page SEO factor, and Screaming Frog helps to identify inconsistencies and other issues with them. Reported data includes Address, Occurrences, Title 1/2, Title 1/2 length, and more. Filters include Missing, Duplicate, Over 60 Characters, Below 30 Characters, Multiple, and more.
Meta descriptions are those little write-ups you see below results on search engine results pages. Though they're not part of Google's ranking considerations, meta descriptions can still have a major impact on click through rates. Reported data for this tab includes Address, Occurrences, Meta Description 1/2, Meta Description 1/2 Length, and more. Filters include Missing, Duplicate, Over 155 Characters, Below 70 Characters, Multiple, and more.
The H1 Tab takes your page's H1 tags under scrutiny. (There's also an H2 Tab that's very similar.) An H1 typically denotes a page's main heading, while H2-H6 represent smaller headings. Headings help human users as well as bots understand how page content is organized. They’re a strong on-page ranking signal. Screaming Frog's reported data for H1s includes Address, Occurrences, H1-1/2, Indexability, and more. Filters include Missing, Duplicate, Over 70 Characters, and Multiple.
The Content Tab is more or less Screaming Frog's version of a content editor. It reports on content elements that can impact your overall SEO. In addition to Word Count, Average Words Per Sentence, Duplicate Content, and Indexability, the Content Tab also includes data on readability, spelling and grammar errors, reading ease, and more. Filters include Exact Duplicates, Near Duplicates, Low Content Pages, Spelling Errors, Grammar Errors, Readability Difficult, and Readability Very Difficult.
In addition to crawling text content, Screaming Frog also crawls your image (and other media) links. The Images Tab reports on both internal and external images and reveals common issues related to images and their alt text. Reported data includes Address, Content, Size, and Indexability. Filters include Over 100kb, Missing Alt Text, Missing Alt Attribute, Alt Text Over 100 Characters, and Background Images.
When multiple pages exist with the same URL (duplicate pages), you can identify one as the most representative page for ranking—the master copy of a page, if you will—by placing a canonical tag (rel="canonical") within the head of the document. Screaming Frog's Canonicals Tab reveals canonical link elements and HTTP canonicals that were turned up during a crawl. Reported data includes Address, Occurrences, Indexability, Canonical Link Element 1/2 etc., HTTP Canonical 1/2 etc., Meta Robots 1/2 etc., and more. Filters include Contains Canonical, Self Referencing, Canonicalised, Missing, Multiple, Multiple Conflicting, and more.
Hreflang is, "an HTML attribute used to specify the language and geographical targeting of a webpage." It helps inform search engines about the relationship between business pages in different languages/regions and to ensure that the most appropriate page is displayed to the correct market segment. Screaming Frog's Hreflang Tab includes hreflang data such as Address, Title 1/2 etc., Occurrences, HTML hreflang 1/2 etc., Sitemap hreflang 1/2 etc., and more. Filters include Contains Hreflang, Non-200 Hreflang URLs, Unlinked Hreflang URLs, Missing Return Links, and more.
In the world of SEO, structured data is the in-page markup that helps search engines understand page content. The Structured Data Tab in Screaming Frog includes structured data details and validation issues. Reported data includes Address, Errors, Warnings, Total Types, Unique Types, and more. Filters include Contains Structured Data, Missing Structured Data, Validation Errors, Validation Warnings, Parse Errors, and more.
Without an XML sitemap, a file that lists a website's most important pages, it can be difficult for search engines to find and understand its structure. Screaming Frog's Sitemaps Tab reveals sitemap information revealed during a crawl. Reported data includes Address, Content, Status Code, Indexability, and more. Filters include URLs In Sitemap, URLs Not In Sitemap, Orphan URLs, Non-Indexable URLs In Sitemap, URLs In Multiple Sitemaps, and more.
To access the full range of data available within the Screaming Frog PageSpeed Tab, you'll have to configure the program to insert a free PageSpeed API key. Once that's done, Screaming Frog will pull data from Lighthouse. Reported data includes overview metrics such as Total Size Savings, Image Count, and Media Count, as well as insights based on both field data and lab data. You'll also have access to potential opportunities and diagnostics. Filters for this tab are plentiful and include Eliminate Render-Blocking Resources, Defer Offscreen Images, Efficiently Encode Images, Reduce JavaScript Execution Time, and many more.
Google Analytics and Search Console integrations are only available with the paid version of Screaming Frog. If you opt to integrate the platform with GA and GSC (which I strongly recommend), you'll have to do the following:
To integrate Google Analytics with Screaming Frog, select it from the Configuration (Integration) menu and follow the steps provided.
Once integrated, Screaming Frog allows you to select up to 30 metrics at a time but automatically collects 10, which include Sessions, % New Sessions, New Users, Bounce Rate, Page Views Per Session, and more. Filters include Sessions Above 0, Bounce Rate Above 70%, No GA Data, Non-Indexable With GA Data, and Orphan URLs.
To integrate Google Search Console with Screaming Frog, select it from the Configuration (Integration) menu and follow the steps provided.
Once integrated, Screaming Frog collects data on clicks, impressions, CTR, and position. You can also opt to enable URL inspection alongside Search Analytics data. Filters include Clicks Above 0, No Search Analytics Data, Non-Indexable With Search Analytics Data, Orphan URLs, URL Is Not On Google, and more.
Different situations are best suited for crawls with different configurations. Fortunately, you can opt to adjust and customize Screaming Frog's crawl settings before unleashing the SEO spider tool on your website. I highly recommend experimenting with different settings and creating custom rules to find the configuration that best fits your specific goals and website structure as a whole.
Screaming Frog provides a wide range of different crawl setting options. Here are a few of my favorites:
Screaming Frog's Database Mode allows big sites with more than 500,000 URLs to be crawled quickly and efficiently. How? By switching from memory storage to database storage. Memory storage is Screaming Frog's default setting and is fine for small and medium sites, but it basically uses your computer's RAM and can really slow down your whole system when overloaded. Using Database Mode speeds up crawls and can save you quite a bit of time and frustration.
In Database Mode, you can automatically save crawls to your hard drive and pull them back up any time for reference and comparative analysis.
Screaming Frog offers access to two main types of interactive visualizations: Crawl visualizations (which provide a snapshot of the path the SEO spider uses to crawl a site); and directory tree visualizations (which reveal the URL architecture of a website). Both types of visualization can be accessed as either a force-directed diagram or a tree graph.
Screaming Frog's visualizations don't actually include any additional data from what's available in the software's tabs, spreadsheets, and other reports. That being said, they provide a unique (and attractive) presentation of data that may allow you to notice new patterns and interpret information in fresh and unexpected ways. Personally, I find that visualizations can be particularly impactful when used in presentations or put forward for brainstorming and analysis sessions. Ask your team what jumps out at them and go from there!
Screaming Frog offers several types of data-driven reports based on crawls (accessible in the Reports menu). They cover everything from canonical errors to unlinked paginated pages. I've highlighted the three reports I use most often below:
Exporting crawl data from Screaming Frog is a great way to ensure that you can access it without having to open the tool itself. The process is straightforward and simple. You can export top window data (tabs and filters), lower window data (URL details, inlinks, outlinks, image details, and more), or you can bulk export data.
The Bulk Export menu is located at the top of the Screaming Frog dashboard and includes many different categories and sub-categories. Screaming Frog will export all of the crawl data from whichever category (or categories) you select. Categories include Queued URLs, Link, Web, Path Type, Security, Response Codes, and many more. Sample sub-categories from Links include All Inlinks, All Outlinks, External Links, All Anchor Text, and more.
Screaming Frog's SEO spider tool is an advanced and flexible SEO crawler that thoroughly audits the on-page elements of small and large websites alike. The software solution's algorithm is continually being honed and updated and the tool as a whole has an outstanding reputation for consistency and accuracy. Integrating Screaming Frog with Google Analytics and Google Search Console (paid version only) further boosts the accuracy of the data it is able to retrieve.
When relying on a tool like Screaming Frog for the most accurate data possible, you want to be certain that it can handle anything you'll throw its way. And, overall, I think Screaming Frog delivers. It's nearly impossible to trip up the SEO spider.
As you've seen throughout this review, Screaming Frog is a highly customizable tool. Sometimes, data accuracy is dependent on knowing how to frame, extract, and interpret the exact information you're looking for, and Screaming Frog's customization options can assist you in doing exactly that. Using the program with finesse may take a little practice, but in the end, you'll find that you can customize just about everything to unearth accurate data with precision.
Screaming Frog crawls websites with a huge range of coding languages and other technologies. From HTML to AJAX, JavaScript, and more, the SEO spider is prepared. Data accuracy remains a strength for Screaming Frog across the internet's many technologies.
Text-based content, video content, audio content, image content, and more can all be handled by Screaming Frog's SEO spider. No matter how the content on your website is arranged or configured, the tool will be able to provide accurate data that can help you to improve its search engine optimization.
Screaming Frog has two prices: Free and paid. Though the free version is limited, it may be adequate for some users. At the very least, it can provide a free trial for you to familiarize yourself with the software before purchasing.
Let's take a closer look at what the Screaming Frog free and paid versions have to offer:
The free version of Screaming Frog has a crawl limit of 500 URLs, which is not much at all. (Depending on the website you're crawling, you could hit 500 URLs within one crawl.) That being said, it could be enough for users in certain situations to get some use out of it. Free users have access to 7 out of the 27 features Screaming Frog lists on its pricing page. One of the features not included is free technical support.
I'd definitely recommend trying the free version before purchasing the paid version, but if you hope to use Screaming Frog in any serious way, I'd highly recommend investing in the paid version.
The paid version of Screaming Frog costs approximately $209 per year for a single license, but the price per license decreases if you purchase multiple licenses annually. As I mentioned earlier, the cost of the software fluctuates based on the exchange rate of the British Pound.
Paid Screaming Frog users have unlimited crawls (unless their own memory or storage is limited) and can access all features as well as free technical support.
Screaming Frog features 21 interesting case studies on its website, each one showcasing the impact that powerful SEO software combined with intelligent and bold digital marketing strategies can have on a business's success. I'd recommend taking a peek for inspiration if you have a few minutes to spare.
As for me, I feel that I'm a bit of a case study myself. Because I actually use Screaming Frog each and every day for every client I work with, I feel that I'm uniquely positioned to talk about its impacts in my line of work (SEO and digital marketing). Using Screaming Frog to crawl the SerpFocus website itself, I was recently able to clean up some 4XX errors, redirects, and security issues. I would love the opportunity to show you what Screaming Frog could do for you in combination with my consulting services.
Screaming Frog crawls websites to collect data on technical and on-page issues that need to be resolved to improve search engine optimization. Users can configure various tabs and filters to focus in on the most relevant data.
The name Screaming Frog was inspired by a frog that was trying to defend itself from two cats in the backyard of one of the platform's founders.
For in-depth information on how to set up and use Screaming Frog, I'd recommend checking out the Screaming Frog User Guide. I'd also be happy to provide some pointers as part of my consulting services.
Overall, Screaming Frog is my top recommendation for site-crawling SEO tools. That being said, it’s limited to Windows and Mac OS X, must be downloaded, and has a pretty limited free version. Screaming Frog alternatives include Semrush, Xenu, Ahrefs, Sitebulb, DeepCrawl, and Wayback Machine (which is free).
Leave a Reply
You must be logged in to post a comment.