JoomlaDay Conference_September 2023 PDF.pdf

12 Advanced Uses
of the Screaming Frog
SEO Spider
@screamingfrog
@oliverbrett
@stephhugman

http://taylortomita.com/seo-graph-portfolio/
@TVYLORTOMITV
@screamingfrog
@oliverbrett
@stephhugman

@screamingfrog
@oliverbrett
@stephhugman

Introduce SF, explain agency
@screamingfrog
@oliverbrett
@stephhugman

Explain history of tool
2011
@screamingfrog
@oliverbrett
@stephhugman

Explain fame
@screamingfrog
@oliverbrett
@stephhugman

“The Swiss
Army Knife
of SEO”
SUN TZU, PROBABLY
@screamingfrog
@oliverbrett
@stephhugman

Screaming
Frog Can…
@screamingfrog
@oliverbrett
@stephhugman

https://www.seerinteractive.com/blog/screaming-frog-guide/
@screamingfrog
@oliverbrett
@stephhugman

We don’t have the
answers.
We have the data.
@screamingfrog
@oliverbrett
@stephhugman

A tool built by SEOs.
For SEOs. And you!
@screamingfrog
@oliverbrett
@stephhugman

And you!
@screamingfrog
@oliverbrett
@stephhugman

3 Rules For Today:
@screamingfrog
@oliverbrett
@stephhugman

1. This isn’t a
sales pitch.
@screamingfrog
@oliverbrett
@stephhugman

€239 A Year!
@screamingfrog
@oliverbrett
@stephhugman

€239 A Year!
Just 60 Pints!!
Just 24 plates of Käsespätzle!!
@screamingfrog
@oliverbrett
@stephhugman

€239 A Year!
Just 24 plates of Käsespätzle!!
@screamingfrog
@oliverbrett
@stephhugman

2. This isn’t a
walkthrough.
@screamingfrog
@oliverbrett
@stephhugman

Links will be
provided for in-depth
guides. Don’t panic.
@screamingfrog
@oliverbrett
@stephhugman

3. Just ask if you
need help!
@screamingfrog
@oliverbrett
@stephhugman

Setting Up and Getting Started
@screamingfrog
@oliverbrett
@stephhugman

Download the software: it runs locally, not online
@screamingfrog
@oliverbrett
@stephhugman

"You need a
supercomputer to
run the Spider"
"You just need 4GB
of RAM and an
SSD"
@screamingfrog
@oliverbrett
@stephhugman
Download the software: it runs locally, not online

Default Screen
Navigating the Interface:
@screamingfrog
@oliverbrett
@stephhugman

Entering a Website
@screamingfrog
@oliverbrett
@stephhugman

Populating Data
@screamingfrog
@oliverbrett
@stephhugman

Crawl Progress
@screamingfrog
@oliverbrett
@stephhugman

Tabs
@screamingfrog
@oliverbrett
@stephhugman

Filtering by Feature
@screamingfrog
@oliverbrett
@stephhugman

Crawl Modes
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

Crawl Modes: Spider Mode vs List Mode
Spider Mode – whole site:
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

Crawl Modes: Spider Mode vs List Mode
Spider Mode – whole site:
List Mode – specific set of URLs:
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

Storage Options
Memory Storage Mode Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

Storage Options
Memory Storage Mode
Uses your RAM to store your
crawl data
Database Storage Mode
Uses your SSD to store your
crawl data
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

Benefits of Database Storage Mode
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Automatically save your crawls

Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Recover previously lost crawls

Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Open crawls significantly quicker

Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:
● Crawl more URLs (millions)

● Crawl more URLs (millions)
● Enable extra features including crawl comparison and change detection
Running a crawl:
@screamingfrog
@oliverbrett
@stephhugman
Running a Crawl:

The Issues Tab
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

The Issues Tab
Issues = an error or issue that should ideally be fixed.
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

The Issues Tab
Opportunities = potential areas for optimisation and improvement.
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

The Issues Tab
Warnings = Not necessarily an issue, but worth checking and fixing
if needed.
Opportunities = potential areas for optimisation and improvement.
Reviewing data:
@screamingfrog
@oliverbrett
@stephhugman
Reviewing Data:

12 Advanced Uses
@screamingfrog
@oliverbrett
@stephhugman

Crawling Complex Sites
@screamingfrog
@oliverbrett
@stephhugman

The number one
question our support
team get asked:
@screamingfrog
@oliverbrett
@stephhugman

Why won’t this Logfile
Analyser crawl my
site?
@screamingfrog
@oliverbrett
@stephhugman

Q: Why isn’t the
spider crawling more
than one page?
@screamingfrog
@oliverbrett
@stephhugman

???
@screamingfrog
@oliverbrett
@stephhugman

A: JAVASCRIPT!!
@screamingfrog
@oliverbrett
@stephhugman

Tricky site 1:
JavaScript websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/CRAWL-
JAVASCRIPT-SEO/
@screamingfrog
@oliverbrett
@stephhugman

Tricky site 2:
big websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/HOW-TO-CRAWL-
LARGE-WEBSITES/
@screamingfrog
@oliverbrett
@stephhugman

What do we mean by
big?
@screamingfrog
@oliverbrett
@stephhugman

(Over) Five Million URLs
@screamingfrog
@oliverbrett
@stephhugman

Do you really need to
crawl the whole
damn site?
@screamingfrog
@oliverbrett
@stephhugman

That’s gonna be a big
Excel doc…
@screamingfrog
@oliverbrett
@stephhugman

Crawl in sections,
subdomains, or
subfolders.
@screamingfrog
@oliverbrett
@stephhugman

Use the exclude
function to avoid
faceted navigation.
@screamingfrog
@oliverbrett
@stephhugman

https://www.johnlew
is.com/browse/men/
mens-
trousers/adidas/alls
aints/gant-
rugger/hymn/kin/sel
ected-
femme/homme/size
=36r/_/N-
ebiZ1z13yvxZ1z0g0g6
Z1z04nruZ1z0s0laZ1z
0vl67Z1z01kl3Z1z0sw
k1 @screamingfrog
@oliverbrett
@stephhugman

Database Storage
Mode
CONFIGURATION > STORAGE > DATABASE STORAGE MODE
@screamingfrog
@oliverbrett
@stephhugman

8GB RAM
@screamingfrog
@oliverbrett
@stephhugman

SSD
@screamingfrog
@oliverbrett
@stephhugman

Tricky site 3:
password protected
websites:
HTTPS://WWW.SCREAMINGFROG.CO.UK/CRAWLING-
PASSWORD-PROTECTED-WEBSITES/
@screamingfrog
@oliverbrett
@stephhugman

Disclaimer:
@screamingfrog
@oliverbrett
@stephhugman

Disclaimer:
You need to know what
the password is.
@screamingfrog
@oliverbrett
@stephhugman

Disclaimer:
You need to know what
the password is.
We can’t help you hack it…
@screamingfrog
@oliverbrett
@stephhugman

(Maybe like a dev site
for example…)
@screamingfrog
@oliverbrett
@stephhugman

Forms Based
Authentication
CONFIGURE > AUTHENTICATION
@screamingfrog
@oliverbrett
@stephhugman

Disclaimer 2:
@screamingfrog
@oliverbrett
@stephhugman

Disclaimer
2:
This will
click every
link on the
page.
@screamingfrog
@oliverbrett
@stephhugman

Disclaimer 2:
This will click every
link on the page.
Yes, really… @screamingfrog
@oliverbrett
@stephhugman

Installed 108
WordPress
Themes
@screamingfrog
@oliverbrett
@stephhugman

Installed 108
WordPress
Themes
Activated 8
themes
@screamingfrog
@oliverbrett
@stephhugman

Installed 108
WordPress
Themes
Activated 8
themes
Deleted posts
@screamingfrog
@oliverbrett
@stephhugman

Installed 108
WordPress
Themes
Activated 8
themes
Deleted posts
Logged itself off
@screamingfrog
@oliverbrett
@stephhugman

Limit the admin
rights of the
account
@screamingfrog
@oliverbrett
@stephhugman

Limit the admin
rights of the
account
Blacklist URLs
using the
exclude feature @screamingfrog
@oliverbrett
@stephhugman

Limit the admin
rights of the
account
Blacklist URLs
using the
exclude feature
Sense Check the
site first…
@screamingfrog
@oliverbrett
@stephhugman

Limit the admin
rights of the
account
Blacklist URLs
using the
exclude feature
Unplug your fax
machine
Sense Check the
site first…
@screamingfrog
@oliverbrett
@stephhugman

API Integration
@screamingfrog
@oliverbrett
@stephhugman

Google
Analytics
HTTPS://WWW.SCREAMINGFROG.CO.UK
/SEO-SPIDER/U SER-
GUIDE/CONFIGURATION/#GOOGLE -
ANALYTICS-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

Google
Analytics
HTTPS://WWW.SCREAMINGFROG.CO.UK
/SEO-SPIDER/U SER-
ANALYTICS-INTEGRATION

Google Analytics (4)
Sessions
% New Sessions
New Users
Bounce Rate/Engagement Rate
Page Views Per Session
Avg Session Duration
Page Value
Goal Conversion Rate
Goal Completions All
Goal Value All
Sessions Above 0
Bounce Rate Above 70%
No GA Data Pages
Non-Indexable with GA Data
Orphan URLs
@screamingfrog
@oliverbrett
@stephhugman

Google Search
Console
HTTPS://WWW.SCREAMINGFROG.C
O.UK/SEO-SPIDER/USER-
SEARCH-CONSOLE-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

Google Search Console
Clicks
Impressions
CTR
Position
Clicks above 0
No GSC Data
Non-Indexible with GSC Data
Orphan URLs
@screamingfrog
@oliverbrett
@stephhugman

ahrefs
H T T P S : / / W W W . S C R E A M I N G F R O G
. C O . U K / S E O - S P I D E R / U S E R -
G U I D E / C O N F I G U R A T I ON / # A H R E F
S
@screamingfrog
@oliverbrett
@stephhugman

Majestic
H T T P S : / / W W W . S C R E A M I N G F R O G
. C O . U K / S E O - S P I D E R / U S E R -
G U I D E / C O N F I G U R A T I ON / # M A J E S
T I C
@screamingfrog
@oliverbrett
@stephhugman

Moz
HTTPS://WWW.SCREAMINGFRO
G.CO.UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#MOZ
@screamingfrog
@oliverbrett
@stephhugman

Get Ya DAs
Done
1. Put the spider into list
mode
2. Connect Moz API, enable
DA
3. Paste in list of coverage
URLs
4. Crawl
5. Export list of URLs + DAs
6. Make link builders happy
@screamingfrog
@oliverbrett
@stephhugman

PageSpeed
Insights
HTTPS://WWW.SCREAMINGFROG.C
O.UK/SEO-SPIDER/USER-
GUIDE/CONFIGURATION/#PAGESPE
ED-INSIGHTS-INTEGRATION
@screamingfrog
@oliverbrett
@stephhugman

Custom Search
@screamingfrog
@oliverbrett
@stephhugman

@screamingfrog
@oliverbrett
@stephhugman
How many of our
products are out of
stock?

@screamingfrog
@oliverbrett
@stephhugman
How many of our
products are out of
stock?
Uhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhh
hhhhhhhhhhhh

@screamingfrog
@oliverbrett
@stephhugman
Which pages are
missing in GA4?

@screamingfrog
@oliverbrett
@stephhugman
Which pages are
missing in GA4?
GA444444444444444
44444444444444444
44444444444444444
4444444444444444?

Check the HTML for:
Contains: Text- ‘Out of Stock’
Does Not Contain: Text- [Google Tag Manager Tracking Code]
@screamingfrog
@oliverbrett
@stephhugman

Custom Extraction
@screamingfrog
@oliverbrett
@stephhugman

Custom Extraction
https://ww
w.screami
ngfrog.co.
uk/web-
scraping/ @screamingfrog
@oliverbrett
@stephhugman

Custom
Search
Custom
Extraction
@screamingfrog
@oliverbrett
@stephhugman

Xpath/Regex
@screamingfrog
@oliverbrett
@stephhugman

JoomlaDay Conference_September 2023 PDF.pdf

Visualising Data
@screamingfrog
@oliverbrett
@stephhugman

HTTPS://WWW.SCREAMINGFROG.CO.UK/SEO-SPIDER-10/
HTTPS://WWW.SCREAMINGFROG.CO.UK/SITE -ARCHITECTURE-CRAWL-
VISUALISATIONS/
@screamingfrog
@oliverbrett
@stephhugman

Crawl Tree Graph
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK
@screamingfrog
@oliverbrett
@stephhugman

Directory Tree Graph
FOLLOWS URL STRUCTURE
@screamingfrog
@oliverbrett
@stephhugman

Force-Directed Crawl
Diagram
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK
@screamingfrog
@oliverbrett
@stephhugman

Force-Directed
Directory Tree
Diagram
FOLLOWS URL STRUCTURE
@screamingfrog
@oliverbrett
@stephhugman

Inlink Anchor Text
Word Cloud
WHAT TEXT ARE YOU LINKING WITH?
@screamingfrog
@oliverbrett
@stephhugman

Body Text Word
Cloud
WHAT IS YOUR CONTENT ABOUT?
@screamingfrog
@oliverbrett
@stephhugman

Are these pointless?
@screamingfrog
@oliverbrett
@stephhugman

Yeah…
@screamingfrog
@oliverbrett
@stephhugman

Can we make more of
em?
@screamingfrog
@oliverbrett
@stephhugman

Also yes!
@screamingfrog
@oliverbrett
@stephhugman

3D Force-Directed
Crawl Diagram
SHORTEST PATH TO THE PAGE THAT THE SPIDER TOOK,
IN ✨3D✨
@screamingfrog
@oliverbrett
@stephhugman

3D Force-Directed
Directory Tree
Diagram
FOLLOWS URL STRUCTURE, IN ✨3D✨
@screamingfrog
@oliverbrett
@stephhugman

Segments
@screamingfrog
@oliverbrett
@stephhugman

● X
X
@screamingfrog
@oliverbrett
@stephhugman

X
@screamingfrog
@oliverbrett
@stephhugman

Auditing Redirects in a Site
Migration
@screamingfrog
@oliverbrett
@stephhugman

Common Issues in a Site Migration

Common Issues with Redirects:
@screamingfrog
@oliverbrett
@stephhugman

Multiple hops in a redirect chain
Start
Finish
@screamingfrog
@oliverbrett
@stephhugman

Start
End
point
Redirecting to the wrong target
Target

Start
Finish
Redirecting to a 404 page

Upload your Original URLs in List Mode
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

Configure the Crawl to Follow Redirects
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

Export a Report of all the Redirects
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

The report includes
every URL in the original
upload and the response codes in a
single report,
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

recording the start and final URLs
and statuses,
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

as well as the full hops if there are
several redirects in
a chain.
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

You can easily see if any URLs are
missing when they shouldn’t be,
or weren't redirected correctly.
@screamingfrog
@oliverbrett
@stephhugman
How to audit
redirects in a
site migration:

Sitemaps
@screamingfrog
@oliverbrett
@stephhugman

Sitemaps
Helping Google better understand your website
@screamingfrog
@oliverbrett
@stephhugman

Generating and Auditing
Sitemaps
@screamingfrog
@oliverbrett
@stephhugman

Crawl a Site,
then Click "Sitemaps" in the Main Nav
@screamingfrog
@oliverbrett
@stephhugman
How to
create
sitemaps:

@screamingfrog
@oliverbrett
@stephhugman
How to
create
sitemaps:
● All pages with a 200 response included
Crawl a Site,

● All pages with a 200 response included
● Can include images, PDFs, and canonicalised/paginated/noindex URLs
Crawl a Site,
@screamingfrog
@oliverbrett
@stephhugman
How to
create
sitemaps:

Configure the Spider
to Crawl Sitemaps Pre-crawl
@screamingfrog
@oliverbrett
@stephhugman
How to
audit
sitemaps:

Enable Post-crawl Analysis
to see Sitemap Data Reports
Reports:
● URLs in sitemap
● URLs not in sitemap
● Orphan URLs
● Non-indexable URLs in sitemap
● URLs in multiple sitemaps
● XML sitemap with over 50k URLs
● XML sitemap with over 50mb
How to
audit
sitemaps:

Auditing Content (with Data)
@screamingfrog
@oliverbrett
@stephhugman

How to Audit Content with Spider Data
@screamingfrog
@oliverbrett
@stephhugman

Content-related Data Reports
Word count and average words per sentence
@screamingfrog
@oliverbrett
@stephhugman

Spelling and grammar errors
@screamingfrog
@oliverbrett
@stephhugman

Readability levels
@screamingfrog
@oliverbrett
@stephhugman

Lorem ipsum placeholder text
@screamingfrog
@oliverbrett
@stephhugman

Exact/near duplicates
@screamingfrog
@oliverbrett
@stephhugman

Low content pages
@screamingfrog
@oliverbrett
@stephhugman

Auditing Structured Data
@screamingfrog
@oliverbrett
@stephhugman

Structured Data
Helping Google better understand your pages and their content,
enabling rich results in the SERPs
@screamingfrog
@oliverbrett
@stephhugman

Structured Data
Helping Google better understand your pages and their content,
enabling rich results in the SERPs
Review
stars
Cooking
time
@screamingfrog
@oliverbrett
@stephhugman

Configure the Spider
to Enable Structured Data Options
@screamingfrog
@oliverbrett
@stephhugman
How to audit
structured data:

Export Relevant Structured Data Reports
● Contains structured data
● Missing
● Validation errors/ warnings
● Parse errors
● Microdata/JSON-LD/RDFa
URLs
@screamingfrog
@oliverbrett
@stephhugman
How to audit
structured data:

Scheduling Crawls
@screamingfrog
@oliverbrett
@stephhugman

Choose the Regularity, URLs, and Give it a Name
How to Schedule Crawls:
@screamingfrog
@oliverbrett
@stephhugman
How to schedule crawls:

Crawling in the Cloud
@screamingfrog
@oliverbrett
@stephhugman

Benefits of Running the Spider in the Cloud
@screamingfrog
@oliverbrett
@stephhugman
● Can run crawls without your computer
even being switched on.

@screamingfrog
@oliverbrett
@stephhugman
● Frees up your resources (RAM) to do
other things.

@screamingfrog
@oliverbrett
@stephhugman
other things.
● Can run several crawls at once in the
cloud on just the one license.

other things.
● Can run several crawls at once in the
cloud on just the one license.
● Easily access your crawls and data
anywhere, anytime.
@screamingfrog
@oliverbrett
@stephhugman

Summary
@screamingfrog
@oliverbrett
@stephhugman

In summary...
● 1. Crawling complex sites
● 2. Integrating with APIs
● 3. Custom Search
● 4. Custom Extraction
● 5. Visualising Data
● 6. Making Segments
● 7. Auditing Migration Redirects
● 8. Generating/Auditing Sitemaps
● 9. Auditing Content with Data
● 10. Auditing Structured Data
● 11. Scheduling Crawls
● 12. Crawling in the Cloud
@screamingfrog
@oliverbrett
@stephhugman

Resources
● support@screamingfrog.co.uk
● @screamingfrog on Twitter/X
● @oliverbrett
● @stephhugman
● https://www.screamingfrog.co.uk/seo-spider/user-guide/
● https://www.screamingfrog.co.uk/seo-spider/tutorials/
● https://www.screamingfrog.co.uk/seo-spider/faq/

Thank you!
https://www.screamingfrog.co.uk/
info@screamingfrog.co.uk
@screamingfrog
@oliverbrett
@stephhugman

JoomlaDay Conference_September 2023 PDF.pdf

More Related Content

Similar to JoomlaDay Conference_September 2023 PDF.pdf (20)

Recently uploaded (20)

JoomlaDay Conference_September 2023 PDF.pdf