SlideShare a Scribd company logo
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
@rachellcostello brightonSEO
TALK ABOUT WHAT YOU KNOW
What we do every day:
Help people to crawl websites
to get insights, no matter how
many URLs they have.
Recommended fixes for a better site.
@rachellcostello brightonSEO
WHAT WE’LL COVER
Tips and tactics for crawling large sites.
Common pitfalls of the biggest brands.
CRAWLING LARGE SITES
Effectively & Efficiently
@rachellcostello brightonSEO
The sheer scale of enterprise sites and
knowing where to start can be daunting.
@rachellcostello brightonSEO
That’s why you need to start smaller.
@rachellcostello brightonSEO
Targeted crawls can run regularly, allowing
even the largest sites to bypass resource
and time constraints.
@rachellcostello brightonSEO
Good news: you don’t
need to crawl every
URL every time.
TACTICAL CRAWLING
@rachellcostello brightonSEO
You only need enough
data to validate issues.
TACTICAL CRAWLING
@rachellcostello brightonSEO
TACTICAL CRAWLING =
@rachellcostello brightonSEO
Getting the data you need as
quickly as possible.1
Building the bigger picture
from smaller parts.2
@rachellcostello brightonSEO
@rachellcostello brightonSEO
With tactical crawling methods, you can
use segments to give a representative
picture of the whole site.
@rachellcostello brightonSEO
Learn more about a site by breaking it into
manageable, predictable chunks.
@rachellcostello brightonSEO
Depending on the vertical, each site should
be able to be split into 5-7 categories:
@rachellcostello brightonSEO
Category PagesProduct Pages Blog Pages Vanity Pages
Navigational
Pages
Transactional
Pages
FAQ Pages
@rachellcostello brightonSEO
SAMPLING SLICING
@rachellcostello brightonSEO
SAMPLING
@rachellcostello brightonSEO
SAMPLING
This method is about
collecting a percentage
of the whole site.
@rachellcostello brightonSEO
SAMPLING METHOD #1
Start a small crawl and find
areas of crawl waste to
exclude for whole crawl.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Tracking URLs
Pagination URLs
Product Page Parameter URLs
Social Tag URLs
@rachellcostello brightonSEO
SAMPLING METHOD #2
Use level limitation to
assess site breadth without
getting lost in the weeds.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Category Page
Subcategory
Page
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Product PageLevel 4
@rachellcostello brightonSEO
SAMPLING METHOD #3
Crawl a certain number of
examples of each page
type.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Product Page 4Category Page 4 Blog Post 4
Category Page 3
Category Page 2
Category Page 1
Product Page 3
Product Page 2
Product Page 1
Blog Post 3
Blog Post 2
Blog Post 1
@rachellcostello brightonSEO
SLICING
@rachellcostello brightonSEO
SLICING
This method is about
analysing an isolated
section of the site.
@rachellcostello brightonSEO
SLICING METHOD #1
All of the products but none
of the categories
(horizontal slice).
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Category Page
Subcategory
Page
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Product PageLevel 4
@rachellcostello brightonSEO
SLICING METHOD #2
Crawl a distinct section that
serves a separate function
(e.g. blog or separate
mobile site).
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Blog
Blog Category
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Blog PostLevel 4
@rachellcostello brightonSEO
@rachellcostello brightonSEO
1. A strong history of organic traffic.
2. Consistent levels of customer engagement.
+
3. Close monitoring of all changes made.
+
HAVE SUBSETS WITH:
Because when you see the needle move
it will be more meaningful.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Supplement your segments in external data sources too.
Use custom properties,
inclusion rules and
filtered reports to
pre-filter data in:
Analytics tools
Log files
Sitemaps
Google Search Console
@rachellcostello brightonSEO
Tactical crawling allows you to get to know
your website and its structure better.
@rachellcostello brightonSEO
Establishing patterns means you can crawl
more efficiently in future.
@rachellcostello brightonSEO
But, remember to also run annual or
semi-annual full crawls and tech SEO
audits.
@rachellcostello brightonSEO
The takeaways from these are the primary
baselines for planning tactical crawls.
@rachellcostello brightonSEO
HOW THE BIGGEST
BRANDS MEASURE UP
@rachellcostello brightonSEO
I analysed the most recent audits
we’ve completed using tactical
crawling methods for our biggest
enterprise clients.
Here are the key takeaways of 3...
@rachellcostello brightonSEO
FOOD
DELIVERY
TRANSPORTATION
ONLINE
CLASSIFIEDS
@rachellcostello brightonSEO
The key overlapping problem
areas...
@rachellcostello brightonSEO
1
2
3
4
5 Mobile
Internal Linking
Site Speed
Indexing
Internationalisation
@rachellcostello brightonSEO
MAIN
INTERNATIONALISATION
FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add x-default
for non-
regionalised
URLs.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add x-default
for non-
regionalised
URLs.
Translate all
titles and
descriptions to
the correct
language.
@rachellcostello brightonSEO
MAIN INDEXING FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
@rachellcostello brightonSEO
Increase
pagination to
show more
results per
page.
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
Increase
pagination to
show more
results per
page.
Review
orphaned URLs
in the sitemaps.
@rachellcostello brightonSEO
MAIN SITE SPEED FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduce page
HTML
size for high
priority
templates.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduce page
HTML
size for high
priority
templates.
Increase
caching time for
static pages
from minutes to
days.
@rachellcostello brightonSEO
MAIN INTERNAL LINKING
FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
Avoid using
parameter URLs
for tracking.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
Avoid using
parameter URLs
for tracking.
Remove
internal links to
canonicalised
search pages.
@rachellcostello brightonSEO
MAIN MOBILE FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all
pages to
responsive
ASAP.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all
pages to
responsive
ASAP.
Ensure each
mobile-desktop
reciprocal
URL pair has
200 status.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all
pages to
responsive
ASAP.
Ensure each
mobile-desktop
reciprocal
URL pair has
200 status.
Fix content
mismatches
between
mobile and
desktop URLs.
@rachellcostello brightonSEO
Luckily, there are already
resources on these topics!
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Internationalisation - bit.ly/internationalisation-dc
Indexing - bit.ly/indexing-dc
Site Speed - bit.ly/site-speed-dc
Internal Linking - bit.ly/internal-linking-dc
Mobile - bit.ly/mobile-dc
RESOURCES
@rachellcostello brightonSEO
TO WIN IN ENTERPRISE SEO:
Run targeted, agile crawls on smaller sections
of your site.
Better understand internationalisation, indexing,
site speed, internal linking and mobile.
Use the findings to deliver quicker and more
impactful insights into site health.
1
2
3
@rachellcostello brightonSEO
THANK YOU!
Any questions? Send me a tweet:
@DeepCrawl www.deepcrawl.com

More Related Content

PDF
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
PDF
How To Get a 100% Lighthouse Performance Score
PDF
So you think you know canonical tags - Sean Butcher Brighton SEO presentation
PDF
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...
PDF
11 Advanced Uses of Screaming Frog Nov 2019 DMSS
PPTX
20 free SEO Tools you should be using - 20180829
PDF
Rendering SEO Manifesto - Why we need to go beyond JavaScript SEO
PDF
Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
How To Get a 100% Lighthouse Performance Score
So you think you know canonical tags - Sean Butcher Brighton SEO presentation
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...
11 Advanced Uses of Screaming Frog Nov 2019 DMSS
20 free SEO Tools you should be using - 20180829
Rendering SEO Manifesto - Why we need to go beyond JavaScript SEO
Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

What's hot (19)

PPTX
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
PPTX
Google PageSpeed: 5 Steps to 100% (Mobile) Success
PDF
Modern Day Link Building by Jon Cooper
PDF
Technical SEO for international markets - Leonie Mann - Brighton SEO 2021
PDF
The Technical Seo Renaissance - Mike King
PDF
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro...
PDF
UK Top 5,000 Websites; Mobile Site Speed Benchmark - BrightonSEO
PDF
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
PDF
The Ultimate Pagination for SEO
PDF
Can Google properly crawl and index JavaScript? SEO Experiments - Results and...
PDF
Moving URLs: Structural Web changes 
without losing rankings #SearchLove
PDF
SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance...
PDF
TechSEO Boost 2021 - Rendering Strategies: Measuring the Devil’s Details in C...
PPTX
Solving Complex JavaScript Issues and Leveraging Semantic HTML5
PDF
Modern SEO Players Guide
PDF
Headless SEO: Optimising Next Gen Sites | brightonSEO 2021
PDF
Introduction to PWAs & New JS Frameworks for Mobile
PPTX
How to build a scalable content production system.
PPTX
Schema.org and the changing world of Rich Results - SEOEdinburgh Meetup
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
Google PageSpeed: 5 Steps to 100% (Mobile) Success
Modern Day Link Building by Jon Cooper
Technical SEO for international markets - Leonie Mann - Brighton SEO 2021
The Technical Seo Renaissance - Mike King
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro...
UK Top 5,000 Websites; Mobile Site Speed Benchmark - BrightonSEO
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
The Ultimate Pagination for SEO
Can Google properly crawl and index JavaScript? SEO Experiments - Results and...
Moving URLs: Structural Web changes 
without losing rankings #SearchLove
SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance...
TechSEO Boost 2021 - Rendering Strategies: Measuring the Devil’s Details in C...
Solving Complex JavaScript Issues and Leveraging Semantic HTML5
Modern SEO Players Guide
Headless SEO: Optimising Next Gen Sites | brightonSEO 2021
Introduction to PWAs & New JS Frameworks for Mobile
How to build a scalable content production system.
Schema.org and the changing world of Rich Results - SEOEdinburgh Meetup
Ad

Similar to How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl (20)

PDF
BrightonSEO 2017 - SEO quick wins from a technical check
PDF
The Toughest Battles in SEO are Internal - Rachel Costello, Technical SEO, De...
PDF
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
PDF
Optimizing Customer Journeys Online | Digital Growth Unleashed 2019 | Rachel ...
DOCX
Read This FirstAnnielytics.com@AnnieCushingNOTES ABOUT THIS WORKBO.docx
PDF
Site migrations | Brighton SEO 2019
PPTX
Inbound Marketing Tools - SearchFest
PPTX
Technical SEO FTW!
PDF
Webmaster tools (ICMK485)
PDF
Tips for big site SEO - Andrew Smith - SearchNorwich 9
PDF
SEO 101: How to Get Started Winning Google Search Traffic
PPTX
Technical SEO: Crawl Space Management - SEOZone Istanbul 2014
PDF
What You Need to Know About Technical SEO
PPTX
Technical SEO Updated
PDF
Comprehensive SEO audit
PDF
SEO Proposal by AOX Apps
PPTX
Older SEO Presales report from 2011
PPTX
Zero Budget Marketing - AdGlobal360
PDF
Technical SEO.pdf
PDF
The 2014 seo checklist
BrightonSEO 2017 - SEO quick wins from a technical check
The Toughest Battles in SEO are Internal - Rachel Costello, Technical SEO, De...
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Optimizing Customer Journeys Online | Digital Growth Unleashed 2019 | Rachel ...
Read This FirstAnnielytics.com@AnnieCushingNOTES ABOUT THIS WORKBO.docx
Site migrations | Brighton SEO 2019
Inbound Marketing Tools - SearchFest
Technical SEO FTW!
Webmaster tools (ICMK485)
Tips for big site SEO - Andrew Smith - SearchNorwich 9
SEO 101: How to Get Started Winning Google Search Traffic
Technical SEO: Crawl Space Management - SEOZone Istanbul 2014
What You Need to Know About Technical SEO
Technical SEO Updated
Comprehensive SEO audit
SEO Proposal by AOX Apps
Older SEO Presales report from 2011
Zero Budget Marketing - AdGlobal360
Technical SEO.pdf
The 2014 seo checklist
Ad

More from DeepCrawl (17)

PDF
Overcoming Technical SEO Challenges for Enterprise Sites | SearchLeeds 2019 |...
PDF
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
PDF
Owning Enterprise SEO with Effective & Scalable Dashboards - Nick Wilsdon, Pr...
PPTX
Data Studio for SEOs - Pint-sized Marketing Meetup 2019
PDF
The State of Pagination & Infinite Scroll - BrightonSEO April 2019 - Adam Gent
PDF
Speed & Performance Optimisation: How to Meet Users' High Expectations - Rach...
PDF
The Business Value of SEO - Jennifer Hoffman, Marketing Director, DeepCrawl
PDF
How To Survive & Thrive After Mobile First Indexing - Rachel Costello, Techni...
PDF
Efficient AF: Automating SEO Reporting With Google Data Studio - Sam Marsden,...
PDF
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
PDF
URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D...
PDF
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...
PDF
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
PDF
What's the Business Value of Search Data - Jennifer Hoffman, Marketing Direct...
PDF
Cut The Crap: Running Content Audits With Crawlers - Sam Marsden, Technical S...
PDF
Increase Conversions with Crawl Data - Rachel Costello, Technical SEO, DeepCrawl
PPTX
Big Site Migrations- Michal Magdziarz, CEO of DeepCrawl
Overcoming Technical SEO Challenges for Enterprise Sites | SearchLeeds 2019 |...
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
Owning Enterprise SEO with Effective & Scalable Dashboards - Nick Wilsdon, Pr...
Data Studio for SEOs - Pint-sized Marketing Meetup 2019
The State of Pagination & Infinite Scroll - BrightonSEO April 2019 - Adam Gent
Speed & Performance Optimisation: How to Meet Users' High Expectations - Rach...
The Business Value of SEO - Jennifer Hoffman, Marketing Director, DeepCrawl
How To Survive & Thrive After Mobile First Indexing - Rachel Costello, Techni...
Efficient AF: Automating SEO Reporting With Google Data Studio - Sam Marsden,...
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D...
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
What's the Business Value of Search Data - Jennifer Hoffman, Marketing Direct...
Cut The Crap: Running Content Audits With Crawlers - Sam Marsden, Technical S...
Increase Conversions with Crawl Data - Rachel Costello, Technical SEO, DeepCrawl
Big Site Migrations- Michal Magdziarz, CEO of DeepCrawl

Recently uploaded (20)

PDF
EVOLUTION OF RURAL MARKETING IN INDIAN CIVILIZATION
PDF
Digital Marketing in the Age of AI: What CEOs Need to Know - Jennifer Apy, Ch...
DOCX
Parkville marketing plan .......MR.docx
PDF
Prove and Prioritize Profitability in Every Marketing Campaign - Zach Sherrod...
PDF
AFCAT Syllabus 2026 Guide by Best Defence Academy in Lucknow.pdf
PDF
AI & Automation: The Future of Marketing or the End of Creativity - Eric Ritt...
PPTX
Ranking a Webpage with SEO (And Tracking It with the Right Attribution Type a...
PPTX
Your score increases as you pick a category, fill out a long description and ...
PPTX
Presentation - MindfulHeal Digital Ayurveda GTM & Marketing Plan.pptx
PDF
UNIT 2 - 5 DISTRIBUTION IN RURAL MARKETS.pdf
PDF
Coleção Nature .
PDF
UNIT 1 -4 Profile of Rural Consumers (1).pdf
PPTX
UNIT 3 - 5 INDUSTRIAL PRICING.ppt x
PPTX
Assignment 2 Task 1 - How Consumers Use Technology and Its Impact on Their Lives
PDF
DIGITAL MARKETING STRATEGIST IN KASARAGOD
PDF
Fly Emirates SEO case study by Rakesh pathak.pdf
PDF
Mastering Content Strategy in 2025 ss.pdf
PDF
NeuroRank™: The Future of AI-First SEO..
PDF
Unlocking Future Growth: Attract Customers with Automation & Fresh Strategies...
PDF
Digital Marketing Agency in Thrissur with Proven Strategies for Local Growth
EVOLUTION OF RURAL MARKETING IN INDIAN CIVILIZATION
Digital Marketing in the Age of AI: What CEOs Need to Know - Jennifer Apy, Ch...
Parkville marketing plan .......MR.docx
Prove and Prioritize Profitability in Every Marketing Campaign - Zach Sherrod...
AFCAT Syllabus 2026 Guide by Best Defence Academy in Lucknow.pdf
AI & Automation: The Future of Marketing or the End of Creativity - Eric Ritt...
Ranking a Webpage with SEO (And Tracking It with the Right Attribution Type a...
Your score increases as you pick a category, fill out a long description and ...
Presentation - MindfulHeal Digital Ayurveda GTM & Marketing Plan.pptx
UNIT 2 - 5 DISTRIBUTION IN RURAL MARKETS.pdf
Coleção Nature .
UNIT 1 -4 Profile of Rural Consumers (1).pdf
UNIT 3 - 5 INDUSTRIAL PRICING.ppt x
Assignment 2 Task 1 - How Consumers Use Technology and Its Impact on Their Lives
DIGITAL MARKETING STRATEGIST IN KASARAGOD
Fly Emirates SEO case study by Rakesh pathak.pdf
Mastering Content Strategy in 2025 ss.pdf
NeuroRank™: The Future of AI-First SEO..
Unlocking Future Growth: Attract Customers with Automation & Fresh Strategies...
Digital Marketing Agency in Thrissur with Proven Strategies for Local Growth

How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl