SlideShare a Scribd company logo
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
@rachellcostello brightonSEO
TALK ABOUT WHAT YOU KNOW
What we do every day:
Help people to crawl websites
to get insights, no matter how
many URLs they have.
Recommended fixes for a better site.
@rachellcostello brightonSEO
WHAT WE’LL COVER
Tips and tactics for crawling large sites.
Common pitfalls of the biggest brands.
CRAWLING LARGE SITES
Effectively & Efficiently
@rachellcostello brightonSEO
The sheer scale of enterprise sites and
knowing where to start can be daunting.
@rachellcostello brightonSEO
That’s why you need to start smaller.
@rachellcostello brightonSEO
Targeted crawls can run regularly, allowing
even the largest sites to bypass resource
and time constraints.
@rachellcostello brightonSEO
Good news: you don’t
need to crawl every
URL every time.
TACTICAL CRAWLING
@rachellcostello brightonSEO
You only need enough
data to validate issues.
TACTICAL CRAWLING
@rachellcostello brightonSEO
TACTICAL CRAWLING =
@rachellcostello brightonSEO
Getting the data you need as
quickly as possible.1
Building the bigger picture
from smaller parts.2
@rachellcostello brightonSEO
@rachellcostello brightonSEO
With tactical crawling methods, you can
use segments to give a representative
picture of the whole site.
@rachellcostello brightonSEO
Learn more about a site by breaking it into
manageable, predictable chunks.
@rachellcostello brightonSEO
Depending on the vertical, each site should
be able to be split into 5-7 categories:
@rachellcostello brightonSEO
Category PagesProduct Pages Blog Pages Vanity Pages
Navigational
Pages
Transactional
Pages
FAQ Pages
@rachellcostello brightonSEO
SAMPLING SLICING
@rachellcostello brightonSEO
SAMPLING
@rachellcostello brightonSEO
SAMPLING
This method is about
collecting a percentage
of the whole site.
@rachellcostello brightonSEO
SAMPLING METHOD #1
Start a small crawl and find
areas of crawl waste to
exclude for whole crawl.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Tracking URLs
Pagination URLs
Product Page Parameter URLs
Social Tag URLs
@rachellcostello brightonSEO
SAMPLING METHOD #2
Use level limitation to
assess site breadth without
getting lost in the weeds.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Category Page
Subcategory
Page
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Product PageLevel 4
@rachellcostello brightonSEO
SAMPLING METHOD #3
Crawl a certain number of
examples of each page
type.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Product Page 4Category Page 4 Blog Post 4
Category Page 3
Category Page 2
Category Page 1
Product Page 3
Product Page 2
Product Page 1
Blog Post 3
Blog Post 2
Blog Post 1
@rachellcostello brightonSEO
SLICING
@rachellcostello brightonSEO
SLICING
This method is about
analysing an isolated
section of the site.
@rachellcostello brightonSEO
SLICING METHOD #1
All of the products but none
of the categories
(horizontal slice).
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Category Page
Subcategory
Page
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Product PageLevel 4
@rachellcostello brightonSEO
SLICING METHOD #2
Crawl a distinct section that
serves a separate function
(e.g. blog or separate
mobile site).
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Homepage
Category Page
Subcategory
Page
Category Page Blog
Blog Category
Subcategory
Page
Level 1
Level 2
Level 3
Product Page Product Page Blog PostLevel 4
@rachellcostello brightonSEO
@rachellcostello brightonSEO
1. A strong history of organic traffic.
2. Consistent levels of customer engagement.
+
3. Close monitoring of all changes made.
+
HAVE SUBSETS WITH:
Because when you see the needle move
it will be more meaningful.
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Supplement your segments in external data sources too.
Use custom properties,
inclusion rules and
filtered reports to
pre-filter data in:
Analytics tools
Log files
Sitemaps
Google Search Console
@rachellcostello brightonSEO
Tactical crawling allows you to get to know
your website and its structure better.
@rachellcostello brightonSEO
Establishing patterns means you can crawl
more efficiently in future.
@rachellcostello brightonSEO
But, remember to also run annual or
semi-annual full crawls and tech SEO
audits.
@rachellcostello brightonSEO
The takeaways from these are the primary
baselines for planning tactical crawls.
@rachellcostello brightonSEO
HOW THE BIGGEST
BRANDS MEASURE UP
@rachellcostello brightonSEO
I analysed the most recent audits
we’ve completed using tactical
crawling methods for our biggest
enterprise clients.
Here are the key takeaways of 3...
@rachellcostello brightonSEO
FOOD
DELIVERY
TRANSPORTATION
ONLINE
CLASSIFIEDS
@rachellcostello brightonSEO
The key overlapping problem
areas...
@rachellcostello brightonSEO
1
2
3
4
5 Mobile
Internal Linking
Site Speed
Indexing
Internationalisation
@rachellcostello brightonSEO
MAIN
INTERNATIONALISATION
FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add x-default
for non-
regionalised
URLs.
@rachellcostello brightonSEO
Removing URLs
for nonsensical
locales.
Use location to
show relevant
results.
Allow users to
toggle between
languages.
Add x-default
for non-
regionalised
URLs.
Translate all
titles and
descriptions to
the correct
language.
@rachellcostello brightonSEO
MAIN INDEXING FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
@rachellcostello brightonSEO
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
@rachellcostello brightonSEO
Increase
pagination to
show more
results per
page.
Replace 302
redirects with
301 redirects.
Fix any server
errors in Google
Search Console
‘Crawl Errors’
report.
Ensure all
content
elements are
unique and
relevant.
Increase
pagination to
show more
results per
page.
Review
orphaned URLs
in the sitemaps.
@rachellcostello brightonSEO
MAIN SITE SPEED FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduce page
HTML
size for high
priority
templates.
@rachellcostello brightonSEO
Lazy-load
below-the-fold
images.
Eliminate use of
unnecessary
plugins.
Remove unused
and/or
render-blocking
scripts.
Reduce page
HTML
size for high
priority
templates.
Increase
caching time for
static pages
from minutes to
days.
@rachellcostello brightonSEO
MAIN INTERNAL LINKING
FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
Avoid using
parameter URLs
for tracking.
@rachellcostello brightonSEO
Remove
redirect loops
and chains.
Reinstate links
to pages with
10+ impressions
and/or visits.
Remove
internal links to
nofollow pages.
Avoid using
parameter URLs
for tracking.
Remove
internal links to
canonicalised
search pages.
@rachellcostello brightonSEO
MAIN MOBILE FIXES
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all
pages to
responsive
ASAP.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all
pages to
responsive
ASAP.
Ensure each
mobile-desktop
reciprocal
URL pair has
200 status.
@rachellcostello brightonSEO
Avoid using
‘maximum-scale
=1’ in viewport
settings.
Change
‘user-scalable=
no’ to ‘yes’ in
viewport
settings.
Migrate all
pages to
responsive
ASAP.
Ensure each
mobile-desktop
reciprocal
URL pair has
200 status.
Fix content
mismatches
between
mobile and
desktop URLs.
@rachellcostello brightonSEO
Luckily, there are already
resources on these topics!
@rachellcostello brightonSEO
@rachellcostello brightonSEO
Internationalisation - bit.ly/internationalisation-dc
Indexing - bit.ly/indexing-dc
Site Speed - bit.ly/site-speed-dc
Internal Linking - bit.ly/internal-linking-dc
Mobile - bit.ly/mobile-dc
RESOURCES
@rachellcostello brightonSEO
TO WIN IN ENTERPRISE SEO:
Run targeted, agile crawls on smaller sections
of your site.
Better understand internationalisation, indexing,
site speed, internal linking and mobile.
Use the findings to deliver quicker and more
impactful insights into site health.
1
2
3
@rachellcostello brightonSEO
THANK YOU!
Any questions? Send me a tweet:
@DeepCrawl www.deepcrawl.com

More Related Content

PDF
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
PDF
How To Get a 100% Lighthouse Performance Score
PDF
So you think you know canonical tags - Sean Butcher Brighton SEO presentation
PDF
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...
PDF
11 Advanced Uses of Screaming Frog Nov 2019 DMSS
PPTX
20 free SEO Tools you should be using - 20180829
PDF
Rendering SEO Manifesto - Why we need to go beyond JavaScript SEO
PDF
Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
How To Get a 100% Lighthouse Performance Score
So you think you know canonical tags - Sean Butcher Brighton SEO presentation
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...
11 Advanced Uses of Screaming Frog Nov 2019 DMSS
20 free SEO Tools you should be using - 20180829
Rendering SEO Manifesto - Why we need to go beyond JavaScript SEO
Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

What's hot (19)

PPTX
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
PPTX
Google PageSpeed: 5 Steps to 100% (Mobile) Success
PDF
Modern Day Link Building by Jon Cooper
PDF
Technical SEO for international markets - Leonie Mann - Brighton SEO 2021
PDF
The Technical Seo Renaissance - Mike King
PDF
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro...
PDF
UK Top 5,000 Websites; Mobile Site Speed Benchmark - BrightonSEO
PDF
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
PDF
The Ultimate Pagination for SEO
PDF
Can Google properly crawl and index JavaScript? SEO Experiments - Results and...
PDF
Moving URLs: Structural Web changes 
without losing rankings #SearchLove
PDF
SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance...
PDF
TechSEO Boost 2021 - Rendering Strategies: Measuring the Devil’s Details in C...
PPTX
Solving Complex JavaScript Issues and Leveraging Semantic HTML5
PDF
Modern SEO Players Guide
PDF
Headless SEO: Optimising Next Gen Sites | brightonSEO 2021
PDF
Introduction to PWAs & New JS Frameworks for Mobile
PPTX
How to build a scalable content production system.
PPTX
Schema.org and the changing world of Rich Results - SEOEdinburgh Meetup
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
Google PageSpeed: 5 Steps to 100% (Mobile) Success
Modern Day Link Building by Jon Cooper
Technical SEO for international markets - Leonie Mann - Brighton SEO 2021
The Technical Seo Renaissance - Mike King
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro...
UK Top 5,000 Websites; Mobile Site Speed Benchmark - BrightonSEO
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
The Ultimate Pagination for SEO
Can Google properly crawl and index JavaScript? SEO Experiments - Results and...
Moving URLs: Structural Web changes 
without losing rankings #SearchLove
SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance...
TechSEO Boost 2021 - Rendering Strategies: Measuring the Devil’s Details in C...
Solving Complex JavaScript Issues and Leveraging Semantic HTML5
Modern SEO Players Guide
Headless SEO: Optimising Next Gen Sites | brightonSEO 2021
Introduction to PWAs & New JS Frameworks for Mobile
How to build a scalable content production system.
Schema.org and the changing world of Rich Results - SEOEdinburgh Meetup
Ad

Similar to How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl (20)

PDF
The Toughest Battles in SEO are Internal - Rachel Costello, Technical SEO, De...
PDF
How to Survive & Thrive After Mobile-first Indexing | Rachel Costello, Techni...
PDF
Optimizing Customer Journeys Online | Digital Growth Unleashed 2019 | Rachel ...
PDF
Inbound 2017: Back to Our Roots with Technical SEO
PDF
Managing filters and facets on your ecommerce website - Bristol SEO and Readi...
PDF
SearchStars 2018: Rachel Costello - Deepcrawl
PPTX
Is it easier to be an SEO for a small business or a big business?
PPTX
Emily Mace BrightonSEO Talk September 2017
PDF
Making HREFLANG Manageable: Search Marketing Summit (Sydney)
PDF
BrightonSEO 2017 - SEO quick wins from a technical check
PPTX
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!
PDF
How to Incorporate ML in your SERP Analysis, Lazarina Stoy -BrightonSEO Oct, ...
PPTX
How to disrupt established markets with SEO in 2015 - LOGIN 2015
PDF
Speed & Performance Optimisation: How to Meet Users' High Expectations - Rach...
PDF
Hreflang: 
Is it really still necessary and 
how to use it successfully in 2023
PDF
SearchLove San Diego 2018 | Aleyda Solis | Moving URLs
PDF
Mauro Cattaneo - Why hreflang is crucial to international SEO success - Brigh...
PDF
Improve your Wordpress SEO Strategy
PDF
Online SEO Meetup
PDF
Website Audit Report Sample
The Toughest Battles in SEO are Internal - Rachel Costello, Technical SEO, De...
How to Survive & Thrive After Mobile-first Indexing | Rachel Costello, Techni...
Optimizing Customer Journeys Online | Digital Growth Unleashed 2019 | Rachel ...
Inbound 2017: Back to Our Roots with Technical SEO
Managing filters and facets on your ecommerce website - Bristol SEO and Readi...
SearchStars 2018: Rachel Costello - Deepcrawl
Is it easier to be an SEO for a small business or a big business?
Emily Mace BrightonSEO Talk September 2017
Making HREFLANG Manageable: Search Marketing Summit (Sydney)
BrightonSEO 2017 - SEO quick wins from a technical check
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!
How to Incorporate ML in your SERP Analysis, Lazarina Stoy -BrightonSEO Oct, ...
How to disrupt established markets with SEO in 2015 - LOGIN 2015
Speed & Performance Optimisation: How to Meet Users' High Expectations - Rach...
Hreflang: 
Is it really still necessary and 
how to use it successfully in 2023
SearchLove San Diego 2018 | Aleyda Solis | Moving URLs
Mauro Cattaneo - Why hreflang is crucial to international SEO success - Brigh...
Improve your Wordpress SEO Strategy
Online SEO Meetup
Website Audit Report Sample
Ad

More from DeepCrawl (16)

PDF
Overcoming Technical SEO Challenges for Enterprise Sites | SearchLeeds 2019 |...
PDF
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
PDF
Owning Enterprise SEO with Effective & Scalable Dashboards - Nick Wilsdon, Pr...
PPTX
Data Studio for SEOs - Pint-sized Marketing Meetup 2019
PDF
The State of Pagination & Infinite Scroll - BrightonSEO April 2019 - Adam Gent
PDF
The Business Value of SEO - Jennifer Hoffman, Marketing Director, DeepCrawl
PDF
How To Survive & Thrive After Mobile First Indexing - Rachel Costello, Techni...
PDF
Efficient AF: Automating SEO Reporting With Google Data Studio - Sam Marsden,...
PDF
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
PDF
URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D...
PDF
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...
PDF
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
PDF
What's the Business Value of Search Data - Jennifer Hoffman, Marketing Direct...
PDF
Cut The Crap: Running Content Audits With Crawlers - Sam Marsden, Technical S...
PDF
Increase Conversions with Crawl Data - Rachel Costello, Technical SEO, DeepCrawl
PPTX
Big Site Migrations- Michal Magdziarz, CEO of DeepCrawl
Overcoming Technical SEO Challenges for Enterprise Sites | SearchLeeds 2019 |...
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
Owning Enterprise SEO with Effective & Scalable Dashboards - Nick Wilsdon, Pr...
Data Studio for SEOs - Pint-sized Marketing Meetup 2019
The State of Pagination & Infinite Scroll - BrightonSEO April 2019 - Adam Gent
The Business Value of SEO - Jennifer Hoffman, Marketing Director, DeepCrawl
How To Survive & Thrive After Mobile First Indexing - Rachel Costello, Techni...
Efficient AF: Automating SEO Reporting With Google Data Studio - Sam Marsden,...
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D...
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
What's the Business Value of Search Data - Jennifer Hoffman, Marketing Direct...
Cut The Crap: Running Content Audits With Crawlers - Sam Marsden, Technical S...
Increase Conversions with Crawl Data - Rachel Costello, Technical SEO, DeepCrawl
Big Site Migrations- Michal Magdziarz, CEO of DeepCrawl

Recently uploaded (20)

PDF
Internet Marketing Company Report KUBS 2025 Spring.pdf
PPTX
Best Leather Products in Trichy | minibaskett.pptx
PDF
Google - Prompt Engineering - como ser um expert
PDF
AI & Automation: The Future of Marketing or the End of Creativity - Eric Ritt...
PDF
DIGITAL MARKETING STRATEGIST IN KASARAGOD
PPTX
Find_Your_Georgia_Campaign_Proposal.pptx
PDF
Digital Marketing in the Age of AI: What CEOs Need to Know - Jennifer Apy, Ch...
PDF
AI Strategy That Scales - Dan Banas, AI, Strategy, Analytics Consulting
PPTX
Social_Media_Marketing_Presentation.pptx
PDF
Future Retail Disruption Trends and Observations
PDF
EVOLUTION OF RURAL MARKETING IN INDIAN CIVILIZATION
PPTX
Final Project parkville.............pptx
PDF
Digital Marketing Trends: Experts Insights on How to Gain a Competitive Edge ...
DOCX
AL-ahly Sabbour un official strategic plan.docx
PDF
Ramjilal Ramsaroop || Trending Branding
PDF
AI, Automation & SEO: The New Marketing PlaybooK - Eric Ritter, Digital Neighbor
PDF
Prove and Prioritize Profitability in Every Marketing Campaign - Zach Sherrod...
PPTX
Ranking a Webpage with SEO (And Tracking It with the Right Attribution Type a...
PDF
Buy verified Advcash accounts In This Year 2025
PPTX
Your score increases as you pick a category, fill out a long description and ...
Internet Marketing Company Report KUBS 2025 Spring.pdf
Best Leather Products in Trichy | minibaskett.pptx
Google - Prompt Engineering - como ser um expert
AI & Automation: The Future of Marketing or the End of Creativity - Eric Ritt...
DIGITAL MARKETING STRATEGIST IN KASARAGOD
Find_Your_Georgia_Campaign_Proposal.pptx
Digital Marketing in the Age of AI: What CEOs Need to Know - Jennifer Apy, Ch...
AI Strategy That Scales - Dan Banas, AI, Strategy, Analytics Consulting
Social_Media_Marketing_Presentation.pptx
Future Retail Disruption Trends and Observations
EVOLUTION OF RURAL MARKETING IN INDIAN CIVILIZATION
Final Project parkville.............pptx
Digital Marketing Trends: Experts Insights on How to Gain a Competitive Edge ...
AL-ahly Sabbour un official strategic plan.docx
Ramjilal Ramsaroop || Trending Branding
AI, Automation & SEO: The New Marketing PlaybooK - Eric Ritter, Digital Neighbor
Prove and Prioritize Profitability in Every Marketing Campaign - Zach Sherrod...
Ranking a Webpage with SEO (And Tracking It with the Right Attribution Type a...
Buy verified Advcash accounts In This Year 2025
Your score increases as you pick a category, fill out a long description and ...

How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl