Read Anytime Anywhere Easy Ebook Downloads at ebookmeta.

com

Understanding Search Engines Dirk Lewandowski

https://ebookmeta.com/product/understanding-search-engines-
dirk-lewandowski/

OR CLICK HERE

DOWLOAD EBOOK

Visit and Get More Ebook Downloads Instantly at https://ebookmeta.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Custom Search - Discover More: : A Complete Guide to

Google Programmable Search Engines 1st Edition Irina
Shamaeva
https://ebookmeta.com/product/custom-search-discover-more-a-complete-
guide-to-google-programmable-search-engines-1st-edition-irina-
shamaeva/
ebookmeta.com

Custom Search Discover more A Complete Guide to Google

Programmable Search Engines 1st Edition Irina Shamaeva
David Michael Galley
https://ebookmeta.com/product/custom-search-discover-more-a-complete-
guide-to-google-programmable-search-engines-1st-edition-irina-
shamaeva-david-michael-galley/
ebookmeta.com

Exodus V Plague Book 13 1st Edition Patton Dirk Dirk

Patton

https://ebookmeta.com/product/exodus-v-plague-book-13-1st-edition-
patton-dirk-dirk-patton/

ebookmeta.com

Biochemistry 7th Edition Reginald H. Garrett

https://ebookmeta.com/product/biochemistry-7th-edition-reginald-h-
garrett/

ebookmeta.com
Scent of Deception (Bonds of Steele Omegaverse Book 3) 1st
Edition Laurel Night

https://ebookmeta.com/product/scent-of-deception-bonds-of-steele-
omegaverse-book-3-1st-edition-laurel-night/

ebookmeta.com

Tracing Textile Production from the Viking Age to the

Middle Ages Tools Textiles Texts and Contexts 2nd Edition
Ingvild Øye
https://ebookmeta.com/product/tracing-textile-production-from-the-
viking-age-to-the-middle-ages-tools-textiles-texts-and-contexts-2nd-
edition-ingvild-oye/
ebookmeta.com

Dragonlance Shadow of the Dragon Queen Dungeons Dragons

Adventure Book 1st Edition Wizards Rpg Team

https://ebookmeta.com/product/dragonlance-shadow-of-the-dragon-queen-
dungeons-dragons-adventure-book-1st-edition-wizards-rpg-team/

ebookmeta.com

Cultural Intelligence in the World of Work Past Present

Future 1st Edition Yuan Liao

https://ebookmeta.com/product/cultural-intelligence-in-the-world-of-
work-past-present-future-1st-edition-yuan-liao/

ebookmeta.com

Honey: A Miraculous Product of Nature 1st Edition

https://ebookmeta.com/product/honey-a-miraculous-product-of-
nature-1st-edition/

ebookmeta.com
I Know Body Parts Mary Rose Osburn

https://ebookmeta.com/product/i-know-body-parts-mary-rose-osburn/

ebookmeta.com
Dirk Lewandowski

Understanding
Search Engines
Understanding Search Engines
Dirk Lewandowski

Understanding
Search Engines
Dirk Lewandowski
Department of Information
Hamburg University of Applied Sciences
Hamburg, Germany

ISBN 978-3-031-22788-2 ISBN 978-3-031-22789-9 (eBook)

https://doi.org/10.1007/978-3-031-22789-9

Translation from the German language edition: “Suchmaschinen verstehen” by Dirk Lewandowski,
# Springer 2021. Published by Springer Vieweg, Berlin, Heidelberg. All Rights Reserved.

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland
AG 2023
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface

This book aims to give a comprehensive introduction to search engines. The

particularity of this book is that it looks at the subject from different angles. These
are, in particular, technology, use, Internet-based research, economics, and societal
significance. In this way, I want to reflect the complexity of the search engines and
Web search as a whole. I am convinced that only such a comprehensive view does
justice to the topic and enables a real understanding.
A German-language version of this book has been available for several years.
This English edition follows the third German edition of 2021. I am pleased that the
publisher has made this international edition possible.
In this translation, care has been taken to adapt to the international context where
necessary. However, for many examples, it does not matter in which country a
search was carried out or a screenshot was taken. However, the references cited in
the text were adapted where English-language sources were available. The further
reading sections at the end of the chapters have also been adapted.
I would like to thank all those who have asked and encouraged me over the years
to produce an English edition. So, here it is, and I hope it will be as valuable to many
readers as the German editions have been.

Hamburg, Germany Dirk Lewandowski

September 2022

v
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 The Importance of Search Engines . . . . . . . . . . . . . . . . . . . . . . 2
1.2 A Book About Google? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Objective of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Talking About Search Engines . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Structure of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6 Structure of the Chapters and Markings in the Text . . . . . . . . . . 9
1.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Ways of Searching the Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Searching for a Website vs. Searching for Information
on a Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 What Is a Document? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Where Do People Search? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 Different Pathways to Information on the World Wide Web . . . . 14
2.4.1 Search Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4.2 Vertical Search Engines . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.3 Metasearch Engines . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.4 Web Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.5 Social Bookmarking Sites . . . . . . . . . . . . . . . . . . . . . . 21
2.4.6 Question-Answering Sites . . . . . . . . . . . . . . . . . . . . . . 21
2.4.7 Social Networking Sites . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3 How Search Engines Capture and Process Content from
the Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1 The World Wide Web and How Search Engines Acquire
Its Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2 Content Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3 Web Crawling: Finding Documents on the Web . . . . . . . . . . . . 33
3.3.1 Guiding and Excluding Search Engines . . . . . . . . . . . . 37
3.3.2 Content Exclusion by Search Engine Providers . . . . . . 39

vii
viii Contents

3.3.3 Building the Database and Crawling for Vertical

Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.4 The Indexer: Preprocessing Documents for Searching . . . . . . . . 42
3.4.1 Indexing Images, Audio, and Video Files . . . . . . . . . . . 47
3.4.2 The Representation of Web Documents in Search
Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.5 The Searcher: Understanding Queries . . . . . . . . . . . . . . . . . . . . 51
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4 User Interaction with Search Engines . . . . . . . . . . . . . . . . . . . . . . . 59
4.1 The Search Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 Collecting Usage Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.3 Query Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.4 Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.5 Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.5.1 Entering Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.5.2 Autocomplete Suggestions . . . . . . . . . . . . . . . . . . . . . 68
4.5.3 Query Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.5.4 Query Length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.5.5 Distribution of Queries by Frequency . . . . . . . . . . . . . 73
4.5.6 Query Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.5.7 Using Operators and Commands for Specific
Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.6 Search Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5 Ranking Search Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.1 Groups of Ranking Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.2 Text Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.2.1 Identifying Potentially Relevant Documents . . . . . . . . . 86
5.2.2 Calculating Frequencies . . . . . . . . . . . . . . . . . . . . . . . 88
5.2.3 Considering the Structural Elements of Documents . . . . 89
5.3 Popularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.3.1 Link-Based Rankings . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.3.2 Usage Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4 Freshness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.5 Locality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.6 Personalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.7 Technical Ranking Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.8 Ranking and Spam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Contents ix

6 Vertical Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

6.1 Vertical Search Engines as the Basis of Universal Search . . . . . . 121
6.2 Types of Vertical Search Engines . . . . . . . . . . . . . . . . . . . . . . . 123
6.3 Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.3.1 News . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.3.2 Scholarly Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.3.3 Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.3.4 Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.4 Integrating Vertical Search Engines into Universal Search . . . . . 133
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7 Search Result Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.1 The Influence of Device Types and Screen Resolutions . . . . . . . 138
7.2 The Structure of Search Engine Result Pages . . . . . . . . . . . . . . 139
7.3 Elements on Search Engine Result Pages . . . . . . . . . . . . . . . . . 146
7.3.1 Organic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
7.3.2 Advertising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.3.3 Universal Search Results . . . . . . . . . . . . . . . . . . . . . . 148
7.3.4 Knowledge Graph Results . . . . . . . . . . . . . . . . . . . . . 150
7.3.5 Direct Answers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.3.6 Integration of Transactions . . . . . . . . . . . . . . . . . . . . . 152
7.3.7 Navigation Elements . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.3.8 Support for Query Modification . . . . . . . . . . . . . . . . . . 154
7.3.9 Search Options on the Result Page . . . . . . . . . . . . . . . 155
7.4 The Structure of Snippets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7.5 Options Related to Single Results . . . . . . . . . . . . . . . . . . . . . . 159
7.6 Selection of Suitable Results . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
8 The Search Engine Market . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8.1 Search Engines’ Business Model . . . . . . . . . . . . . . . . . . . . . . . 165
8.2 The Importance of Search Engines for Online Advertising . . . . . 166
8.3 Search Engine Market Shares . . . . . . . . . . . . . . . . . . . . . . . . . . 167
8.4 Important Search Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
8.5 Partnerships in the Search Engine Market . . . . . . . . . . . . . . . . . 170
8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
9 Search Engine Optimization (SEO) . . . . . . . . . . . . . . . . . . . . . . . . . 175
9.1 The Importance of Search Engine Optimization . . . . . . . . . . . . . 176
9.2 Fundamentals of Search Engine Optimization . . . . . . . . . . . . . . 178
9.2.1 Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
9.2.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9.2.3 HTML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
x Contents

9.2.4 Trust . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

9.2.5 Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9.2.6 User-Related Factors . . . . . . . . . . . . . . . . . . . . . . . . . 183
9.2.7 Toxins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9.2.8 Vertical Search Engines . . . . . . . . . . . . . . . . . . . . . . . 184
9.3 Search Engine Optimization and Spam . . . . . . . . . . . . . . . . . . . 185
9.4 The Role of Ranking Updates . . . . . . . . . . . . . . . . . . . . . . . . . 185
9.5 Search Engine Optimization for Special Collections . . . . . . . . . 186
9.6 The Position of Search Engine Providers . . . . . . . . . . . . . . . . . 187
9.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
10 Search Engine Advertising (SEA) . . . . . . . . . . . . . . . . . . . . . . . . . . 191
10.1 Specifics of Search Engine Advertising . . . . . . . . . . . . . . . . . . 194
10.2 Functionality and Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
10.3 Distinguishing Between Ads and Organic Results . . . . . . . . . . . 197
10.4 Advertising in Universal Search Results . . . . . . . . . . . . . . . . . . 198
10.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
11 Alternatives to Google . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
11.1 Overlap Between Results from Different Search Engines . . . . . . 204
11.2 Why Should One Use a Search Engine Other Than Google? . . . 204
11.2.1 Obtaining a “Second Opinion” . . . . . . . . . . . . . . . . . . 205
11.2.2 More or Additional Results . . . . . . . . . . . . . . . . . . . . . 205
11.2.3 Different Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
11.2.4 Better Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
11.2.5 Different Result Presentation . . . . . . . . . . . . . . . . . . . . 207
11.2.6 Different User Guidance . . . . . . . . . . . . . . . . . . . . . . . 207
11.2.7 Avoiding the Creation of User Profiles . . . . . . . . . . . . . 208
11.2.8 Alternative Search Options . . . . . . . . . . . . . . . . . . . . . 208
11.3 When Should One Use a Search Engine Other Than Google? . . . 208
11.4 Particularities of Google due to Its Market Dominance . . . . . . . 210
11.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
12 Search Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
12.1 Source Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
12.2 Selecting the Right Keywords . . . . . . . . . . . . . . . . . . . . . . . . . 218
12.3 Boolean Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12.4 Connecting Queries with Boolean Operators . . . . . . . . . . . . . . . 222
12.5 Advanced Search Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
12.6 Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
12.7 Complex Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
12.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Contents xi

13 Search Result Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

13.1 Criteria for Evaluating Texts on the Web . . . . . . . . . . . . . . . . . 231
13.2 Human vs. Machine Inspection of Quality . . . . . . . . . . . . . . . . 232
13.3 Scientific Evaluation of Search Result Quality . . . . . . . . . . . . . . 235
13.3.1 Standard Test Design of Retrieval Effectiveness
Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
13.3.2 Measuring Retrieval Effectiveness Using
Click-Through Data . . . . . . . . . . . . . . . . . . . . . . . . . . 240
13.3.3 Evaluation in Interactive Information Retrieval . . . . . . . 241
13.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
14 The Deep Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
14.1 The Content of the Deep Web . . . . . . . . . . . . . . . . . . . . . . . . . 249
14.2 Sources vs. Content from Sources, Accessibility of Content
via the Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
14.3 The Size of the Deep Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
14.4 Areas of the Deep Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
14.5 Social Media as Deep Web Content . . . . . . . . . . . . . . . . . . . . . 256
14.6 What Role Does the Deep Web Play Today? . . . . . . . . . . . . . . 258
14.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
15 Search Engines Between Bias and Neutrality . . . . . . . . . . . . . . . . . 261
15.1 The Interests of Search Engine Providers . . . . . . . . . . . . . . . . . 262
15.2 Search Engine Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
15.3 The Effect of Search Engine Bias on Search Results . . . . . . . . . 265
15.4 Interest-Driven Presentation of Search Results . . . . . . . . . . . . . 267
15.5 What Would “Search Neutrality” Mean? . . . . . . . . . . . . . . . . . . 270
15.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
16 The Future of Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
16.1 Search as a Basic Technology . . . . . . . . . . . . . . . . . . . . . . . . . 276
16.2 Changes in Queries and Documents . . . . . . . . . . . . . . . . . . . . . 277
16.3 Better Understanding of Documents and Queries . . . . . . . . . . . . 278
16.4 The Economic Future of Search Engines . . . . . . . . . . . . . . . . . 278
16.5 The Societal Future of Search Engines . . . . . . . . . . . . . . . . . . . 279
16.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
Introduction
1

This book is about better understanding the search tools we use daily. Only when we
have a basic understanding of how search engines are constructed and how they
work can we use them effectively in our research.
However, not only the use of existing search engines is relevant here but also
what we can learn from well-known search engines like Google when we want to
build our own search systems. The starting point is that Web search engines are
currently the leading systems in terms of technology, setting the standards in terms
of both the search process and user behavior. Therefore, if we want to build our own
search systems, we must comply with the habits shaped by Web search engines,
whether we like it or not.
This book is an attempt to deal with the subject of search engines comprehen-
sively in the sense of looking at it from different angles:

1. Technology: First of all, search engines are technical systems. This involves the
gathering of the Web’s content as well as ranking and presenting the search
results.
2. Use: Search engines are not only shaped by their developers but also by their
users. Since the data generated during use is incorporated into the ranking of the
search results and the design of the user interface, usage significantly influences
how search engines are designed.
3. Web-based research: Although, in most cases, search engines are used in a
relatively simple way – and often not much more is needed for a successful
search – search engines are also tools for professional information research. The
fact that search engines are easy to use for everyone does not mean that every
search task can be easily solved using them.
4. Economy: Search engines are of great importance for content producers who
want to get their content on the market. Because they are central nodes in the
Web, they also play an important economic role. Here, the main focus is on search
engine visibility, which can be achieved through various online marketing
measures (such as search engine optimization and placing advertisements).

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 1

D. Lewandowski, Understanding Search Engines,
https://doi.org/10.1007/978-3-031-22789-9_1
2 1 Introduction

5. Society: Since search engines are the preferred means of searching for informa-
tion and are used massively every day, they also have an enormous significance
for knowledge acquisition in society. Among other things, this raises the question
of whether search results are credible and whether search engines play a role in
spreading misinformation and disinformation, often treated under the label of
“fake news.”

My fundamental thesis is that one is impossible without the other: we cannot

understand search engines as technical systems if we do not know their social
significance. Nor can we understand their social impact if we do not know the
underlying technology. Of course, one does not have to have the same detailed
knowledge in all areas; but one should achieve a solid basis.
Of course, an introductory book cannot cover the topics mentioned comprehen-
sively. My aim is instead to introduce the concepts central to discussing search
engines and provide the basic knowledge that makes a well-founded discussion of
search engines possible in the first place.

1.1 The Importance of Search Engines

In this book, I argue that search engines have an enormous social significance. This
can be explained, on the one hand, by their mass use and, on the other hand, by the
ranking and presentation of search results.
Search engines (like other services on the Internet) are used en masse. Their
importance lies in the fact that we use them to search for information actively. Every
time we enter a query, we reveal our interests. With every search engine result page
(SERP) that a search engine returns to us, there is a (technically mediated) interpre-
tation of both the query and the potentially relevant results. By performing these
interpretations in a particular way, a search engine conveys a specific impression of
the world of information found on the Web.
For every query, there is a result page that displays the results in a specific order.
Although, in theory, we can select from all these results, we rely heavily on the order
given by the search engine. De facto, we do not select from the possibly millions of
results found but only from the few displayed first.
If we consider this, societal questions arise, such as how diverse the search engine
market is: Is it okay to use only one search engine and have only one of many
possible views of the information universe for each query?
The importance of search engines has already been put into punchy titles such as
“Search Engine Society” (Halavais, 2018), “Society of the Query” (the title of a
conference series and a book; König & Rasch, 2014), and “The Googlization of
Everything” (Vaidhyanathan, 2011). Perhaps it is not necessary to go so far as to
proclaim Google, search engines, or queries as the determining factor of our society;
however, the enormous importance of search engines for our knowledge acquisition
can no longer be denied.
1.1 The Importance of Search Engines 3

If we look at the hard numbers, we see that search engines are the most popular
service on the Internet. We regard the Internet as a collection of protocols and
services, including e-mail, chat, and the File Transfer Protocol (FTP). It may seem
surprising that the use of search engines is at the top of the list when users are asked
about their activities on the Internet. Search engines are even more popular than
writing and reading e-mails. For instance, 76% of all Germans use a search engine at
least once a week, but “only” 65% read or write at least one e-mail during this time.
This data comes from the ARD/ZDF-Onlinestudie (Beisch & Schäfer, 2020), which
surveys the use of the Internet among the German population every year. Compara-
ble studies confirm the high frequency of search engine use: the Eurobarometer
study (European Commission, 2016) shows that 85% of all Internet users in
Germany use a search engine at least once a week; the figure for daily use is still
48%. Germany is below the averages of the EU countries (88% and 57%,
respectively).
Let’s look at the ARD/ZDF-Onlinestudie to see which other Internet services are
used particularly often. We find that, in addition to e-mail and search engines,
messengers (probably WhatsApp in particular) are the most popular. On the other
hand, social media services only reach 36%.
A second way of looking at this is to look at the most popular websites (Alexa.
com, 2021). Google is in the first and third place (google.com and google.de),
followed by YouTube (second place), Amazon, and eBay. It is striking that not
only Google is in the first place but eBay and Amazon are two major e-commerce
companies that not only offer numerous opportunities for browsing but also play a
major role in (product) searches.
The fact that search engines are a mass phenomenon can also be seen in the
number of daily queries. Market research companies estimate the number of queries
sent to Google alone at around 3.3 trillion in 2016 (Internet Live Stats & Statistic
Brain Research Institute, 2017) – that’s more than a million queries per second!
An additional level of consideration arises when we look at how users access
information on the World Wide Web. While there are theoretically many access
points to information on the Web, search engines are the most prevalent. On the one
hand, Web pages can, of course, be accessed directly by typing the address (Uniform
Resource Locator; URL) into the browser bar. Then there are other services, such as
social media services, which also lead us to websites. But none of these services has
achieved a level of importance comparable to that of search engines for accessing
information on the Web, nor is this situation likely to change in the foreseeable
future.
Last but not least, search engines are also significant because of the online
advertising market. The sale of ads in search engines (ads in response to a query)
accounts for 40% of the market (Zenith, 2021); in Germany alone, search engine
advertising generated sales of 4.1 billion euros in 2019 (Statista, 2021).
This form of advertising is particularly attractive because, with each search query,
users reveal what they want to find and thus also whether and what they might want
to buy. This makes it easy for advertisers to decide when they want to offer their
product to a user. Scatter losses, i.e., the proportion of users who see an
4 1 Introduction

advertisement but have no interest in it at that moment, can be significantly reduced

or even avoided altogether in this way.
Search engine providers, like other companies, have to earn money. So far, the
only model search engines have used to make money is the insertion of advertising
in the form of text ads around the search results. Other revenue models have not
caught on. This also means that search engine providers do not, as is often claimed,
align their search engines solely with the demands and needs of users but also with
their own profit intentions and those of their advertising customers.
For companies, however, the importance of search engines is not only a result of
being able to use search engines as an advertising platform but also because of being
found by users in the organic search results. The procedures that serve to increase the
probability of being found are subsumed under the title of search engine
optimization.
Already at this point, we see that if we consider search engines not only as
technical systems but also as socially relevant, we are dealing with at least four
stakeholder groups or actor groups (see Röhle, 2010, p. 14):

1. Search engine providers: On the one hand, search engine providers are interested
in satisfying their users. This involves both the quality of the search results and
the user experience. On the other hand, search engine providers’ second major
(or even more significant?) interest is to offer their advertisers an attractive
environment and earn as much money as possible from advertising.
2. Users: The users’ interest is to obtain satisfactory search results with little effort
and not to be disturbed too much in their search process, for example, by intrusive
advertising.
3. Content producers: Anyone who offers content on the Web also wants to be found
by (potential) users. However, another interest of many content producers is to
earn money with their content. This, in turn, means that it is not necessarily in
their interest to make their content fully available to search engines.
4. Search engine optimizers: Search engine optimizers work on behalf of content
producers to ensure that their offerings can be found on the Web, primarily in
search engines. Their knowledge of the search engines’ ranking procedures and
their exploitation of these procedures to place “their” websites influence the
search engine providers, who attempt to protect themselves against manipulation.

This brief explanation of the stakeholders already shows that this interplay can
lead to conflicts. Search engine providers have to balance the interests of their users
and their advertisers; search engine optimizers have to ensure the maximum visibil-
ity of their clients’ offerings but must not exploit their knowledge of how search
engines work to such an extent that they are penalized by search engine providers for
manipulation.
Clearly, we are dealing with complex interactions in the search engine market.
Only if we look at search engines from different perspectives are we able to classify
these interactions and understand why search engines are designed the way they are.
1.2 A Book About Google? 5

Search engines have to meet the needs of different user groups; it is not enough for
them to restrict their services to one of these groups.
When we talk about search engines and their importance for information access,
we usually only consider the content initially produced for the Web. However,
search engines have been trying to include content from the “real,” i.e., the physical
world, in their search systems for years. Vaidhyanathan (2011) distinguishes three
types of content that search engines like Google capture:

1. Scan and link: External content is captured, aggregated, and made available for
search (e.g., Web search).
2. Host and serve: Users’ content is collected and hosted on their own platform (e.g.,
YouTube).
3. Scan and serve: Things from the real world are transferred into the digital world
by the search engine provider (e.g., Google Books, Google Street View).

Vaidhyanathan (2011) summarizes this under “The Googlization of Everything”

(which is also the title of his book) and thus illustrates not only that search engine
content goes beyond the content of the Web (even if this continues to form the basis)
but also that we are still at the beginning when it comes to the development of search
engines: So far, only a small part of all the information that is of interest to search
engines has been digitized and thus made available for search.
Furthermore, there is a second, largely taken-for-granted assumption, namely,
that a search process must necessarily contain a query entered by the user. However,
we see that search engines can increasingly generate queries by themselves by
observing the behavior of a user and then offering information that is very likely
to be useful to them. For example, suppose a user is walking through a city with their
smartphone in their pocket. In that case, it is easy to predict their desire for a meal
option at lunchtime and suggest a restaurant based on that user’s known past
preferences and current location. To do this, a query (made up of the above
information) is required, but the user does not have to enter it themselves. We will
return to this in Chap. 4.

1.2 A Book About Google?

When we think of search engines, we primarily think of Google. We all use this
search engine almost daily, usually for all kinds of search purposes. Here, again, the
figures speak for themselves: in Germany, well over 90% of all queries to general
search engines are directed to Google, while other search engines play only a minor
role (Statcounter, 2021).
Therefore, this book is based on everyday experience with Google and tries to
explain the structure and use of search engines using this well-known example.
Nevertheless, this book aims to go further: to show which alternatives to Google
there are and when it is worthwhile to use them. But this book will not describe all
possible search engines; it is rather about introducing other search engines, utilizing
6 1 Introduction

examples, and thus, first of all, getting the reader to think about whether Google is
the best search engine for precisely their research before carrying out more complex
searches.
To a certain extent, it can also be said that if you know one search engine, you will
be better able to deal with all the others. We will learn about the basic structure of
search engines and their most important functions by looking at Google, which we
all already know, at least from the user side. The acquired knowledge can then easily
be transferred to other search engines.
Most of the search examples and screenshots also come from Google. In most
cases, however, the examples can be transferred to other search engines. Where this
is not the case, this is indicated.
Regarding the similarity between the different search engines, we can generally
say that Google’s competitors are in a dilemma: Even if they offer innovative
functions and try to do things differently from Google, they are fundamentally
oriented toward Google’s idea of how a search engine should look and work. This
orientation toward Google cannot be blamed on the other search engine providers
because, on the one hand, they can only win over users if those who are used to
Google find their way around immediately; on the other hand, they have to distin-
guish themselves from Google to be able to offer any added value compared to this
search engine.

1.3 Objective of This Book

By its very nature, this book is restricted in its function as an introductory book and
is intended as a general overview. This also means that many topics cannot be dealt
with in detail, but we must remain “on the surface” instead. However, this does not
mean that the contents must be superficial. On the contrary, I have tried to present the
contents as simply as possible but without sacrificing the necessary accuracy. Some
topics are explored via a specific example (such as a vertical search engine), which is
explained in more detail, so that this information can then be applied to other topics.
This book is about transfer: what you learn from one or a few search engines
should be transferable to others. Therefore, it does not matter that some of the
contents in this book – especially when it comes to details of a particular search
engine – may have already changed by the time this book is published. This is
unavoidable, especially in rapidly evolving fields, but the goal is to convey basic
knowledge about search engines that can then be applied to all search engines.
This book is not a substitute for introductory works on, for example, information
retrieval or searching the Web, even though topics from these areas are covered. The
relevant introductory literature on the respective topics is mentioned in the respective
chapters. This book aims to provide an overview and a consideration of different
perspectives on search engines, not an all-encompassing presentation of the individ-
ual topics.
Students in particular often fear that they will only be able to understand search
engines if they delve into algorithms and technical details. In this book, the essential
1.5 Structure of This Book 7

procedures are described in a concise and understandable manner, but the main aim
is to understand the ideas underlying the technical processes. This will enable us to
assess why search engines work as well or as poorly as they do at present and what
prospects there are for their further development.
It is only natural that in any attempt to look at a topic from different perspectives,
one gravitates toward one’s own subject and focuses on the interests of one’s own
discipline. Thus, my interest and the focus of my consideration naturally follow the
subject area and the methods of information science, which always (also) considers
technical information systems from the perspective of humans. In addition, however,
I have made an effort to also consider the perspective of other subjects such as
computer science and media and communication studies (including their literature).

1.4 Talking About Search Engines

To talk about an object, you need a consistent vocabulary. You must know that when
you use certain terms, you are talking about the same thing. In order to avoid talking
past each other, it is therefore necessary to agree on terminology. Since there is
currently no single, unified terminology in the field of search engines, and search
engine optimizers, information scientists, and communication scientists, for exam-
ple, each speak a language of their own, this book is also intended to contribute to
mutual understanding. At the end of the book, there is a glossary that lists and
explains all important terms in alphabetical order. I have made an effort to include
synonyms and related terms so that readers who have already gained some knowl-
edge from the literature can find “their” terms and quickly get used to the terminol-
ogy I have used.

1.5 Structure of This Book

Of course, you can read this book from cover to cover, which was my primary
intention when writing it. However, if you only want to read about a specific topic,
the chapter structure allows you to do so.
Following this introductory chapter, Chap. 2 covers different ways of searching
the Web. Indeed, search engines like Google are not the only form of access to
information on the Web, even if the form of the algorithmic universal search engine
has become widely accepted. The various forms of search systems are briefly
introduced, and their significance is discussed in the context of searching the Web.
Chapter 3 then explains the basic technical structure of algorithmic search
engines. It explains how search engines obtain content from the Web, how this
content is prepared so that it can be searched efficiently, and how users’ queries can
be interpreted and processed automatically.
After these two technical chapters, we consider the user side in Chap. 4: what is
actually searched for in search engines, how are queries formulated, and how do
users select the most suitable results?
8 1 Introduction

Closely related are the ranking procedures, i.e., the arrangement of search results.
Chapter 5 describes the basic procedures and explains their significance. Although it
is often claimed that the ranking of search results is the big secret of every search
engine, knowledge of the most important ranking factors can at least fundamentally
explain the arrangement of search results, even if the concrete ranking depends on a
multitude of weightings that cannot be traced in detail. This understanding, in turn,
can help us both in our searches and in preparing our own content for search engines
or even in creating our own information systems.
Chapter 6 then shows how search results from the general Web Index are
extended by adding so-called vertical collections such as news, images, or videos.
For this purpose, the well-known search engines have built and integrated numerous
vertical search engines whose results are displayed on the search engine result pages.
Chapter 7 is devoted to the presentation of search results. For some years now, the
well-known search engines have deviated from the usual list form of search result
presentation and have instead established new forms of compiling search results with
concepts such as universal search and knowledge graph. This has made the result
pages more attractive and increased the choices available on these pages. With this
type of result presentation, the search engines also deliberately guide the users’
attention.
This brings us to the economic realities related to search engines. In Chap. 8, we
deal with the search engine market and thus, among other things, with the question
of how Google has succeeded in almost completely dominating the search engine
market (at least in Europe). Of course, the question of whether such a situation is
desirable and how it could be changed is also raised here.
Chapter 9 is devoted to the side of those who want to make their content best
available via search engines and their helpers, the search engine optimizers. They use
their knowledge of search engines’ indexing and ranking procedures to make content
easier to find and to bring traffic to their customers. Their techniques range from
simple text modification to complex procedures that consider the Web’s linking
structure.
Chapter 10 provides a detailed description of the advertising displayed in search
engines. On the one hand, it deals with the ads shown on search result pages as a type
of search result, and on the other hand, with the question to what extent users can
distinguish these ads from the organic search results.
Chapter 11 deals with alternatives to Google. First, it is important to answer the
question of what makes a search engine an alternative search engine. Is it enough that
it is simply a search engine other than Google? Then, based on fundamental
considerations and concrete situations in the search process, we will explain in
which cases it is worth switching to another search engine.
In Chap. 12, we change the perspective again and consider search engines as tools
for advanced Internet research. In the chapter on user behavior, it became clear that
most users put little effort into formulating their queries and sifting through the
results. Therefore, we want to see what strategies and commands we can use to get
the most out of search engines.
1.7 Summary 9

Another topic related to searching, but also to the general evaluation of search
engines, is the question of the quality of the search results, which we will address in
Chap. 13. The quality of search results can be viewed from two perspectives: One is
about the user’s result evaluation in the course of their search; the other is about
scientific comparisons of the result quality of different search engines.
Chapter 14 deals with the contents of the Web that are not accessible to general
search engines, the so-called Deep Web. An enormous treasure trove of information
cannot be found with Google and similar search engines or at least to a limited
extent. We will see why this content remains hidden from the search engines and
with what methods we can nevertheless access it.
While the previous topics dealt with aspects from the areas of technology, use,
and Web-based research, Chap. 15 deals with the societal role of search engines.
What role do search engines play in knowledge acquisition, and what role should
they play?
Finally, Chap. 16 focuses on the future of search. Of course, a book like this can
only ever offer a snapshot, and 10 years ago, it would have provided a different
picture than today. However, the “problem” of search has by no means been solved
(and may never be solved), so it is worth looking at today’s search engines not only
in their evolution toward the current state but also to venture a look into the (near)
future.

1.6 Structure of the Chapters and Markings in the Text

By their very nature, chapters on different topics must be structured differently.

Nevertheless, the chapters in this book have certain similarities: At the beginning of
each chapter, there is an introduction that defines the topic and briefly describes its
significance for the book. Detailed explanations follow this. At the end of each
chapter, there is a summary that reviews the most important points. Each chapter also
contains a bibliography and, in a separate box, a list of recommendations for further
reading for those who wish to delve deeper into the topic.
There are also examples in boxes throughout the text that illustrate and deepen
what is said in the main text but are not essential for understanding the main text.

1.7 Summary

Search engines are essential tools for accessing the information on the World Wide
Web. In this book, we will look at them from the perspectives of technology, usage,
economic aspects, searching the Web, and society.
The significance of search engines results from their mass use and the fact that
they are by far the preferred means of accessing information on the World Wide
Web. However, one search engine, in particular, Google, is used for most searches.
We should not only consider search engines as technical systems. Due to the
interactions of different stakeholders (search engine providers, users, content
10 1 Introduction

producers, and search engine optimizers), there are numerous factors which influ-
ence search results and which the search engine providers do not exclusively direct.
In terms of content, search engines no longer only capture the Web content but
also offer platforms on which users can create content themselves, which is then
made searchable. Furthermore, search engine providers offer various vertical search
engines in addition to Web search, whose results are included in the general search
engine result pages. Finally, content from the physical world is transferred to the
digital world and integrated into search.

References
Alexa.com. (2021). Top sites in Germany. https://www.alexa.com/topsites/countries/DE.
Beisch, N., & Schäfer, C. (2020). Ergebnisse der ARD/ZDF-Onlinestudie 2020: Internetnutzung
mit großer Dynamik: Medien, Kommunikatio, Social Median. Media Perspektiven, 51(9),
462–481.
European Commission. (2016). Special Eurobarometer 447 – Online Platforms. European Com-
mission. https://doi.org/10.2759/937517
Halavais, A. (2018). Search engine society. Polity.
Internet Live Stats, & Statistic Brain Research Institute. (2017). Anzahl der Suchanfragen bei
Google weltweit in den Jahren 2000 bis 2016 (in Milliarden). In Statista – Das Statistik-
Portal. https://de.statista.com/statistik/daten/studie/71769/umfrage/anzahl-der-google-
suchanfragen-pro-jahr/.
König, R., & Rasch, M. (Eds.). (2014). Society of the query reader: Reflections on web search.
Institute of Network Cultures.
Röhle, T. (2010). Der Google-Komplex: Über Macht im Zeitalter des Internets. Transcript. https://
doi.org/10.14361/transcript.9783839414781
Statcounter. (2021). Search engine market share. https://gs.statcounter.com/search-engine-market-
share.
Statista. (2021). Prognose der Umsätze mit Suchmaschinenwerbung in Deutschland in den Jahren
2017 bis 2025 (in Millionen Euro). In Statista – Das Statistik-Portal. https://de.statista.com/
prognosen/456188/umsaetze-mit-suchmaschinenwerbung-in-deutschland.
Vaidhyanathan, S. (2011). The Googlization of everything (and why we should worry). University
of California Press. https://doi.org/10.1525/9780520948693
Zenith. (2021). Prognose zu den Investitionen in Internetwerbung weltweit in den Jahren 2018 bis
2022 nach Segmenten (in Milliarden US-Dollar). In Statista – Das Statistik-Portal. https://de.
statista.com/statistik/daten/studie/209291/umfrage/investitionen-in-internetwerbung-weltweit-
nach-segmenten/.
Ways of Searching the Web
2

At first glance, searching the Web may seem trivial: we enter a query and receive a
search engine result page (SERP) on which we select a result. But this is only one of
the many ways to access information on the Web. In this chapter, we introduce the
different ways of accessing the information on the Web and explain why access via
search engines has become dominant.

2.1 Searching for a Website vs. Searching for Information

on a Topic

First, we need to ask what we want or can achieve with a search. For now, it is
sufficient to distinguish three cases. We will explain these cases with the help of the
Firefox browser starting page shown in Fig. 2.1:

1. A user wants to go to a specific website they already know. To do this, they enter
the URL into their browser’s address bar (Fig. 2.1, top line). Then, on the website,
they either read something directly, conduct a search, or click on further
documents. This process has little to do with our intuitive understanding of
searching, but it is a means of getting to the information we are looking for.
For example, a user interested in news on a current topic can go directly to a news
website and either read relevant articles directly on the front page, click on them
there, or search for articles using the internal search function of the news website.
2. A user wants to go to a specific website they either already know or do not yet
know about and searches for it via the address bar (combined URL and search
bar) (Fig. 2.1, top line) or the search field (Fig. 2.1, field placed in the middle).
The search is carried out in the previously set search engine (for settings, see
Chap. 8). Such a search may be for a known website – in which case searching is
merely a “shortcut” to direct entry in the address bar (e.g., entering “ny times” in
the search field instead of “www.nytimes.com” in the address bar) or it may be to
help if the user can no longer remember the exact address of the website they are

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 11

D. Lewandowski, Understanding Search Engines,
https://doi.org/10.1007/978-3-031-22789-9_2
12 2 Ways of Searching the Web

Fig. 2.1 Start page of the Firefox browser with the address bar and Google search as the default
start page (August 26, 2022)

looking for (e.g., if they no longer know whether a website ends with .com or .
org). When searching directly for a website that is not yet known, the user at least
assumes that such a website exists and searches accordingly.
3. The third type is a user looking for information not yet known to them. This type
differs fundamentally from the previous two as the user is not looking for a
specific website but for information on a topic. Here, it is impossible to predict
with certainty whether this information can be found on a particular website or
whether the information from a single website is sufficient to satisfy the
information need.

We will discuss the subdivision of search queries according to intentions or

information needs in more detail in Sect. 4.3. For the time being, it is sufficient to
distinguish between a search for known websites and a search for unknown infor-
mation. To be able to assess different ways of accessing information on the Web, it is
essential that we can already distinguish between these cases.
In our example, we have already seen that search queries can be entered in
different places. We will return to the significance of the search engine preset in a
browser’s search box or address bar and the preset start page in the chapter on the
search engine market (Chap. 8).

Is Searching the Web Like Looking for a Needle in a Haystack?

Searching the Web is often compared to looking for a needle in a haystack.
This picture is meant to illustrate that it is difficult to find the right thing (the
needle) because of the vast amount of information available (the haystack).

(continued)
2.3 Where Do People Search? 13

But this image is skewed. In the case of the haystack, we know what the
needle looks like, and there is only one needle, so we can tell when our search
is finished.
However, in the case of searching for previously unknown information, we
do not always have such a clearly defined idea of what we want to find. There
could be several needles that might also serve our purpose differently. And it
could also be that we are only satisfied when we have found several needles
that complement or confirm each other.

2.2 What Is a Document?

By its very design, the Web is multimedial and contains much more than just text. In
this respect, search engines are not only there to find text on the Web but also other
types of information – even if search today is (still) primarily text-based. But
regardless of whether it is a text, an image, or a video, we will speak of a document
or, alternatively, an information object.
So what is a document? When we think of documents, we might first think of
official documents, such as those issued by public authorities with a stamp and
signature. In information science, however, the term is defined much more broadly: a
document is a record of information, regardless of whether it is in written form (text
document) or, for example, in pictorial form (image document). Concerning search
engines, this means that every piece of content they display (text, images, videos,
etc.) is a document. Therefore, we sometimes speak of information objects instead to
make it clear that we are not only talking about textual documents.

2.3 Where Do People Search?

The times when search engines were used mainly in the same context, namely, on
desktop computers or laptops, are long gone. People now search on a wide range of
devices, ranging from smartphones and tablets to wearables and purely voice-
activated devices. Especially in the case of tablets and smartphones, we must
distinguish between Web search and search within specific applications: For
searches within apps, only a limited amount of data has to be searched, whereas
Web search is about “the whole thing,” i.e., the most complete representation of the
Web possible. No matter which device we use for searching: (Web) search is a
central part of our Internet use. However, as we will see, user behavior differs
depending on the context (e.g., mobile vs. at home) and device (e.g., large screen
on a laptop vs. small screen on a smartphone). Search engines are adapted to deliver
adjusted results and result displays on different devices and in different contexts (see
Sect. 7.1 for more details).
14 2 Ways of Searching the Web

2.4 Different Pathways to Information on the World Wide Web

Search engines are by no means the only way to access the information on the Web.
In the following, we will present the different types of access and related systems.
We will then put them into relation to search engines, which will again be the
exclusive focus of the subsequent chapters. We will start with Web search engines
themselves, as they are our starting point, and we will then compare the advantages
and disadvantages of the other systems with them.
In general, a distinction can be made between search engines and other systems:

2022 Search engines include general search engines, vertical search engines, hybrid
search engines, and metasearch engines.
2022 Other systems include Web directories, social bookmarking sites, question
answering sites, and social networks.

To understand the idea of search engines, it is essential to realize that the different
ways of accessing Web content also have different objectives. For example, it would
be unfair to compare the scope of the databases of search engines and Web
directories, as they have very different requirements regarding the comprehensive-
ness of their databases. It is also relevant whether a system aims to support ad hoc
searches (i.e., searches based on the input of a search query) or whether the system is
to support browsing of content or monitoring specific sources. For example, the
latter is the case with social networks, where users “follow” people or accounts by
subscribing to their new messages. This means that content from these accounts is
displayed regularly without having to repeatedly conduct a new search.

2.4.1 Search Engines

When we speak of search engines, we usually mean Web search engines (also
known as general, universal, or algorithmic search engines). These engines claim
to cover the content of the Web as completely as possible and, if necessary, to enrich
it with additional content (see Sect. 3.2). Figure 2.2 schematically shows which
contents of the Web search engines cover. The cloud represents the universe of the
Web. It contains a multitude of documents stored within websites (illustrated by the
hierarchical structure of documents). The content that is captured by the search
engine is highlighted. Although search engines aim to capture the total content of the
Web, this objective is not achieved and cannot be reached either. We will look at the
reasons for this in more detail in Chap. 3.
Nevertheless, search engines achieve greater coverage of the Web than any other
type of search system. This is due, on the one hand, to their universal claim and, on
the other hand, to the fact that they capture the content automatically. This process is
described in detail in Chap. 3; at this point, it should suffice to say that search engines
can capture a huge number of documents on the Web and make them searchable.
2.4 Different Pathways to Information on the World Wide Web 15

Fig. 2.2 The contents of

search engines SEARCH ENGINES
As many documents as possible from the web

= =
= = =
=
=
= = = = =
= = =
= = = =
= = = = = =
= = =
=
= = =
= = = =
= =
= = =
=

Fig. 2.3 Start page of the AltaVista search engine (1996); https://web.archive.org/web/1
9961023234631/http://altavista.digital.com/

The Concept of the Algorithmic Search Engine in the 1990s

The idea of the search engine as we know it already evolved in the early days
of the Web. Early search engines such as Lycos and WebCrawler already
worked on the same principle as Google and other search engines do today:
They gather the pages available on the Web by following links and return
ranked lists of results in response to search queries. This process is fully
automatic.

(continued)
16 2 Ways of Searching the Web

Perhaps the best way to illustrate the similarity between earlier and today’s
search engines is to look at the homepage of AltaVista, the leading search
engine at the time, in 1996 (Fig. 2.3).
Firstly, the similarity with today’s search engines like Google is striking:
There is a centrally placed search field, next to which is a button that can be
used to submit the search. In principle, users can enter whatever they want
without having to learn a specific query language. Whether single words,
whole sentences, or questions: it is up to the automatic processing of the
search engine to deliver results that match the search queries.
Secondly, the AltaVista homepage contains information about the size of
its database. It states 30 million documents – a large number at the time,
considering that the Web was still in its infancy. In the meantime, the Web has
grown many times over, but the challenge of capturing its content in a
complete and up-to-date manner and making it available for search has
remained (see Chap. 3).
Thirdly, it should be pointed out that AltaVista made its search results
available via other portals, including Yahoo, as early as 1996. Even then,
many providers did not build their own search engines but used the results of
one of the big search engines in cooperation. We will return to such coopera-
tion in Chap. 8 and see its influence on the current search engine market.
However, the differences between then and now should not be concealed.
Already above the search box, some things are different from today’s search
engines: With AltaVista, you could choose between different search modes
directly on the start page, in this case, between the preset simple search with
only one search field and an advanced search. Already in the simple search,
you could choose the collection to be searched (default: Web) and the format
in which the search results would be displayed. In later chapters, we will get to
know the advanced search and different result presentations.
The texts on the AltaVista home page around the search box are also
illuminating. On the one hand, a link to a mirror site is offered; such “mirrors”
are nothing more than copies of websites available in another geographical
location, in this case, Australia. Internet connections in 1996 were much less
developed than they are today, and one often had to wait quite a long time for
responses from remote Web servers. Mirrors were created to shorten these
waiting times. Today, search engines have data centers spread around the
world that distribute the search engine’s database and the processing of search
queries. However, users no longer have to select one of these data centers
explicitly, but both the index and the processing of search queries are
distributed automatically.
2.4 Different Pathways to Information on the World Wide Web 17

2.4.2 Vertical Search Engines

There is a distinction to be made between general search engines and vertical search
engines. Vertical search engines aim to capture as many documents as possible from
selected websites. The term “vertical search engines” is often used instead of
“special search engines”; in this terminology, universal search engines are referred
to as horizontal search engines.
Vertical search engines are restricted to a specific topic and thus make a more
targeted search possible. The ranking can be specially adapted to the documents they
index, as can the subject indexing of the documents. Finally, there are also
advantages in the presentation of the results, which can be adapted to the individual
purpose of the vertical search engines and the proficiency of the target audience. The
fact that vertical search engines cannot be replaced by universal search engines
results from the problems of the latter (for a detailed explanation, see Sect. 6.1):

1. Universal search engines have technical restrictions and (despite the label univer-
sal) cannot cover the entire Web.
2. There are financial hurdles that restrict the collection of content and its indexing.
3. Universal search engines are geared toward the average user.
4. They have to provide consistent indexing of all content so that everything is
searchable together.1

Vertical search engines intentionally restrict themselves to a specific area of the

Web (see Fig. 2.4). Usually, they are limited to certain sources, i.e., websites. These

Fig. 2.4 Contents of vertical

search engines VERTICAL SEARCH ENGINES
As many documents as possible from
selected websites

= =
= = =
=
=
= = = = =
= = =
= = = =
= = = = = =
= = =
=
= = =
= = = =
= =
= = =
=

1
Of course, universal search engines can carry out individual indexing for certain types of content or
certain databases. However, this cannot be done for the mass of offerings to be indexed.
18 2 Ways of Searching the Web

websites are typically selected by hand. For example, if one wants to build a vertical
search engine for news, it makes sense first to compile the relevant news websites,
which the search engine then continuously scans for new content (pages).
A website is a self-contained offering on the Web that can contain several Web
pages. Differentiation is made via the domain (e.g., nytimes.com), subdomains, or
directories (e.g., archive.nytimes.com or nytimes.com/section/world).
On the other hand, a Web page is a single document usually composed of text and
associated media elements (images, videos, etc.).

Examples of Vertical Search Engines

Vertical search engines can be restricted to very different topics. Examples
include Google News (https://news.google.com/), which is restricted to news,
and Swiggle (https://swiggle.org.uk/), which is restricted to content suitable
for children.

Hybrid search engines are a particular type of vertical search engine. Like vertical
search engines, they cover a selected part of the World Wide Web but add additional
content from databases to the resulting inventory. This database content is not part of
the WWW and, therefore, cannot be found through standard search engines (for
technical details, see Chap. 14). Figure 2.5 illustrates the hybrid search engine
model.

HYBRID SEARCH ENGINES

As many documents as possible from
selected websites
+ content from databases

= = Database
= = =
=
=
= = = = =
= = =
= = = =
= = = = = =
= = =
=
= = =
= = = = Database
atabase
= =
= = =
=

Fig. 2.5 Contents of hybrid search engines

2.4 Different Pathways to Information on the World Wide Web 19

2.4.3 Metasearch Engines

At first glance, metasearch engines look like other search engines. They also provide
the user with the same service, namely, potential access to all World Wide Web
content. However, they differ from the “real” search engines in that they do not have
their own index but, as soon as the user enters a query, they retrieve results from
several other, “real” search engines, merge them, and display them in their own
results display (see Fig. 2.6).
The idea behind metasearch engines is that no one search engine can cover the
entire Web. Therefore, combining the results of several search engines that cover
different areas of the Web would be worthwhile. A second advantage is supposed to
lie in a better relevance ranking of the results since the best results are already
fetched by each of the giving search engines, from which a ranking of the best is then
created.
However, there is considerable criticism of the concept of metasearch engines,
which is mainly directed at the fact that the supposed advantages of metasearch are
claimed but not empirically proven (Thomas, 2012). It can also be argued that
metasearch is an outdated idea, as today’s search engines no longer have the
coverage problems that search engines had in the 1990s, when the concept of
metasearch engines was born. At the very least, the benefits of better coverage
only play a role in a few cases today.
Even the supposed advantage of ranking no longer exists today, at least not to the
same extent as it did in the past: For one thing, the universal search engines have
become far better in this respect, and for another, metasearch engines do not have
access to all the documents of the providing search engines (see Fig. 2.6). Rather,

METASEARCH ENGINES
The best results from several search engines

Search engine 1 Search engine 2 Search engine 3

1. 1. 1.
2. 2. 2.
3. 3. 3.
4. 4. 4.
5. 5. 5.
6. 6. 6.
7. 7. 7.
8. 8. 8.
9. 9. 9.
10. 10. 10.

Fig. 2.6 Contents of metasearch engines

20 2 Ways of Searching the Web

they only receive the documents that are already at the top of the list from each of the
search engines. Additionally, they do not receive the documents themselves, but
only the titles that appear in the result list, the short descriptions, and the URLs.
Therefore, it is at least questionable whether a better ranking can be achieved
from this.

2.4.4 Web Directories

Web directories aim to find the best websites in a given category and arrange them in
a hierarchical directory. Thus, their approach differs fundamentally from that of
search engines. While search engines automatically locate and capture content, Web
directories are created by people. People select the websites to be included in the
directory (see Fig. 2.7); they select the appropriate positions in the classification
system and describe them.
There are also differences on the side of searching. Although it is generally
possible to search directories using keywords, their strength lies precisely in their
hierarchical arrangement and, thus, the possibility of navigating from the general to
the specific. Once one finds a suitable category, an overview of selected websites on
the topic is presented. One can then continue browsing there.
In other words, Web directories do not lead to individual documents but to
sources (websites). This can be both an advantage and a disadvantage.
Web directories no longer play a role in accessing the contents of the Web; search
engines have overrun them. Nevertheless, they are discussed here because, on the
one hand, they represent a fundamentally different approach to that of search engines

Fig. 2.7 Contents of Web

directories WEB DIRECTORIES
The best websites, arranged hierarchically

= =
= = =
=
=
= = = = =
= = =
= = = =
= = = = = =
= = =
=
= = =
= = = =
= =
= = =
=
2.4 Different Pathways to Information on the World Wide Web 21

and, on the other hand, Web directories were the second most important approach
alongside search engines, especially in the early days of the Web (see Hamdorf,
2001). For example, Yahoo was founded as a Web directory and only later evolved
into a Web portal, which today naturally includes a search engine.
Both the Yahoo Web directory and the second major directory, the Open Direc-
tory Project (ODP; “DMOZ directory”), have since been discontinued. The end of
the Open Directory Project in 2017 thus also meant the end of the “era of Web
directories,” even if the volunteers of the ODP now continue to run their directory on
their own at https://curlie.org/ and specialized directories still flourish in some
niches.

2.4.5 Social Bookmarking Sites

In social bookmarking sites, too, people decide which documents are included in the
database. Whereas in Web directories, people act in defined roles as voluntary
editors, social bookmarking sites are based on the principle that all users can add
documents to the database by storing them as bookmarks in the system. This differs
from the bookmarks that a user stores in their browser in that a user’s bookmarks are
also accessible to other users and that the bookmarks are indexed using tags. In this
way, the stored pages can also be found under keywords not found on the page itself,
as they are considered relevant only by the users. Another way of using tags for
search is to use frequency metrics to determine which documents are considered
particularly important by users. A set of tags is called a folksonomy (from folk and
taxonomy; see Peters, 2011). The popularity of a particular URL can be determined
by the number of users who have saved it. This, in turn, can be used to rank and
evaluate the search results.
Some major social bookmarking sites (such as del.icio.us) have ceased to exist.
Similar to Web directories, social bookmarking sites are more niche today and have
not been able to establish themselves as a viable alternative to search engines.

2.4.6 Question-Answering Sites

A (community-based) question-answering site allows its users to ask specific

questions that volunteers can then answer. In principle, any (registered) user can
answer questions.
The advantage for the questioner is that – in contrast to the formulation of queries
in general search engines – they can describe their information need in detail and, if
necessary, enter into dialogue with the answerers to arrive at a detailed answer to
their question.
Apart from the issue of the quality of the answers provided, one of the main
problems with this approach is that it constitutes an asynchronous search. This
means that the questioner does not get an answer directly, as with search engines,
but has to wait until another user is ready (and able) to answer the question. One way
22 2 Ways of Searching the Web

out is to search the archive of questions that have already been asked (and answered).
In addition, questions in question-answering sites are also often tagged, which can
improve findability.
Thus, question-answering sites are used in a different way than search engines:
They are particularly suitable when searching with a search engine has not yielded
satisfactory results, or the question is so specific that it can best be answered by a
human being who thinks their way into the searcher’s information problem.
Yahoo Answers was a well-known example of a question-answering site, which
was discontinued in 2021. There, one could find a great variety of questions and
answers sorted by categories.

2.4.7 Social Networking Sites

Social networking sites have gained enormous importance in recent years and are
among the most popular services on the Web, especially for younger users (Beisch &
Schäfer, 2020). For many users, they are – along with search engines – the main
route of access to the Web’s content. In social networking sites, users are mainly
exposed to content via recommendations from contacts (or, more generally, people
or services they follow); targeted searches by entering keywords play only a minor
role. In this respect, they are often complementary to search engines: when it comes
to “discovering” content, social networking sites are used; for targeted searches,
search engines are used.
On social networking sites, content can only be found or discovered if other users
have linked to it or if users themselves have created it. In this respect, social
networking sites simultaneously have more and less content to offer than search
engines: On the one hand, the content generated by users can be found, which is
usually not publicly visible and, therefore, not covered by search engines (see
Chap. 14). But, on the other hand, they do not offer a complete picture of the
Web’s content like the general search engines, as only what other users have
previously created or linked to is shown.

2.5 Summary

Search engines provide access to information on the Web. While general search
engines (Web or universal search engines) dominate in use, numerous types of
search engines have emerged. In addition, there are other systems for accessing
the information on the Web. However, effective integration of the different
approaches has not yet been achieved.
Web search engines claim to capture the entire Web content and make it search-
able. On the other hand, vertical search engines deliberately restrict themselves to
certain areas of the Web that they want to cover more completely. They also have the
advantage of offering a range of results adapted to the respective topic and, possibly,
more detailed search options.
References 23

Hybrid search engines combine content from the Web with content from
databases that general search engines cannot access. They are a subgroup of vertical
search engines.
Metasearch engines do not have their own database but draw on the results of
several other search engines and combine them in a new ranking.
With Web directories, websites are classified and described by humans in a
classification system. While this approach was on an equal footing with search
engines in the early days of the Web, it no longer plays a role today.
Social bookmarking sites allow users to save and share bookmarks. Both one’s
own bookmarks and those of other users can be searched.
Question-answering sites are community-based services that allow asynchronous
searches: the searcher asks a question, which is then answered (with a time delay) by
volunteers. Similar to conventional search engines, however, the archive of
questions already submitted in the past can be searched.
Social networking sites provide access to content created and linked by other
users. Since user-generated content is largely inaccessible to general search engines,
it is an important supplement, especially when it comes to “discovering” content – as
opposed to targeted searches.

Further Reading
Unfortunately, there are no books on the history of search engines. However,
in Battelle’s (2005) book on Google, you can find some information about
their early years. There is also no comprehensive work on the classification of
tools for indexing the Web; here, one may have to resort to finding separate
books.

References
Battelle, J. (2005). The search: How Google and its rivals rewrote the rules of business and
transformed our culture. London: Portfolio. Brealey.
Beisch, N., & Schäfer, C. (2020). Ergebnisse der ARD/ZDF-Onlinestudie 2020: Internetnutzung
mit großer Dynamik: Medien, Kommunikation, Social Media. Media Perspektiven, 9, 462–481.
Hamdorf, K. (2001). Wer katalogisiert das Web?: Dokumentarische Arbeit als Big Business und
Freiwilligen-Projekt. Information Wissenschaft Und Praxis, 52(5), 263–270.
Peters, I. (2011). Folksonomies und Kollaborative Informationsdienste: Eine Alternative zur
Websuche? In D. Lewandowski (Ed.), Handbuch Internet-Suchmaschinen 2: Neue
Entwicklungen in der Web-Suche (pp. 29–53). Akademische Verlagsgesellschaft AKA.
Thomas, P. (2012). To what problem is distributed information retrieval the solution? Journal of the
American Society for Information Science & Technology, 63(7), 1471–1476. https://doi.org/10.
1002/asi.22684
How Search Engines Capture and Process
Content from the Web 3

This chapter describes the technical basis of search engines. This basis includes how
the documents available on the Web are brought into the search engine and how they
are made searchable, as well as how the link between a search query and the
documents in the database is established. Details on the workings of the crawler,
the indexer, and the searcher are given.
While we have already named some characteristics of search engines in the
previous chapter to distinguish them from other tools for accessing the contents of
the Web, we now need a precise definition since we want to deal with search engines
in detail:

A search engine (also: Web search engine; universal search engine) is a computer system
that captures distributed content from the World Wide Web via crawling and makes it
searchable through a user interface, listing the results in a presentation ordered according to
relevance assumed by the system.

To understand this definition in its details, we will explain the individual elements
as follows:

1. Computer system: First of all, the definition specifies that a search engine is a
computer system. The word “system” already indicates that it is typically more
than one computer, namely, a multitude of computers linked together to perform
different functions which jointly form the search engine.
2. Distributed content from the World Wide Web: Here, there is a restriction to
specific content. The World Wide Web is a part of the Internet, and search
engines are limited to this part first (even if the Web content is sometimes
supplemented by other content; see Sect. 3.2). If we were to expect search engines
to search the entire Internet, this would also include all e-mails, for example, since
e-mail is part of the Internet but not of the World Wide Web.
This content is inherently distributed, i.e., there is no central repository where it is
stored. Instead, documents are stored on Web servers that are initially

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 25

D. Lewandowski, Understanding Search Engines,
https://doi.org/10.1007/978-3-031-22789-9_3
26 3 How Search Engines Capture and Process Content from the Web

independent of each other. Thus, it is only through the links between documents
that a network is created which can be navigated by users (moving from docu-
ment to document via the links) as well as by search engines.
3. Crawling: We only refer to search engines when the systems in question capture
the contents of the Web by following links; this process is called crawling. It
starts from known documents; the links contained in these documents are
followed, and thus, new documents are discovered, whose links are followed
once again. Eventually, this process will theoretically – according to our first
assumption, which we will discuss in detail below – create a complete image of
the World Wide Web, which will then finally be made searchable. However, the
focus on crawling does not mean that content cannot also additionally be brought
into the search engine by other methods. We will discuss this in more detail in
Sect. 3.2.
4. User interface: To access the content provided by the search engine, a user
interface is required in which the user can enter their search queries and view
the results.
5. Relevance as assumed by the system: We usually speak of the results being
ordered according to relevance. It is more accurate to use the phrase “according to
relevance as assumed by the system” since there can be no single correct order of
search results. Different documents can be relevant for different users. We will
see that there are multiple models and ideas of relevance. It is often wrongly
assumed that there are right and wrong ways of ordering search results (see
Chap. 5).
6. Presentation ordered according to relevance: This (perhaps unwieldly) term
expresses that the presentation of results by a search engine is not necessarily a
simple list, but that complex forms of result presentation are possible. We will
discuss the presentation of results in search engines in detail in Chap. 7.

What are the advantages and disadvantages of this definition that we use in
this book? First of all, it is relatively open, which means that we will probably be
able to classify future systems (whatever they may look like then) as search engines
according to this definition. However, one limitation of the definition is that it
explicitly refers to the World Wide Web and crawling. More precisely, we would
have to speak of Web search engines instead of search engines, but this does not
correspond to the general usage, which simply refers to search engines.

Is YouTube a Search Engine?

YouTube is accessible on the Web and allows users to search for videos – so
one could assume that it is a search engine. However, if we check the six
criteria from our definition, we see that YouTube is not a search engine in our
sense:

(continued)
3 How Search Engines Capture and Process Content from the Web 27

While YouTube is clearly a computer system, it does not capture

distributed content from the World Wide Web via crawling. Instead, users
upload videos directly to the YouTube platform, even if they are then embed-
ded in HTML documents on the YouTube website, thus allowing other search
engines to crawl them. However, the other criteria from the definition are
fulfilled.
In contrast, Google Video (https://www.google.com/videohp) is a (vertical)
search engine or hybrid search engine: Here, videos from different sources are
retrieved from the Web through crawling and then made accessible. However,
we can assume that Google does not crawl its own content from YouTube but
instead captures it in a structured manner.
Google Video offers a much more extensive selection of videos than
YouTube; many videos that cannot be found on YouTube can be found on
Google Video.

Search engines have several components. Again, there are several variants of the
basic structure, which are, however, substantially similar. We will use the division of
components according to Risvik and Michelsen (2002); see Fig. 3.1. A search engine
is divided into four components: the crawler, the search engine database (local store;
ideally a complete and current copy of the Web), the indexer, and the searcher. The
task these components fulfill is to mediate between the content of the World Wide
Web and individual users. Before we describe the individual components in detail in
the following sections, we will briefly give an overview of these components.
In the figure, the World Wide Web is depicted as a cloud, which illustrates that it
has no clearly defined boundaries and is constantly changing. We are dealing with a
nebulous quantity that we can never quite grasp. Moreover, as we shall see, it is not
possible for search engines to fully map the content of the Web.
However, it must first be emphasized that when we search with a search engine,
we are never searching the contents of the Web itself, but always just a copy of the
World Wide Web prepared by the search engine. It is a characteristic of the quality of
a search engine that this copy should always be as complete and current as possible.
To access the content from the Web in the first place, a crawler follows the links
on each page as described above, moving from page to page. The pages found in this

WWW Crawler Indexer Searcher End users

Local Store
(W copy)

Fig. 3.1 Basic structure of a search engine (Adapted from Risvik and Michelsen, 2002, p. 290)
28 3 How Search Engines Capture and Process Content from the Web

way are stored in memory before they can be accessed. This memory is known as the
“local store” and is the raw material for the index.
The indexer creates the index, processing the documents so that they can be
searched efficiently. This means that instead of simply storing text, images, etc., a
representation of each document is created, and indexes are created to enable these
representations to be found quickly. Each document is represented by a file that
contains all information relevant for its retrieval. In addition, the document repre-
sentation includes further information that is used for ranking (see Sect. 3.4.2 for
more details). Thus, the index can be compared to the index of a book, which refers
readers to the sections of the book where the relevant information can be found.
Finally, the searcher mediates between users and content. Here, documents are
sorted by their assumed relevance and displayed to the user.

Why Do Search Engines Only Search a Copy of the Web and Not the Web
Itself?
If the Web were searched directly when a user entered a query, all content
would always be up to date and complete. However, this is impossible since
such a search engine would first need to send the crawler whenever a search
query is submitted. The crawler would then have to scan all pages of the Web
and search for the desired term on each page, then compile the results found,
and rank them. On the one hand, this would take far too long with the many
billions of documents on the Web. On the other hand, complex calculations
could not be carried out with the help of which documents are processed for
ranking already during the indexing process. Such an approach would simply
not be practical. We can easily see the negative effect of this approach when
searching for a keyword in a long document in a word processing program:
One after another, all occurrences are displayed; the occurrences are simply
listed according to their position in the text. Such a search takes a long time
and does not include any evaluation of the relevance of the entries.

3.1 The World Wide Web and How Search Engines Acquire Its
Contents

We already distinguished between the Internet and the World Wide Web, which is a
part of the Internet. What are the characteristics of the Web? It consists of
documents, mainly in HTML format, which have a unique address and can be
connected to each other via links. HTML is short for Hypertext Markup Language
and represents a language that can be used to represent documents and their links.
Each published document is given a URL (Uniform Resource Locator), which
uniquely defines the document’s address.
Other documents can be embedded into HTML. For example, the videos that can
be found on YouTube are embedded in HTML pages. Of course, today’s Web
3.1 The World Wide Web and How Search Engines Acquire Its Contents 29

browsers can display videos and other non-HTML documents directly, but we
usually see this content within HTML pages.
The goal of a search engine is to create a complete representation of the World
Wide Web. However, to determine how well this has been achieved, one must first
know how large the Web is. Since the Web is dynamic (i.e., it is constantly
changing) and there is no central “registry,” but anyone can simply add or delete
their own documents, this question is tough to answer.
Since no central authority manages all Web pages, the World Wide Web’s size is
also not being measured centrally. There are several methods to determine the size of
the World Wide Web. All of them are based on extrapolations, so we can never
expect exact numbers here.
The most important calculation methods are:

1. Examining a representative sample of websites to see how many documents they

contain. On this basis, we then extrapolate the total number of documents present
on the Web. Steve Lawrence and C. Lee Giles (1999) took this approach and
published a well-known study in 1999, in which they also found that the search
engines of the time covered only a small portion of the Web.
2. Using crawling to index Web documents and then determining the total number
of documents found. The problem here is that this method can only determine the
number of documents found in this single crawling process, but not how many
documents have not been found. Therefore, it is not possible to determine the size
of the Web in this way, even though search engine companies have repeatedly
published figures on the size of the Web itself based on their own data collection
(e.g., Alpert & Hajaj, 2008; see also Sullivan, 2005). A somewhat recent figure of
this kind comes from Google; in 2016, they stated that the search engine knows
130 trillion URLs (Schwartz, 2016).
3. A third variant measures the overlaps between different search engines and
extrapolates the number of documents available on the Web on this basis. Bharat
and Broder (1998) and later Gulli and Signorini (2005) used this method. This
method can determine the total number of documents covered by search engines;
however, all documents not covered by any of the search engines examined
remain excluded.
4. Extrapolation of the index sizes and the size of the Web based on the number of
hits reported by the search engines for selected queries. Here, the size of the Web
is determined as the size of the chosen search engines’ data sets, minus overlaps.
Van den Bosch et al. (2016) use this method to consistently calculate the size of
the data sets of the two major search engines, Google and Bing, and from this
calculate the assumed size of the Web indexed by the search engines (https://
www.worldwidewebsize.com). However, it must be kept in mind that the hit
counts provided by the search engines are themselves only imprecise projections
(Sánchez et al., 2018; Uyar, 2009) and are not ultimately verifiable. Together with
the assumptions used in the overlap calculation, the result is, at best, a very rough
projection, which is about 55 billion documents at the beginning of 2021.
30 3 How Search Engines Capture and Process Content from the Web

So, what can we conclude from this discussion about the size of the World
Wide Web? First of all, that we are dealing with a structure that we cannot grasp
exactly. It is equally difficult for search engines to fully grasp the Web. This, in turn,
means that not only do we not know how big the Web is but we also do not know
exactly which part of the Web is covered by search engines.
Second, we should note that the Web consists of many billions of documents and
its coverage, therefore, requires extensive technical resources and a significant
financial investment. We will discuss this in detail in Chap. 8.
Third, we need to ask ourselves what a document is in the context of the Web.
Today, many documents on the Web are created automatically from ever new
combinations of information, making it questionable whether documents on the
Web should be counted at all. Due to the sheer volume of documents and the
associated impossibility of capturing everything, some experts consider search
engines to have already failed in their claim to offer at least near-complete access
to the content of the Web.

Which Pages of a Blog Are Documents?

When you write a text and publish it on a blog, it is definitely a new document.
However, what about the overview page that is automatically generated by
blog software? It lists the teasers to the current articles, and the primary
function of this page is to direct readers to the individual articles. Should
search engines treat and capture these overview pages as separate documents?
Even if we still consider this reasonable for the overview pages – what about
all the other overview pages that outline the articles by date, by the assigned
tags, etc. and are automatically generated by the blog software?
The example is intended to show how new “documents” are automatically
compiled and published without any particular input. In principle, all content
available on the Web can be combined in ever new forms. However, for search
engines, which have a limited capacity, it is essential to identify those
documents that contain content relevant to the users.

To illustrate the crawling problems that arise due to the structure of the Web, we
will use the model from Broder et al. (2000). This model, also known as the bow-tie
model (Fig. 3.2), is based on an extensive empirical data collection of the links on the
Web and shows how documents are linked to each other and how they are not linked
to each other. Only some of the documents on the Web are strongly connected; this is
the core of the Web (strongly connected component; SCC). All documents in the
core can be reached directly via a link (Broder et al., 2000, p. 310). Links from the
core lead to an area called OUT; search engines can easily follow these links.
In contrast, links in the area called IN, which point to the core but are not linked
back to from the core, are problematic for search engines to track because they
cannot be reached via links from any page in the core. Connections between IN and
OUT exist only occasionally (tubes). In addition to the connected areas, there are
3.2 Content Acquisition 31

Tendrils

+...
IN SCC .++
OUT

Tubes
Disconnected components

Fig. 3.2 The structure of the Web: “bow-tie model” (Broder et al., 2000, p. 318)

so-called tendrils connected to one of the three large areas but are relatively isolated
in general. In an empirical study of 200 million documents, similar sizes were found
for the four areas: core, IN, OUT, and tendrils.
Of course, the numbers from the Broder et al. article are hopelessly outdated. The
relative proportions of the areas of the Web may also have changed in the meantime.
However, the model itself can show us that it is still difficult for search engines to
cover the Web by simply following links. It clarifies that it is not sufficient to simply
follow all links from a starting page and then repeat this in the documents found until
no more new documents are found. Broder et al. were able to show that this ideal-
type conception of the Web does not apply in reality and that more complex
procedures must be applied to cover the Web as completely as possible.

3.2 Content Acquisition

Search engines aggregate content from the Web in their databases; this is referred to
as content acquisition. According to Vaidhyanathan (2011), there are three areas of
content acquisition:

1. Scan and link: External content is collected, aggregated, and made available for
search. This is the basis of search engines and thus the most important form of
content acquisition. This section will distinguish between the two most important
forms in this area: crawling and content delivery via feeds, i.e., the delivery of
structured data by content producers.
Another Random Scribd Document
with Unrelated Content
CHAPTER XIV
 RETURN TO ERIVAN
September 25.—We passed the morning upon the mound, in the little open
summer-house, face to face with the airy snowfields which we had scaled to
their topmost vaulting, with the cavernous recesses which we had
penetrated to their inmost core. Such is the silence of Nature at the foot of
this solemn mountain that the faintest sound reaches the ear. I was therefore
startled by a clamour of voices in the direction of the cantonment, and I
hurried down towards the noise. A booted figure in drab uniform, covered
with dust from head to foot, was gesticulating under the influence of
extreme excitement to a little group of Russian military in their white
tunics, accompanied by some languid Orientals at a respectful interval. It
was the officer of Cossacks who had joined our party near Takjaltu, and
who had left us at Sardar Bulakh. Suiting his gestures to his words, he was
narrating a thrilling story of a night encounter with the Kurds. His little eyes
were bloodshot and distended with emotion; his legs were parted and his
feet planted firm. His detachment had fallen in with a band of marauders,
who had carried off some cattle from over beyond Akhury, and made away
towards the Turkish frontier. They had fired on the Kurds, who had returned
their fire; they had recovered the cattle and chased the Kurds away. I
enquired what bag he had made of these human vultures, and he replied,
with a sigh, that they had carried off their dead.

On the further side of the Araxes, opposite Aralykh, is situated the

celebrated monastery of Khor Virap, which marks the spot where, according
to Armenian tradition, Saint Gregory, the founder of Christianity in
Armenia, was imprisoned for thirteen years in a deep pit. The country about
and behind the cloister is extremely rich in historical and archæological
interest, and I would recommend the traveller to prolong his excursion up
the romantic valley of the Garni, whence he can return across the mountains
to Erivan. He will examine the sites of Artaxata and Dvin, and, proceeding
up the river, will reach the gorge with the basaltic columns, and the
platform where once stood the temple of King Tiridates—a beautiful Greek
shrine given to these solitudes, like the temple of Segesta to the lonely
Sicilian hills. Hard by this platform above the river are found the relics of
the city of Garni; and, near the sources of the stream, at a distance of some
five miles from Garni, the caves and monastery of Surb Geghard, reputed to
have been founded by St. Gregory, respond to the spirit of a landscape
which for grandeur and severity is unsurpassed among these wilds. I was
anxious to make the acquaintance of some at least among these antiquities;
we therefore despatched our luggage with the Swiss and the cook to Erivan,
and, availing ourselves of the offer of a victoria as far as Khor Virap,
resolved to trust to fortune for the remainder of the way.1

Had we been able to procure riding-horses, we might probably have ridden

from the ferry over the Araxes direct to the cloister across the plain. In a
carriage we were obliged to retrace our steps as far as Kamarlu, where the
road which runs parallel to the course of the river crosses the road to
Erivan. The stage which we had made after nightfall between that village
and Aralykh was now performed in the light of day. The alluvial flats
between the Araxes and the base of Ararat are channelled by a network of
irrigation runnels, which diffuse the stream of the Kara Su. From the fields
and marshes rise luxuriant cotton and castor oil plants, the one with yellow
single blossoms, like a wild rose, and drooping fruit, resembling flakes of
snow; the other, higher than these, raising a tender, juicy stem to shining,
palm-shaped leaves. Here and there, where the water fails, bushes of hardy
camelthorn spring up, like weeds, upon the fallow land. The oppressive
climate of Aralykh, no less than the plague of insects which infest it, are
due to the sand upon the pedestal of the mountain, and to these swamps
with their effluvia and mosquito swarms. Even at this season the sun beats
fiercely upon the plain; and, when we reached the ferry, a herd of buffaloes
and bullocks, awaiting transport, were rolling parched tongues and casting
longing eyes at the river from the bank of crumbling mud.

A double pontoon, staged across with planks, received our carriage, and
was swiftly impelled along the hawser by the force of the stream. From the
opposite margin a dreary tract of baked alluvial soil extends to the zone of
gardens and orchards which commences at Kamarlu. I have already alluded
to the excellence of the road within that zone; but by day you will be loth to
hasten along it, such is the charm and so great the interest of the scene. The
traffic from the lower Araxes, from Persia and distant Mesopotamia, finds
its way along this chaussée to Erivan. The district is inhabited by well-to-do
people, who can afford the richness of their national dress. Beneath the
foliage of the needle poplars, between the well-maintained mud walls—
over which you look to the vineyards and to the vegetable gardens, where
the tomato and the chili abound—a stream of wayfarers, some on
horseback, fill the pleasant avenue, chatting and smiling under the
expansive influence of ease and shade. At intervals you pass a house or
cluster of houses, where groups of Armenian women in their holiday attire
are gathered before the open doors. They are clad in their gayest cottons,
and wear their picturesque head-dress and veils of white gauze. Some
among them nurse their babes at the open bosom, the little infant cleaving
to the full breasts. Tartars, with their black lambskin hats and dark blue or
black garments, compose an element which a cynic would be loth to
dispense with in such a scene of piping peace; yet it would be difficult to
detect a trace on their clean-shaven faces of passions which have, perhaps,
been blunted by time. Laden waggons pass, and numerous bullock-carts,
with their heavy, creaking wheels. We were amused by the appearance of a
curious pair of riders who, to judge from the deference which was bestowed
upon them, were evidently of exalted rank. The man wore a flowing beard
and was dressed in Oriental apparel; but he held in his hand a parasol of
European pattern, and his locks were surmounted by an English billycock
hat. His wife was by his side, astride of her Arab; but the graceful animal
was almost invisible beneath her, his withers overtowered by the huge bulk
of her stomach, and his back enveloped in the folds of her robes. It was an
Assyrian bishop, journeying from Mosul.

Kamarlu is perhaps a type of these villages of the campagna, in which the

population is composed of Armenians and Tartars, of lambs and lions living
side by side. It can boast a Russian schoolhouse, a necessary institution in
the case of the Tartars, to judge by the barbarous and hideous frescos which
enliven the façade of their little mosque. The Armenians have their school,
and there are two Gregorian churches in which they satisfy their spiritual
needs. The houses are built of sun-baked bricks and mud; wooden stages
rise to some height above the flat roofs, and provide airy sleeping-places for
the inhabitants during the summer heats. After regaling ourselves with the
delicious white grapes of the district, we turned aside from the road to
Erivan. Crossing the outskirts of the village, we remarked the huge clay
wine jars which were strewn about in the courtyards. Beyond a few fields,
planted with cotton, we again entered the open desert, and pursued our way
over the crumbling mud. A rude and winding track leads towards the river
through patches of dusty desert shrubs. Ararat fills the landscape, and is
rarely seen to greater advantage than from such tracts of naked land. On our
left hand rose a buttress of the Sevan mountains which had been a landmark
from the slopes of Ararat. It is composed of a sandy rock of various hues,
which has weathered into fanciful shapes. In the delicate evening lights it is
invested with the appearance of some castle in fairyland.

From time to time we passed strings of three or four large waggons, drawn
by teams of oxen. Whole families of Armenians were gathered within them,
well dressed and well-to-do. They were returning to their dwellings within
the zone of gardens from a pilgrimage to Khor Virap. The men were
emptying their little glasses, which they would replenish from wine-skins,
and feasting on water melons.

We arrived at the mound which rises from the flats about the river and can
be clearly seen from Ararat. According to Dubois,2 it consists of a mass of
dolomite, isolated on the surface of the plain. The church and cloister have
been built on the side of the eminence; the monastic dwellings screened the
church from our view. St. Gregory’s dungeon is situated within the
precincts; and it would appear that the place was famous in the saint’s
lifetime for a much-frequented temple of the fire-worshippers.

We were scarcely beneath the walls when the figure of a horseman springs
forward from some recess into the road. Throwing his white Arab on to his
haunches at a few yards before our carriage, he challenges and constrains us
to pull up dead. This proceeding on his part, no less than his forbidding
countenance, throws me completely off my guard. On Russian soil one is
obliged to smother the irritation which is always threatening to burst forth
from a British breast. I shout to him to move aside, or we will whip the
horses and drive through him; to this he answers by drawing his revolver
and threatening to shoot. I ask him by what right he dared to obstruct the
roadway; he replies by enquiring by what credentials we presume to pass. It
flashes through me that the game is in the hands of this ruffian—we had
been spoilt by the attentions of the high officials, and to such an extent that
we had forgotten to bring even our passports, which had gone in our
despatch box to Erivan. It was useless to urge that one could not be obliged
to show a passport in order to be allowed to visit a church. He paid no heed
to any of our arguments, and compelled us to return with him to Kamarlu.
He even added the insult of requiring us to suit our pace to his, and to
follow at a walk or amble by his side. This we flatly refused to do, and,
taking the reins from the trembling coachman, proceeded at a brisk trot.
Simon Ter-Harutiunoff—such was the name of this ferocious person—is
linked in our memory with the companion picture of Ivan the Terrible, our
stern custodian during the Akhaltsykh days. Both are Armenians, and either
might be taken as a model for the embodiment of the fighting instincts in
man. Tartars and Cossacks are amenable creatures besides them; and of the
two, we were inclined to bestow the palm upon Simon. His face was black
with exposure to the sun; the eyes were yellow round the dark iris and shot
with red veins. His features were large and pronounced, but of singular
deformity; the massive head was placed upon broad shoulders above a
frame of great bulk and iron strength. He wore two medals, won during the
war with Turkey through personal bravery. His function in time of peace
was to police the Persian frontier in the district of Khor Virap.

These particulars we learnt in the office of the Pristav, upon our return
under such escort to Kamarlu. We claimed and were permitted to proceed to
Erivan; but the chapars were instructed to prevent us from diverging, and to
hand us over to the Nachalnik at the provincial capital. In this manner we
were foiled in our antiquarian researches among these ancient sites. At Khor
Virap we saw nothing but some slight convexities in the surface of the
ground, which may be caused by buried remains. Beyond the mound we
observed a natural wall of rock, rising like a gigantic ruin above the plain.

Evening had approached as we left the village, and proceeded through the
gardens, and crossed to the barren zone beyond. From the rising ground we
looked back over the forest of poplars to the sun setting behind the peaks of
the Ararat chain. The satellite range wore the same tints of deep, opaque
opaline which fretted the horizon during our outward journey. It was
shadowed upon the same ground of orange and amber; and the opal hues of
the land forms extended round the circle and included the huge, horizontal
outline of Alagöz. But the Sevan mountains, in the opposite segment, were
touched with pink and luminous yellows; the higher summits were white
with fresh snow. In the south-east the landscape was dim and vaporous; nor
could the eye distinguish among the gathering shadows the basal slopes of
Ararat. The snow-fields of the dome shone with a cold light in the sky,
above vague banks of cloud. It was after eight o’clock when we reached the
pleasant town garden, and discussed our adventures with the Nachalnik
over a cigar.

1 For Artaxata, Dvin, Khor Virap, etc., see Ker Porter’s Travels (vol. ii. pp. 619 seq.);
Morier (Second Journey, p. 316 and pp. 339 seq.); Dubois (op. cit. vol. iii. pp. 382 seq.);
Smith and Dwight (op. cit. pp. 273 seq.). Dubois mentions, but was unable to visit, the
grottoes of Okhtchapert on the direct road between Erivan and Garni, p. 402. They are
mentioned by Telfer (Crimea and Transcaucasia, vol. i. p. 210), who passed by them on
his way to Garni from Erivan. Telfer’s book should be consulted by English readers for an
account of these various antiquities. I would also recommend to the archæologist who is
desirous of investigating the question of the site of Artaxata a reference to Dubois (vol. iii.
p. 449). ↑
2 Op. cit. vol. iii. p. 480. ↑
CHAPTER XV
 AT ERIVAN
Oriental cities—and Erivan is still essentially Oriental—may perhaps be said to be built
upon two planes. There is the plane of the street, and there is the plane of the flat roofs,
all at about the same level. Where the climate during summer renders the rooms of the
house untenantable after the walls have been heated through by the sun, the daily life of
the inhabitants undergoes a corresponding division into the life of the street and the life
of the roof. About an hour before sunset the entire population mounts from the lower
apartments, or even from the cellars, to the open platforms, floored with mud and
sometimes protected by a low balustrade, which receive the freshness of the evening
breeze. It is there that the last and first meals of the day are served, and the quilts spread
upon which sleep is enjoyed beneath the stars. A strange scene it is when the faint light
of morning has broken, and when the recumbent forms commence to stir. The divisions
made by the narrow streets are scarcely perceptible; your own roof appears to join the
roofs of your neighbours, and these to compose a single and elevated stage above the
landscape of dim earth and flashing stream. Figures, erect from the waist, are revealed in
every posture; and it may happen that the cotton drapery has dropped from a woman’s
shoulders as she stretches her arms in the fancied seclusion of some partial screen. Such
scenes are the daily accompaniment of a summer sojourn in the towns upon the lowlands
through which the Euphrates and the Tigris flow. In Armenia, with a mean level of
several thousand feet above the sea, the practice of sleeping in the open is confined to
the depression of this plain of the Araxes; and even here it is only partially indulged. The
better-to-do among the inhabitants take refuge in the adjacent mountains when their
dwellings have become little better than furnaces. The traveller is advised to swelter
within four walls rather than tempt fever from the expanse of irrigated land by exposing
himself to the night air.
 Fig. 40. Ararat from a house-top in Erivan.

Yet the twofold division of the city into an upper and a lower region is nowhere more
productive of startling contrast than in this town of gardens which is Erivan. In the
streets, lined as they are with the rude stone walls of the enclosures, surmounted by a
crumbling ridge of clay, the vistas are confined by inexorable foliage to the space of a
stone’s throw. The central park, with its wide spaces, enjoys no further landscape than
that which is limited by the zones of the adjacent buildings or by its own lofty forest
trees. Where you are not threading the narrow alleys of the more thickly inhabited
quarters, you will be winding by irregular ways, deep in white dust, by the side of
swirling water or within hearing of its murmur beyond the bulwark which screens the
orchard from the lane. But from the standpoint of the roof the horizon expands to
boundaries which are so remote that they are scarcely conceivable by a European mind.
The foliage or the hollow of the site eliminates the middle distance; and the opposite
piles of Great Ararat in the south (Fig. 40) and of Alagöz in the north (Fig. 41) rise
immediately from the soft foreground of the embowered houses. The landscape from the
high ground on the north, as you approach Erivan by the road from Tiflis, is difficult to
forget (Fig. 42). The whole fabric of Ararat is exposed from base to summits; but so tall
are the poplars and luxuriant the countless varieties of fruit trees, that they almost
conceal the domes of the mosques and the cupolas of the churches, spread over the
straggling township at your feet.
 Fig. 41. Alagöz from a house-top in Erivan.

All this verdure is mainly due to the river Zanga, the Hrazdan of the Armenians, which
collects the drainage of a section of the southern slopes of the border range, and which is
fed by the waters of Lake Sevan, called also Gökcheh, from its sky-blue colour, and by
Armenian writers the Lake of Gegham. This beautiful alpine sea is surrounded by lofty
mountains and has an area 2½ times as large as that of Geneva. It produces salmon trout
of delicious flavour which are seldom absent from the bill of fare in the provincial
capital. It finds an outlet through the Zanga into the Araxes at a difference in the level of
3600 feet. The brawling Zanga, already weakened by the canals which diffuse its waters,
pursues a devious course at the foot of high and rocky banks on the western outskirts of
the town. Further eastwards the irrigation is supplied by the Kirk Bulakh, a stream of
which the name signifies forty springs, and which has its sources at no great distance
from Erivan. Such abundance of running water should secure to this growing city a large
measure of prosperity under settled government. As the centre of the most populous of
the Armenian provinces of the Russian Empire, to which it gives its name, it is already a
place of some pretensions. But the inhabitants do not at present number more than
15,000, of whom half are Tartars and half Armenians. This total also comprises about
300 Russians, whose most conspicuous units are the drivers of the carriages on hire,
belonging, I believe, exclusively to the Molokan sect.1
 Fig. 42. Erivan and Ararat from the North.

Erivan does not possess any monuments of first-rate merit or of great antiquity. Her
origin is obscure. Noah may quite well have lived here before the Deluge, as one of the
earliest of modern European visitors was informed by his Armenian friends.2 The
popular derivation of the name is from the Armenian verb erevel, and it is said to signify
appearing. The place would, indeed, be about the first locality in the plain region to
appear to the eyes of the patriarch of old.3 Hither may have been directed his steps and
those of his family when the waters had receded from a world renewed. This may be the
site of the original city of Noah, perhaps preserved beneath the soil upon which is built
the present town. The more learned are inclined to a much later foundation, but do not
yield in point of philological plausibility to the champions of the identification with
Noah’s city. They say that the name has been shortened from Erovantavan, which they
render the place where Erovant was defeated. Erovant or Ervand was an Armenian
monarch of the first century who was vanquished in this region by the lawful heir to the
throne of the Arsakids at the head of a Persian army. The event and the survival of the
name Erovantavan are attested by Moses of Khorene.4 The Mohammedan derivation
from Revan Kul, a prince of the reign of Shah Ismail (1502–1524),5 who is said to have
fortified the place by his master’s order, cannot be reconciled with the fact that Erivan
was already in existence certainly in the eleventh and probably in the seventh century.6
But it played no prominent part whether in ancient or mediæval history until the advent
of the Ottoman Empire. From the sixteenth century into modern times it was continually
disputed between the Sultans and their powerful neighbours on the east, the Persian
Shahs. The enumeration of the sieges it sustained at the hands of Turks and Persians
would be a tax upon my reader’s patience which I am not disposed to levy. When the
Russians appeared on the scene it was in Persian possession; and an unsuccessful
attempt on their part to capture the fortress in 1804 supplied the ground for the firm
belief in its impregnability which was cherished by its Persian governors. This
confidence was rudely shattered by Paskevich in October 1827. His shells wrought
fearful havoc in the unsubstantial town, and one is said to have pierced the dome of the
mosque in the citadel, whither thousands of the wretched inhabitants had fled for
protection against the hail of the cannon. The Russian army entered the place without
encountering any serious obstacle, and the Russian flag has waved there ever since.7

One might expect to find some mosques of considerable age in a city which flourished
under its Mohammedan masters. One must, however, recollect that the Ottoman Turks
are Sunnis and the Persians Shiahs; what the one may erect the other loves to destroy.
We are expressly told that when Shah Safi took the place in A.D. 1635 all the mosques
built by the Turks were razed to the ground.8 About the same time the position of the
town, or perhaps only of the fortress, underwent a change, being removed some eight
hundred paces to its present site on the rocky cliffs at the foot of which the Zanga
flows.9 The Persians do not appear to have enriched it at that period with any remarkable
buildings; and it was recovered by the Turks in 1724.10 Some ten years later it again fell
into the hands of the Persians as one of the conquests of Nadir Shah. The principal
mosque is said to date from the reign of this monarch. The curious old tower which was
seen by Chardin as well as by Tournefort, and of which the lineaments have been handed
down to us by the former of these travellers, has long since disappeared.

Still the buildings which at present exist are well worth a visit; and I propose to invite
my reader to accompany me in a leisurely ramble through the alleys of Erivan. The more
populous quarters are divided into a western and an eastern half, at first by the broad,
metalled road which comes from Tiflis, and, further south, by the central park. Speaking
generally, the eastern half is inhabited by the Tartars and the western by the Armenians.
In the one you will discover the mosques, in the other the churches. But the churches are
either small and quite insignificant stone structures, or have been restored beyond
recognition in comparatively recent and tasteless times. I counted no less than six,
including the Russian church at the southern extremity of the town. Of these the oldest
foundation would appear to be that of Surb KatholikeSurb Katholike, which stands in a
pleasant walled garden, adjoining the great road, in the upper or northern quarter. An
ancient elm dwarfs the humble oblong edifice, which is entered from a portal on the
south side, added in 1861. The interior, which is very low, is disposed in a nave and
aisles, an apse and two side apses or chapels. Chardin attributes a church of this name to
the latest kings of Armenia, and the priests assured me that it was indeed the earliest in
date at Erivan. It was here that in Persian times the katholikos would officiate, while
residing in the provincial capital.

A little lower down the road we pass Paulos PetrosPaulos Petros. (Paul and Peter), the
largest and the least pleasing of the town churches. But once we have left the wide
avenue to become involved in the network of gardens on the north and north-west, any
mediocrity in the buildings we visit is amply compensated by the charm of the
enclosures in which they stand. Such verdure of every shade and constant hum of
flowing water! To Surb JoannesSurb Joannes. we come first—four walls and a metal roof,
to which is attached a wooden belfry, painted green. You see the Zanga issuing from a
cleft in the barren hills, of which the hardness contrasts with the foliage at their base.
The little portal of Joannes is quite a pretty feature, and I was informed that the church
dates from the latter part of the seventeenth century. A more ambitious structure is Surb
ZoravarSurb Zoravar., situated some little distance in an easterly direction, but still within
the zone of these high slopes on the north. It is surrounded by old gardens and
overshadowed by walnut trees. The body of the church is quite plain, four walls and a
roof of low pitch; but an elaborate portal, surmounted by a belfry and supported by four
massive piers, extends the whole length of the west front. Two piers in the centre are
panelled and richly carved by the most delicate of chisels. There is a very old doorway
on the south side with spiral mouldings, and the frescos over the principal entrance—a
rare feature—are well drawn and show good feeling for colour. I understand that the
present church has supplanted an older building; but I will not vouch for the statement
that the portal is due to Moses Katholikos (A.D. 1629–1632), as I was informed by the
aged and ignorant priest. He came at last, after many peals from the belfry, his tottering
frame supported by a lay companion. The clergy of Erivan are not more enlightened than
the most backward of their profession in remote districts of the Turkish provinces.
 Fig. 43. Entrance to Gök Jami, Erivan.

On the other hand the greater material well-being of the laity is made manifest by the air
of comparative comfort presented by the interiors of their places of worship. Of course
one misses the pews of our English churches, or the serried lines of chairs which furnish
the temples of the Continent. But the floors are well carpeted and the bare walls kept in
repair. From Surb Zoravar one may readily regain the Tiflis road and pass in a southerly
direction along the central park. Thence it is no great distance to the principal mosque of
the city, the Gök JamiGök Jami. or mosque of heaven. This edifice is situated in the
western half of Erivan, and is surrounded by dwellings of Tartars in considerable
number, overlapping into the Armenian quarters. It is approached from the narrow streets
of a bazar consisting of booths, and is entered by a handsome doorway at the side of an
imposing minaret, of which the surface is diversified by designs in polychrome tiles
(Fig. 43). You pass through a vaulted passage into the great court (Fig. 44). It is a vast
place, shady and serene. Lofty elms of great age shadow the basin of overflowing water
which bubbles in the centre of the paved spaces. Upon its margin are gathered figures in
long robes and turbans, or attired in the Persian fashion and wearing the Persian
lambskin hat. These are busy with their ablutions; while elsewhere, beneath the shade,
mollahs are instructing groups of their younger pupils, seated on mats spread upon the
flags. Beds of single dahlias refresh and please the eye. Of life and movement there is no
lack; people are coming and going; there in the distance a train of shapeless forms in
deep blue draperies makes its way to the women’s mosque. But the absence of the least
suspicion of haste spreads an atmosphere of delightful repose. It requires no small
fortitude—they would call it diseased curiosity—to pace from side to side and ascertain
that this quadrangle measures 87 paces by 58. The latter is the dimension of the side on
the south, upon which is built the temple itself (Fig. 45). Beneath the spacious dome men
and women are gathered indiscriminately, the women veiled in Persian fashion. There is
nothing very remarkable in the architecture of the mosque; but the floral paintings which
adorn the ceiling of a companion and smaller edifice on the north side of the court are of
very high merit. The remainder of the quadrangle is taken up by rows of low buildings,
containing chambers in which the older scholars pursue their studies. One wonders what
they may be learning. A mollah of importance informs us that the Gök Jami was built in
the time of Nadir Shah (A.D. 1736–1747) by the sirdar, Hoseyn Ali Khan.

Fig. 44. Court with basin of Gök Jami, Erivan.

With the exception of the mosque in the fortress, the religious edifices of the
Mohammedans are extremely well maintained. I counted three mosques in the Tartar
quarters. That of Haji Nusrallah Bey and the Shehr JamiHaji Nusrallah Bey.
Shehr Jami. (town mosque) are almost exactly similar in design. The former is evidently a
replica of the latter, which displays a Turkish inscription on the outer door with a date
which we read as 1098 (A.D. 1687). But it must have been restored since that time.
Although much smaller than Gök Jami, it bears some resemblance to that building; and
the walled court with its fountain and beds of long-stalked dahlias is as pleasant a refuge
from dusty alleys as man could desire.

Fig. 45. The Temple. Gök Jami.

But perhaps the most interesting monument is the kiosque of the sirdarsKiosque of the
sirdars., in the extreme southern angle of the town. We may approach it from the west,
and take Surb SargisSurb Sargis. on the way. That church and pleasant terrace on the high
land above the Zanga commands an extensive view over the southern quarters and across
the plain to Ararat. The deeply-bedded river is flowing on an easterly course towards the
fortress and the gardens of the sirdars outside its walls. After skirting those parapets it
will turn abruptly in the reverse direction, and pursue a more tranquil career to the
Araxes. The fortress to which we proceed is still some distance off, and the walls of mud
and rubble which line the cliffs on the left bank of the Zanga are rapidly falling into total
ruin. While they are flanked by the swirling stream they may once have possessed some
power of resistance; but after the river has deserted the site beyond the abrupt bend, the
town is exposed immediately to the plain. The sirdar’s palace composes the kernel of the
fortified area, and its windows overlook the river. But the extensive buildings of his
well-stocked harem, the magazines of his garrison and the abodes of his courtiers have
either disappeared altogether or are rapidly crumbling away. From among a heap of ruins
rises intact a single edifice, which is kept in repair by the Russians. It is the pavilion in
which the sirdar was wont to beguile his leisure. From the window in the alcove of this
elaborate interior (Fig. 46) he would feast his eyes on the landscape—the river at his
feet, his own shady garden in the plain, the dim spaces backed by the fabric of Ararat.
Here he exercised his skill as a marksman upon the donkeys of the unfortunate peasants,
sending a ball through them as they wound along the road on the right bank of the Zanga
towards the bridge with its two pointed arches.11 This bridge is placed just below the
pavilion, and is still the only avenue of communication between Erivan and the country
beyond the river. What consummation of Oriental felicity to sit on cushions in this
glittering apartment and watch the caravans which fill your coffers defiling below! From
time to time there may come an embassy to your overlord of Persia, and there will be a
report to dictate upon the size and splendour of the cavalcade. The beauties of Georgia
and Circassia luxuriate in the adjoining halls, and water flows in abundance everywhere.
The governor of Erivan was quite a little king in the country, and, when he travelled, the
inhabitants of the villages along his route would immolate an ox in his honour.12

Fig. 46. Erivan: Interior of the Kiosque of the Sirdars.

The incrustation which my reader may admire upon the vaulting of the alcove is
composed of pieces of mirror which shine like the facets of a jewel. An encrusted
cornice of the same material surmounts the walls of the pavilion below a ceiling
profusely adorned with floral designs, conspicuous being the iris and the rose. Eight
paintings on canvas, applied to shallow recesses, are distributed around the room. I
believe they are copies, made since the Russian occupation, of originals which had fallen
into decay. The two which are comprised by my illustration, one on either side of the
alcove, represent on the left hand the figure of Hoseyn Khan Sirdar, and, on the right, the
Persian hero Feramez. Of the remainder, three are portraits—Fath Ali, Shah of Persia
(1797–1834), his son Abbas Mirza and Hasan Khan, brother to the Sirdar Hoseyn; while
an equal number are indifferent renderings of heroic personages—the warriors Sherab
and Rustem, and a Persian Amazon. One of my predecessors has recorded that at the
time of his visit in 1834 the panels in the alcove were adorned with four pictures setting
forth subjects which were well conceived to amuse the fancy of an old debauchee. A
Mussulman was receiving wine from a fair Georgian in the presence of the monks of
Edgmiatsin, whose arguments had been less potent to effect his conversion than the
fleshly charms of the Christian girl. A Persian beauty in loose trousers and diaphanous
upper garment was making her obeisance to the Shah. Here a prince of the blood royal in
costume of the chase dallied with a maiden while her aged father lay asleep; there the
beautiful features of Joseph spread havoc among the assembled ladies at the house of the
wife of Potiphar.13 These various incitements to delight no longer grace the forlorn
kiosque, and perhaps their disappearance is no great loss to the world of art. The original
decoration, which is quite intact, upon the walls and ceiling enables us to judge how
great had been the artistic decadence of Persia since her painters displayed their skill
upon the walls of the Chehel Situn, the noble pavilion on the banks of the Zenda Rud.

From this kiosque we may make our way to the adjoining mosque of the fortressMosque
of the fortress., which is now no longer frequented by the faithful. It stands a little east of
the old palace; the interior beneath the spacious dome is decorated with much skill by
means of little bricks of many colours. The great court is already ruinous. An old henna-
stained attendant informed us that it was erected in the reign of Fath Ali Shah and that it
was known as the Abbas Mirza Jami. Walls and palace and mosque are, I conclude,
already doomed. Hard by their crumbling remains are seen the barracks of the Russian
garrison and the metal roof of a Russian church. The last of the sirdars is already long
since dead, he whose portrait hangs on the wall of the pavilion. He died in a miserable
stable, bereft of everything but the squalid garment which clothed his aged body. Yet his
memory is pleasantly associated with one of the favourite episodes of Persian romance.
It is related that a young Georgian travelled to this fortress above the Zanga to catch a
glimpse of his betrothed in the sirdar’s harem. The girl, espying her lover, precipitated
herself towards him from the window, and was saved from certain death by a willow
which broke her fall. The pair were captured; but the incident touched the heart of her
jealous owner, who pardoned them both and let them go. His generous speech has been
preserved: “Hearts so closely united let no man endeavour to part.”14
Perhaps the best introduction to the population of a city consists in a visit to the schools.
Erivan is better supplied in respect both of elementary and secondary education than any
other town in the Armenian provinces of the Russian Empire. But, before recording my
personal impressions of what I saw during a brief inspection, I should like to review the
conditions which govern the schools. When Russia became mistress of a large portion of
Armenia, her rulers found that their Armenian subjects were already in possession of a
school system of which, with their customary tenacity, they were extremely jealous, and
which probably dated from the invention of the Armenian alphabet as early as the fifth
century. The Church has been for long ages the pillar of Armenian nationality; and the
schools were affiliated to the Church. There were not therefore wanting all the elements
of a bitter quarrel; and if any question more than another has envenomed the relations
between the Armenians and their Russian rulers it is this question of the schools.

When the constitution of the Armenian Church and its relations to the Government were
embodied in a State document, a chapter was inserted by virtue of which the Tsar of
Russia formally recognised the Church schools.15 They were stated to have as their
object the religious and moral education of the children, and to be under the guidance
and supervision of the bishops. It was provided that their rules and curricula should be
submitted to the synod at Edgmiatsin, and that this body should in turn transmit them for
acceptance to the Minister of the Interior. A rider was added to the effect that it was a
matter of importance that the clergy should become acquainted with the Russian
language, and with the history and geography of the Russian Empire.

It is only fair to the Government to remark, by way of parenthesis, that although a period
of over half a century has elapsed since the promulgation of this document, few teachers
and still fewer pupils have yet displayed even moderate proficiency in the speaking and
writing of Russian. With the growth of material prosperity, which was the outcome of
the Russian occupation, the Armenian schools prospered and their standards rose. The
teachers, who were laymen, were taken from good families; and one may safely assert
that at the present day the Armenian youth are instructed by the best educated and best
informed among their countrymen. Many of them have studied in Europe, principally in
Germany, and are men of far higher attainments in the field of knowledge than such as
might be required by the teaching which they are permitted to dispense. The first step
taken by Government to cut the wings of the national schools was the limitation of the
standard of instruction. The class is in Russia the measure of this standard, the first class
standing at the bottom of the scale. Schools of five classes were frequently attached to
the churches; and the scholars who desired to pursue their studies still further passed to
the so-called seminary of the diocese in which they lived. In this manner it was possible
for a youth to receive all but the highest university education in his native language and
through his native institutions. It is true that the Minister of the Interior had a right of
censorship; but in view of the gravity of the fancied danger this safeguard was only
partial. So the Government drew the pen through the third, fourth, and fifth classes and
left the Armenians nothing more than the elementary course. Such action was thought to
be arbitrary in view of the fact that these schools are supported by purely voluntary
contributions.

Empire! what insidious wickedness, surpassing the horrors of war, is committed in the
name of empire! Surely it is a right as elementary as that of security for life and property
to supervise the education of your children. One might sympathise with the Russian
Government had they merely required that the standard of instruction should not fall
below the standard of schools in Russia. Nor should we be inclined to withhold our
sympathy if they had only renewed their insistence upon the necessity of a knowledge of
Russian. That was the wise as well as the humane policy. The ukase of 1884 was
conceived in a very different spirit, and may be branded as an infamous document. It
provided that Church schools with more than two classes should be placed upon the
same basis as private schools in Russia, that is to say that the whole of the instruction
should be conducted in the Russian language. This was tantamount to closing such
schools. The supreme control of the elementary schools was transferred from the
Ministry of the Interior to the Department of Education. The seminaries were suffered to
exist upon the basis of the decree of 1836, but their object was defined to be the
preparation of clergymen to meet the requirements of the Armenian Church.

The synod at Edgmiatsin, although already placed in leading strings by Government, did
not see their way to accept this decree. They urged that, since it had been issued during a
vacancy of the Chair, its consideration should be postponed until the election of a new
katholikos. Government retaliated by closing the schools. Nor were they again opened
until in 1886 the pontiff Makar signified his consent to the provisions of the ukase,
subject to some small concession as to the scope of the curricula in schools of two
classes. The higher classes remained closed. Such was the situation at the time of my
visit. It had, however, been further enacted that after the lapse of a prescribed period
every teacher in an Armenian school should be required to possess a certificate from the
Russian Department of Education. In order to obtain this certificate the candidate must
pass an examination conducted in the Russian language. The term of grace was coming
to an end in a few months, and I gathered that few teachers had acquired the necessary
linguistic proficiency.16

Education is not a department of human activity which can be properly conducted upon
military principles. The only discipline healthy for the mind is that which is derived from
the unfettered exercise of the faculties with which it has been endowed. In Erivan I had
occasion to remark the contrast in intellectual atmosphere between the Russian and the
Armenian school. Here were offered two typical examples of these diverse species, still
existing side by side. As the capital of a diocese, the Church has still the right to possess
a seminary in the town of Erivan. The seminary embraces the standards which we may
call secondary education, and has no less than six classes. It has contrived to evade the
restrictions which are in the spirit of the ukase of 1884 in respect of the character of its
pupils. It was quite obvious that very few were destined to take orders, although perhaps
the majority of the 360 scholars were included in the elementary classes. There was no
trace of any clerical bias in the choice of treatises; and the teachers in secular subjects
were, I believe, all laymen. One at least was a young man of exceptional ability, trained
in Europe at his own expense. It would be difficult to find among the staff of our
secondary schools a master better equipped for his task. The pupils, whose age extended
from ten to twenty years, did not appear to acquire knowledge by rote. The Principal
spoke the German language fluently and was in touch with the thought of the West. Yet
even this privileged institution has been clipped of much of its usefulness by being
placed at an unfair advantage as compared to the Russian school. It is interdicted the
seventh and eighth classes, although there can be no doubt in respect of the competency
of its staff. It is perhaps for this reason that it is not as a rule attended by sons of the
richest citizens. Its income of £1800 a year is principally subscribed by Armenians of
means. Only about a sixth of the sum comes from the pupils. The majority receive their
education free of charge.

The subjects taught in the highest class are theology and psychology, mathematics,
physics, logic, modern history and modern languages. In the latter category they are
restricted by order to Russian and French. The instruction is conducted in Armenian
except in the case of Russian language and literature, when the Russian tongue is used.
Their text-book in psychology was a Russian translation of Alexander Bain and in logic
of W. S. Jevons. Besides this seminary, which is attached to the church of Surb Sargis,
there is a school for girls with 200 pupils.

The Russian school is mainly supported by the State out of revenues derived from
taxation. It has the rank and is known by the name of a gymnasium in the German
acceptation of that term. Its subvention produces a yearly income of £4500, which is
supplemented by the fees paid by nine-tenths of the scholars, amounting to about £4 a
head. Out of 260 boys and youths some 26 were boarders and the rest day pupils. The
boarders sleep in a long dormitory, kept scrupulously clean and neat. The majority pay
for their maintenance £25 a year; the poorer can only afford £15. The school is housed in
a commodious building in the centre of the town and exhibits every sign of prosperity. It
has large and well-furnished reception rooms for days of fête. The class rooms, with
their rows of forms and large black-boards, inspire a salutary awe. The library is well
stocked and does the Russian Director great credit, as does the general organisation of
the institution.

But the spirit of the place is that of the camp; the methods are purely military, and one
almost expects the sound of a bugle to announce which lesson shall be rehearsed. Since
human memory is of brief span and the recollection of facts is of no great value, it is not
so much this faculty that requires cultivation as the habit of study and the power to
collate facts. The education dispensed by this school will not produce scholars or
thinkers; indeed the pen is here the servant of the sword. But at least it serves to sharpen
the wits, and to induce a nimbleness of mind which can scarcely fail to be of use to its
Mohammedan members.

All who can afford to buy a uniform appear in trousers and tunic of blue cloth, enlivened
with brass buttons. A dress of similar material is worn by the ushers. The pupils are
drilled and put through simple military exercises; they may be seen marching with music
at their head. Yet this is a civil institution. It is the only gymnasium or High School in the
Russian provinces of the Armenian plateau. At the time of my visit the school list
contained the names of 159 Armenians, 67 Russians, 9 Georgians, 7 Poles and 18
Tartars. Only the last belonged to the Mohammedan religion.

When it is remembered that the Tartars compose one-half of the inhabitants and are
numerous in the districts about Erivan, the poor show which they make among the
inmates of this important school is a very significant fact. As a body, they shut
themselves off from Western education; and for this reason they appear destined to be
edged out by the Armenians, as a species unable to adapt itself to the new environment.
They are still in possession of some of the richest land in the province, and many among
them are wealthy men of leisure. These khans occasionally send a son to the school. But
the Director informed me that youths of this class were rarely successful; they were
indolent and left at an early age. Those who belonged to the middle class stayed longer
and were much more hopeful. Although I passed through every room while the students
were pursuing their tasks, I only counted six Tartars, all told. The method of procedure
was extremely entertaining. Accompanied by the amiable Director, I was introduced to
the presiding usher, who would descend from his daïs and extend his hand. Some fifty to
a hundred bright black eyes were focussed upon us; all were standing, not a muscle
moved and not a sound was heard. Then some such little comedy as this would be gone
through:—

The Director (addressing myself in German). “This is the Latin class. Permit me to
present you to M. ——off”. (In Russian) “Pupils, you may sit down” (a single clap and
shuffle—perfect silence). “You, Sir, will please address the Professor in the Latin
tongue.”

Myself (after a long and embarrassed pause). “Gratias ago; clementiam, benigne rector,
reposco. Consuetudinem linguæ Latinæ parum conservo. Verum versus video in nigra
ista tabula inscriptos, mihi valde familiares: ‘O utinam tunc quum Lacedæmona classe
petebat, obrutus insanis esset adulter aquis.’ Vellem interrogare discipulos quisnam ille
fuerit adulter.”
The Usher (a forlorn and crushed individual. At first listless; but he encounters the
flashing eyes of the little Director, and stammers). “Sv ... svit ... niet, niet ...” (and he
proceeds in Russian).

The Director. “My colleague desires me to state that he quite understands what you said.
You wished to express admiration of our new blackboards. I thank you in his and my
name. Is there any question you would like to put?”

Myself. “There appear to be about thirty boys in this class. I wonder what proportion
Tartars bear to Armenians among them.”

The Director. “Russians, stand up!” (some four or five fair-haired and closely-cropped
youths rise in their places. Their faces show intelligence, and one likes them)
—“Armenians, stand up!” (the first batch sit down; practically the whole class springs to
its feet)—“Tartars, stand up!” (one little boy at the extreme end of the class confronts his
seated schoolmates).

One feature of this institution seemed specially well conceived; it was the manner in
which the religious difficulty was solved. Two different religions—the Mohammedan
and the Christian—and three distinct professions of the latter—the Gregorian Armenian,
Roman Catholic (Poles), and so-called Russian Orthodox—were represented among the
pupils and were expounded to their several votaries by as many diverse types of the
holders of sacerdotal office. Separate rooms were set aside in which the mollah taught
Islam, and the papa or padre or vardapet explained the New Testament. In this manner
each youth received instruction in the faith of his fathers at the hands of one of its
official exponents; while the rub and wear of continual intercourse in the secular classes
accustomed Mohammedan and Christian, Russian Orthodox and Gregorian Armenian to
respect their classmates and to tolerate each other’s faith. The extension of such a system
over the whole of these provinces would be likely to work incalculable good; and, side
by side with glaring defects in the methods of secular instruction, it is a real pleasure to
be able to congratulate the State schools upon such a salutary feature and cordially to
wish them success.

The Tartars of Erivan are for the most part of Turkish descent, and of kindred race to the
bulk of the inhabitants of the neighbouring Persian province of Azerbaijan. But some of
the number included under this name in the statistics may more properly be designated
as Persians. All profess the Shiah tenets. I had expected to find them extremely fanatical,
judging by my experience of their co-religionists in Persia, and by an account given of
them by a French traveller.17 But not only are Christians permitted to enter their
mosques; they are even received with cordiality by the groups assembled in the outer
courts. I do not know whether this altered demeanour may be due to a policy of no
nonsense pursued by the Russian Government. If such be the case it is a significant fact.
How often have I stood before the door of a mosque in Persia, casting eager glances at
the vista of priceless treasures within! On each occasion I have in vain appealed to the
Governor, who would urge that he could not be responsible for my safety, and beg me
not to attempt to enter. At Erivan I was invited to penetrate into every part, and to stand
by the side of the faithful while they prayed. I have already stated that the Tartar
inhabitants include many men of means, who live on the proceeds of their extensive
gardens. But a good proportion of the large shop-keepers belong to this race, and are
well-mannered and fairly well-educated men. I fancy, however, that they would scarcely
be able to compete with the Armenians, were it not for the support of patrons of their
own blood. For the rest, the small hucksters and the sellers of fruit are in a very large
proportion Tartar. So, almost exclusively, are the workers in mud after their various
kinds: plasterers, embankers, makers of ducts to water the gardens. The gardeners and
drivers of carts largely belong to this nation; but there is scarcely a carpenter or a skilled
mason who is not an Armenian.

While the Tartars are reputed to hoard, the Armenians are excessively lavish, and spend
large sums in building themselves fine houses. Many an ornate villa in Italian style may
be seen emerging from the foliage of the gardens. Here and there quite a little palace
faces the street. Yet, with all their comparative wealth, they have not yet emerged from
the material stage, and I searched in vain for a bookseller. Indeed, in spite of many signs
of progress and of her favourable geographical position, Erivan can scarcely yet be said
to be connected with the pulse of the great world. Here is a city not so far from Europe,
and needing capital for her development; yet scarcely any capital has found its way in.
Teheran, although much more distant, has a numerous European colony; and there is not
an enterprise, from banks to electric lighting and tramways, which a number of
candidates are not contending with each other to supply. You will not meet a single
foreign industrialist in Erivan, nor be able to purchase any but Russian newspapers.
Even the Armenians are not encouraged to develop the resources of the country. The
following question which I addressed to a prominent Armenian capitalist may exhibit,
together with the answer, the magnitude of those resources and the reasons assigned for
the fact that they are not exploited.

Q. “Can you explain to me why so little use is made of your natural advantages—the
immense extent of idle soil and the abundance of water? In the north you have the vast
reservoir of Lake Sevan; in the south the Araxes, running in full stream to the Caspian
Sea. Cultivation might surely be increased to many times its present area without any
great expense.”

A. “The waste lands are for the most part in the hands of the Russian Government, and
they are not inclined to sell or lease them to Armenians. They are believed to be keeping
them for Russians, but the Russians do not come. A successful piece of reclamation has
been made by General Cheremetieff in the neighbourhood of Ararat. We have made
repeated proposals to take lands and irrigate them, but we have never been able to obtain
permission.”

Perhaps, if these lines come to the eyes of M. Witte, he will give the matter the attention
which it deserves.

The same exclusive economical policy, as manifested in protective duties, has deflected
commerce from the natural avenue of the valley of the Araxes, and caused it to pursue
more lengthy and less convenient routes. There is scarcely any transit trade with Persia.
The prosperity of the place is therefore dependent on native industries, which comprise
the cultivation and export of cotton, wine and rice. Cotton to the value of about £400,000
is annually despatched by waggon or camel to the station of Akstafa on the Tiflis
railway, and thence, viâ Batum and the Black Sea or Baku and the Caspian, to the
manufacturing centres of Russia. Three large Russian firms are locally represented by
offices and factories, where the cotton is purchased and cleaned and pressed. The
presses, which are of English make, are driven by horse power. While this industry is in
the hands of Russians the trade in wine is conducted by Armenians; and very excellent
wine have they succeeded in producing. The value of the yearly export, which goes
exclusively to Russia, is as yet only £20,000. But the enterprise of M. Karapet Afrikean,
who has closely studied his subject in Germany, has already effected a marked
improvement in the quality of the wine, and is likely to lead to a great increase in the
demand. Rice is also exported and in considerable quantities to Erzerum and the Turkish
provinces. The fruits of Erivan are almost unrivalled in the world; but I do not know that
they are preserved and sent away.

Such is the city which, with its vast and populous province, absorbs all the time and all
the energies of its Russian governor, sitting at his green baize table overlooking the park.
General Frese has a real affection for that table, which he has shaped to fit his figure.
From early morning to late night his erect and military form is condemned to that
inactive but rigid posture. He never indulges in the relaxation of an arm-chair. While you
puff your cigarette among his hospitable cushions, he will discourse upon the mighty
rivers and forests of Siberia from across the field of green baize. Dinner is served in a
room displaying all the skill of Persian artists, and overlooking, through a window
composed of tiny panes of glass, a miniature garden disposed as for the stage of a
theatre. I need hardly say that this work of fancy was not created by the order of the
present occupant of Government House. Still the fare at his table is worthy of the most
refined palate; such excellent trout and tender chickens and the pick of the native wine!
Immediately after the meal he resumes his seat in the adjoining room behind the green
baize. He attributes the backwardness of the country to excessive centralisation at St.
Petersburg, a process which has been tending to assume increasing proportions now that
the Caucasus is no longer administered by a Grand Duke.
1 According to the Jesuit, Père Monier, who wrote an account of the mission at Erivan in the eighteenth
century, there were only 4000 inhabitants of the town proper in his day. Of these only one-fourth were
Armenians (Lettres Édifiantes, Mémoires du Levant, Paris, 1780, vol. iii. p. 25). In the thirties of last
century the usual estimate seems to have been 2500 families or at least 10,000 souls, of whom some 700 to
1000 families were Armenian (Smith and Dwight, Missionary Researches, p. 279; Sijalski, Aufenthalt in
Erivan, Das Ausland, Augsburg, 1839). The Armenians are rapidly turning the tables upon the Tartars. ↑
2 Chardin, edit. Paris, 1811, vol. ii. p. 169. ↑
3 “Erivân, apparens, quia regio ista prima apparuit Noe cum descenderet ex monte Ararat” (Villotte,
Dict. Arm. p. 273, quoted by Langlès ap. Chardin, loc. cit.). ↑
4 Moses of Khorene, vol. ii. p. 46. ↑
5 Lane Poole, Mohammedan Dynasties, London, 1894, p. 259. ↑
6 For the Mohammedan tradition see Travels of Evliya, translated by Von Hammer, London, 1850, vol.
ii. p. 150. “In the year 810 (A.D. 1407) Khoja Khan Lejchani, a rich merchant of Timur’s suite, settled
here (at Erivan) with all his family and servants, cultivating plantations of rice, by which means a great
Kent was soon formed. Five years later Shah Ismail gave to Revan Kul, one of his khans, an order to build
a castle here, which, being finished in seven years, was named after him Revan or Erivan.” The five years
of Evliya are incomprehensible to me. Erivan is mentioned by John Katholikos, who wrote in the eleventh
century, as having been a considerable place in the seventh (Saint-Martin’s translation, Paris, 1841, p.
80). ↑
7 Dubois de Montpéreux, Voyage autour du Caucase, Paris, 1839, vol. iii. pp. 346 seq. When Morier,
secretary to the British Embassy to Persia, visited the sirdar or governor of Erivan in 1814, he was told by
his host with great gravity that “if three or four of the kings of Fireng (Europe) were to unite to take this
castle, they might just take the trouble of going back again, for their labours would be in vain” (Morier,
Second Journey, London, 1818, p. 319). The sirdar’s view was not held by British officers, one of whom,
in giving an account of his visit in 1837, says, “I had expected to find the castle almost impregnable from
the honours which were heaped upon the Marshal Paskevich for its capture, and was quite surprised to find
a mere Turkish fort, strong indeed by nature on one side, but on the other three defended merely by a mud
wall, and commanded from all the adjoining hills” (Wilbraham, Travels in the Transcaucasian Provinces,
etc., London, 1839). ↑
8 “In dieser abermahligen Veränderung seynd auch alle Türkische Moscheen der Stadt übern Hauffen
geworffen ... also das etliche dergleichen Tempel bis zum Fundament erniedriget und übel ärger von
Persianen verwüstet als jemahl die Kirchen der Christen von Türcken zugerichtet worden seynd. So
züchtiget Gott die Mahumetaner mit Mahumetanern” (Schillinger, Persianische und Ost-Indianische Reise
vom Jahr 1699 bis 1702, Nürnberg, 1707). ↑
9 Tavernier, edit. of Paris, 1679, vol. i. p. 37; Père Monier, op. cit. vol. iii. p. 24. ↑
10 Von Hammer, Geschichte des Osm. Reiches, vol. vii. p. 321. ↑
11 Morier, Second Journey, p. 320. ↑
12 Dubois de Montpéreux, op. cit. vol. iii p. 452. ↑
13 Dubois, ibid. pp. 339 seq. and Atlas. ↑
14 Dubois, ibid. p. 346, and Morier, Hajji Baba. ↑
15 Chapter viii. of the Polojenye of 1836. ↑
16 I was informed by a competent authority that, including Tiflis and the whole of Russian
Transcaucasia, there were not less than 400 Armenian schools in existence at the time of my visit. About
one-third of the number would be schools for girls. ↑
17 Müller-Simonis (Du Caucase au Golfe Persique, Paris, 1892, p. 62), speaking of the celebration of
the ceremonies in honour of Ali and Hoseyn at Erivan, says: “Le soir les fanatiques qui devront représenter
les martyres à la grande procession, font une promenade aux flambeaux, armés de sabres et de gourdins.
Ils agitent en mesure leurs flambeaux et leurs armes, criant en même temps à tue-tête: ‘Hussein, Ali,
Hussein, Ali.’ Les reflets rouges des torches, ici découpant les blanches silhouettes des maisons, là
plongeant en lueurs étranges sous la verdure des arbres, puis éclairant en plein les figures hideuses de ces
dévots, forment un spectacle sauvage et fantastique.” The picture is true to life. I have little doubt that such
a procession may still be witnessed at Erivan. ↑
CHAPTER XVI
 EDGMIATSIN AND THE ARMENIAN CHURCH
At five o’clock in the afternoon of the 4th of October we set out for Edgmiatsin. It is a drive
of about thirteen miles across the plain. Our luggage was consigned to a waggon of the post,
and we ourselves enjoyed the luxury of a light victoria, drawn by four horses abreast. They
covered the distance in an hour and forty minutes, although the road is in many places a
mere track.

What a drive! It is so well within reach of Europe that it ought to be included, like the
journey to Italy, in the programme of a liberal education. The railway will before long arrive
at Erivan, and then the pilgrimage will be still easier to undertake. Not all the tourists in the
world will disturb the harmony of this landscape; the screeching trains, the loud hotels, the
Babel of tongues will be lost, like a flight of starlings, in this expanse. It is here that the
spirit of Asia is most intensely present—an inner sanctuary to those outer courts through
which the traveller may have wandered and never crossed the threshold of this plain. And it
is a spirit and an influence which arouse deep chords within us and send them sounding
through our lives.

The landscape at once combines and accentuates the salient features of the Asiatic highlands.
There is the plain which was once the bed of an inland sea. It stretches west and east without
visible limits; and this evening it has all the appearance of water. In the west it is mirage
which produces this effect. The long north-western slope of the Ararat fabric assumes the
character of a dark and narrow promontory rising on an opposite shore. From the east,
beyond the train of the Little Ararat, a cold mist—may it be from the Caspian?—is slowly
wafted over the steppe, and the illusion is complete. Into those liquid spaces sweep the basal
vaultings of Alagöz—the boulder-strewn declivities which we keep on our right hand, and
which seem to embody on a typical scale that quality of hopeless sterility which is
characteristic of vast portions of the continent. But the same vague distance receives the
Zanga, diffused into many channels, and lost beneath luxuriant foliage. For over a quarter of
an hour after leaving Erivan we pass at a rapid trot between the walls of orchards; and in
places the water gushes from the conduits across the road. Once outside this intricate zone
the track wanders over the idle soil, skirting the stony slopes in the north. In the opposite
direction the plain blooms with fields of cotton and rice, sustained by a small canal which
pursues a westerly course before it falls into the Araxes, if indeed it flow so far.

And there are the mountains of Asia—the volcanoes with their vaulted summits, as well as
those long ridges with their serrated outline which represent the operation of less impetuous
forces through longer spaces of time. To this second category belongs the fine chain on the
west of Ararat which gains in definition as we proceed. It stands a little back and behind the
fabric of Ararat, and volcanoes too have built themselves up upon this wall. But its rugged
and tumbled appearance is the feature which predominates, in striking contrast to the
symmetry of the mountain of the Ark. That giant overpowers the lesser Ararat and
appropriates their common base. One stands in wonder at the force which could have rent
that massive pedestal and opened the yawning chasm which fronts the plain. Night creeps
into those recesses, where the blaze of a Kurdish camp-fire calls attention to the
extraordinary transparence of the air. The snow-fields, bare and cold above the amber of the
sunset, are already free of their coronal of cloud. One full-puffed vapour still floats behind
the uppermost pinnacle; another clings to the bastion on the north-west. While we admire
this stately scene, made more impressive by the heavy silence, a grove of trees rises from the
steppe on our point of course. Two little conical shapes just emerge above their outline, and
are recognised as the domes of Edgmiatsin.

We pass through the thin plantation, sustained by runnels derived from Alagöz, and come to
a halt before the doorway of a lofty mud wall with round towers at intervals. It might belong
to a Persian fortress; but it is the outer wall which surrounds the cloister with the cathedral
of St. Gregory. The massive gate is closed, and we thump and thump for some time in vain.
The parapet with its crumbling surface betrays no sign of the life within. But there is just
sufficient light to reveal the surroundings of the fortified enclosure—a straggling village of
above-ground houses, outlying churches, poplars, dust.1

Fig. 47. Pilgrims’ Court, Edgmiatsin.

At last the hinges creak and the porter appears. We are ushered into a court, like that of a
college at Cambridge, adjoining the great gate which is in the south wall. It is known as the
pilgrims’ court (Fig. 47). Low buildings, rudely built, with a continuous wooden verandah,
compose the quadrangle. The windows are all lit up behind a line of young trees of which