Search Relevance: Standard Operating Procedures

WPX Search Relevance SOP (Version 3.2)

2 l
Contents
WPX Search Relevance SOP (Version 3.2).......................................................................................................................... 1

56 tia
Change Log: ........................................................................................................................................................................ 2
Task Overview .................................................................................................................................................................... 3
Illustrative Examples ...................................................................................................................................................... 4
Assessing the Relevance Match Labels .............................................................................................................................. 5
Product Attributes and Impact of Mismatches .............................................................................................................. 6

84 n
Additional Important Considerations .......................................................................................................................... 12

eProduct Variations from Product Detail Page .......................................................................................................... 12

Ambiguous Queries.................................................................................................................................................. 13
Misspelled Queries .................................................................................................................................................. 14
Broad Queries .......................................................................................................................................................... 14
22 id
In-Category Search ................................................................................................................................................... 14
Very Vague Queries ................................................................................................................................................. 14
Products Not Sold on Amazon ................................................................................................................................. 15
Labeled Examples ............................................................................................................................................................. 17
nf

Appendix .......................................................................................................................................................................... 20
Examples of Applying the Labeling Process ................................................................................................................. 20
Example: pickup tents for camping short bed.......................................................................................................... 20
Example: bobs shoes with dogs on them ................................................................................................................. 23
Co

Working with Categories .............................................................................................................................................. 25

Category specific examples .......................................................................................................................................... 27
1. Electronics .................................................................................................................................................... 27
2. Media and Games ........................................................................................................................................ 27
3. Fashion ......................................................................................................................................................... 29
4. Luggage and bags ......................................................................................................................................... 32
5. DIY & Tools ................................................................................................................................................... 33
6. Beauty .......................................................................................................................................................... 34
7. Medicines ..................................................................................................................................................... 35
8. Grocery......................................................................................................................................................... 35

1|Page
Search Relevance: Standard Operating Procedures

9. Fresh……………………………………………………………………………………………………………………………………………………….39
10. Pets…………………………………………………………………………………………………………………………………………………………41
11. Toys .............................................................................................................................................................. 41
12. Home ............................................................................................................................................................ 42

2 l
13. Kitchen ......................................................................................................................................................... 42

56 tia
14. Automobiles ................................................................................................................................................. 42
15. Shoes ............................................................................................................................................................ 42
16. Theme .......................................................................................................................................................... 43
17. Adult ............................................................................................................................................................. 43
18. Books ............................................................................................................................................................ 43

84 ne19.
20.

Change Log:
Brand affinity ............................................................................................................................................... 47
Subscription ................................................................................................................................................. 48
More Labeling Nuances.................................................................................................................................................... 49
22 id
Change description Page # Date changed/updated Sign off by
*Size - rule updated 10
*Genre - updated example 10
*Product Attributes – ‘Other Attributes’ - Added 11
nf

example
*Additional important consideration –
‘Ambiguous queries’ - explanation updated 13
* Additional important consideration –
‘Incategory Search’ - Example updated 14 1/12/23 Arindam Routh
* Additional important consideration – ‘Very
Vague queries - Added example 15
Co

* Additional important consideration –

‘Unjudgeable’ - explanation added for foreign 16
language
* Category specific examples- ‘Media & Games’- 27
Video games: Added example
* Category specific examples- ‘Media & Games’- 27
Movies/Prime video: Complementary section
updated
* Category specific examples- ‘Fashion’- Softlines: 29
Complementary section updated
* Category specific examples- ‘Beauty’- Bodycare 34
& products: Added example
* Category specific examples- ‘Pets’: updated 41
example

2|Page
Search Relevance: Standard Operating Procedures

* Category specific examples-Toys: updated 41

example

* New category addition of Brand affinity 47-48

* New category addition of Subscription with 48 3/20/23

2 l
revised examples and decisions Arindam Routh
*’More labelling nuances’ section updated with 49-50

56 tia
examples for Irrelevant
* Additional important consideration – ‘Product 15
not sold on Amazon’ – New example added
*Books SOP updated
 Target audience 44-47 4/3/23 Arindam Routh/ Mattia
 Language difference Serrano
 Complementarity in Books

84 n
Task Overview
e
The goal of the task is to assess the relevance of the search results returned to the Amazon customers. Each task will have a
search query and an e-commerce product as the input, and the annotator will provide their judgement on the type of a relevance
22 id
match between the query and the product. The context of the search query (SQ) is specified as

1. Search query/keywords, e.g., red nike shoes

2. Search category, e.g., aps (stands for All Products Search)
3. Marketplace where the search takes place, e.g., amazon.com.
nf

In addition to the above, the judges will be provided links to the Google and Amazon search engine results for the queries as well
as Google search engine results restricted to Amazon retail site.

The information about the offered product (OP) consists of

1. Product title,
2. Product image,
Co

3. Current/live product detail page (PDP) accessible via a link in the user interface (UI). (This is usually done by providing
the Amazon Standard Identification Number, or ASIN, for the offered product.)

The annotators will determine and mark the most likely type of the relevance match for the query-ASIN pair from the perspective
of a reasonable Amazon customer. The high level description of the types of Relevance Match is provided below.

Match Type Description

Exact The offered product fully satisfies the intent of the query, matches on all intended product attributes.

The offered product does not fully match the intent of the query, but serves the same purpose as the intended
Substitute
product and could be used instead to address the customer’s need.

3|Page
Search Relevance: Standard Operating Procedures

The offered product does not serve the purpose of the intended product, but could be of interest to the customer
Complement either because it could be used or worn together with the intended product, or because the intended and the
offered products are related by brand, genre, theme, occasion, or a similar non-functional characteristic.

The offered product is either entirely unrelated to the intended product or is unsuitable for the customer due to a

2 l
Irrelevant
critical to the intent of the query mismatch on a product attribute.

56 tia
The relevance match relationship cannot be assessed because the query cannot be understood, the query has a
Unjudgeable non-product intent, or the information about the offered product is not available or not sufficient to determine if it
matches the query.

Illustrative Examples

84 n
Query and Offered Product

Query: red hanes cotton tshirt

e
Category: aps (All Products)
Marketplace: amazon.com

Product Title: Hanes T-Shirt,

Product Image Judgement

Exact

The offered product matches the query on

all intended product attributes: product
type (shirt), color (red), brand (Hanes),
material (cotton), shirt type (t-shirt).
22 id
Beefy Classic Crewneck Cotton
Tee
ASIN: B00WQMX5G6
nf

Query: panasonic waterproof Substitute

electric shaver
Co

The offered product serves the same

Category: aps (All Products) purpose as the intended product (shaves),
Marketplace: amazon.com matches on the product type (shaver),
feature (waterproof), another feature
Product Title: Braun Electric (electric), but mismatches on brand
Razor for Men, Waterproof Foil, (Panasonic intended, but Braun
Shaver, Series 7 7027cs, Wet & suggested).
Dry Shave, With Beard Trimmer,
Rechargeable, Charging Stand
and Travel Case Included, Silver
ASIN: B07Z8P11Y5

4|Page
Search Relevance: Standard Operating Procedures

Query: samsung galaxy z fold 4 Complement

Category: aps (All Products) The offered product is designed to be

Marketplace: amazon.com used together with the intended product,
has a product type (smartphone case)

2 l
Product Title: SUPCASE Unicorn which is complementary to the intended
Beetle Pro Case for Samsung product type (smartphone) and is

56 tia
Galaxy Z Fold 4 5G (2022), Full- compatible with the intended product
Body Dual Layer Rugged Case model (Samsung Galaxy Z Fold 4).
with Built-In Screen Protector &
Kickstand & S Pen Slot (Black)
ASIN: B0B7MC26MS

84 n
Query: nike sneakers for men

Category: aps (All Products)

e
Marketplace: amazon.com

Product Title: Nike Women’s Flex

Trainer 4 Sneaker
Irrelevant

While the offered product matches the

intended product on the product type
(shoes), shoe type (sneaker), and brand
(nike), the offered product fails to fulfill
the purpose of the intended product due
22 id
ASIN: B07HDS7TVT to a gender mismatch (as shoes designed
for women typically have a different width
from those designed for men).

Query: my past purchases Unjudgeable

nf

Category: aps (All Products) The query has a help intent, not a product
Marketplace: amazon.com intent.

Product Title: JUST MY SIZE

Women’s Apparel Women’s Plus
Size 2 Pocket Pull on Short,
Co

Indigo, 2X
ASIN: B075LSDMDW

Assessing the Relevance Match Labels

While providing the labels for many of the tasks will be intuitive and straightforward, it is strongly suggested to follow the
following basic steps when assessing the relevance, especially when encountering non-trivial instances or when training.

5|Page
Search Relevance: Standard Operating Procedures

1. Understand the query: research and identify the intended product type, the broader purpose of the product, and the
desired product attributes; use Google or Amazon search results for the provided queries, and collect additional
information from third party sites if needed. If the query cannot be understood or interpreted or does not have a
product intent, then assign Unjudgeable label and stop.
2. Understand the ASIN: research and identify the offered product’s attributes corresponding to those identified in the

2 l
query; use the product title, image, information on the PDP (product detail page) including the product description, and
in the rare cases of inconsistency among the product images, title, and description, use the information in the product

56 tia
reviews and Q&As. If the product cannot be understood, then assign Unjudgeable label and stop.
3. Identify the product attribute matches and mismatches: for each of the understood product attributes in the query,
determine whether query and the offered product agree, and take a mental inventory of the attribute matches and the
attribute mismatches (the latter also referred to as defects).
4. Assign the label based on the identified defects: If all of the attributes match, then assign Exact label. If the intended
purpose is preserved and none of the defects make the product unsuitable, then assign Substitute label. If the offered
and intended products can be used together or are of the same brand/genre/theme/occasion and product category, and
if none of the defects make the offered product unsuitable, then assign Complement label. Otherwise, assign Irrelevant
label. See Product Attributes and Impact of Mismatches.

84 n
Examples of Applying the Labeling Process contains a few illustrations of applying the process above.

Product Attributes and Impact of Mismatches

e
E-commerce search queries contain mainly the information about the attributes of the desired product (referred to as DP
throughout this section). The table below lists the most common types of product attributes that the judges are likely to
encounter together with how the mismatch on the attribute would impact the assigned relevance match label.
22 id
Important note: any critical attribute defect makes the offered product unsuitable for the intent and leads to Irrelevant label. It
would take only one such defect among the attributes to lead to Irrelevant label, regardless of the matches on other
attributes. For example, for a query nike size 9 black shoes for men, a pair of size 9 black nike women’s shoes would be Irrelevant
despite matching on the product type, brand, color, and size.
nf

Relationship
Attribute Definition Mismatch Examples
Type
Purpose is the minimum viable SQ: mens belt
shopper’s need as described or implied OP: belt sander
by the query. Explanation: The purpose
is to keep the pants in Irrelevant
place, not to shape wood.
Co

When assessing whether the OP serves

the same purpose as the DP, consider Exception: OP
The OP is unrelated
whether the task which the DP is SQ: office desk and DP are of
to the purpose of DP.
designed for can be accomplished by OP: coffee table the same brand
This implies that OP
the OP. neither fulfills the Explanation: The purpose (see Brand
is to acquire a surface Affinity related
Purpose purpose of DP nor
Examples: can be used as an capable of supporting the products and
Brand Affinity
- [coffee grinder] refers to a product accessory to DP (see activities for office work
(e.g., writing, reading, unrelated
with the utility of grinding coffee Lead product vs
computer work). A coffee products),
beans into a coarse powder. Other accessory).
table’s purpose is to genre, or
devices able to do this, e.g., theme/occasio
support beverages,
multipurpose crusher, would serve n.
remote controls,
the same purpose.
decorative objects, and
- [acne cream] refers to a product small items (wiki), not
with the utility of reducing the designed for office work.

6|Page
Search Relevance: Standard Operating Procedures

symptoms of acne. There are

products not specializing in acne
treatments but which nonetheless
can be used for that purpose, e.g.,
tea tree oil.

2 l
SQ: Migrain relief pill
What is a product type?
OP: Headache & Migraine

56 tia
OP does not match relief hat
In SQ, product type is the term or the product type of
phrase that captures the idea of the DP but can be used to Substitute
product the customer is looking for. In SQ: treadmill
satisfy the same
an item, it is a class of common generic utility/purpose as DP. OP: Elliptical Machine
goods or services. Amazon catalog Explanation: Both are
would contain hundreds or thousands cardio fitness equipment
of products of the same product type. If SQ: 13-inch MacBook Air

84 ne a term in the query or a product is

covered by another product attribute, it
should not be a part of the product
type.

Examples in queries:
- Simple: [popcorn], [4 ft rug square]
- Keeping it minimal: [dri fit t shirts for
men], [mens black loafers dress
OP is an accessory to
DP.

OP is either an
independent
interdependent
complementary
or an
laptop
OP: Carrying case
compatible with a 13-inch
MacBook Air laptop
SQ: office desk
OP: office chair

SQ: shower curtain

Complement

Complement
22 id
product to DP. OP: shower curtain rod
shoes], but [raised garden bed]
- Can refer to an accessory to another DP is an accessory,
product: [charger for dell laptop] and OP is a different SQ: iphone 13 case
Irrelevant
accessory to the lead OP: iphone Charger
Product product.
Type SQ: iphone 13 case
nf

Lead product vs accessory OP is the lead

OP: iphone 13 Irrelevant
product to the DP.
smartphone
Some products are commonly used SQ: Electric Guitar
together but can be either sold OP is a bundle, and
OP: Electric Guitar, Exact
separately or together, e.g, a Quest 2 DP is its lead product.
amplifier, stand, cover
Pro Bundle (ASIN: B09DCM6CNZ)
consists of a virtual reality (VR) headset, OP is a bundle, and
Co

an improved headstrap, and a carrying DP is part of this

bundle with another SQ: Shampoo
case.
either inter- or OP: Shampoo + Substitute
- If a product can fulfill its basic
independent Conditioner
purpose on its own without
complementary
depending on other product, it is product.
considered a lead product (the VR
OP is a bundle with a SQ: pastel paper pad
headset).
lot of items, and DP is OP: 54-piece drawing and Irrelevant
- An accessory is a product used to
just one of the items. sketching art set
enhance the lead product, but
OP is a bundle, and
cannot function on its own (the SQ: ipad cover
DP is an accessory in Irrelevant
headstrap and the carrying case). OP: ipad with a cover
this bundle.
- Sometimes the products are
DP is a bundle, and SQ: scanner with scanner
interdependent in order to function
OP product is the cover Substitute
(e.g., camera body and lens) in which
lead product in it. OP: scanner

7|Page
Search Relevance: Standard Operating Procedures

case the items are complementary,

but neither is a lead product or an
SQ: printer and scanner
accessory.
OP: scanner
- Similarly, products can function DP is a bundle, and
independently, (e.g., shampoo and OP is in it but not a Complement

2 l
SQ: scanner with scanner
conditioner combo in separate lead product.
cover
bottles) in which case they are also

56 tia
OP: scanner cover
complementary without either
being an accessory or a lead product.
The brand of OP is
not the same as that Query: Nike shoes
of DP, but the Offered Product: Puma Substitute
By brand we refer to a proprietary products have the Shoes
language, licensed trademark, same functionality.

84 ne manufacturer, or a retailer associated

with the intended product. Brands can
also market series of products under a
different name referred to as a product
line. We won’t distinguish the two for
the purpose of defects.

Examples of brands (and possibly,

Brand Affinity,
related products
The brand of OP is
the same as that of
DP, and the products
are neither
functionally
SQ: Dove shampoo
OP: Dove deodorant
Explanation: Both are
personal care items
sharing the brand

SQ: yogalicious biker

shorts
Complement
22 id
Brand or product lines) in queries: [nike jordans equivalent nor OP: yogalicious fleece
Trademark shoes men], [garmin fenix 5], [almay complementary, but robe
concealer] are from the same Explanation: Both are
product category. clothing items sharing the
Shoppers build a connection based on brand
their experience with the branded Brand Affinity,
products and develop an affinity
nf

unrelated products
towards them. This affinity can The brand of OP is
potentially make the offered product the same as that of
relevant to the shopper even if the SQ: Samsung laptop
DP, the products are
product does not have the intended OP: Samsung tv remote Irrelevant
neither functionally
functionality. control
equivalent nor
complementary, and
Co

do not have a related

purpose.
Model usually refers to a smaller class
of products within a product line. The
class could be further restricted to a The specific model of
Model/Mo specific product with the model OP is not the same as
SQ: Samsung S7
del number. that of DP, but the Substitute
OP: Samsung S10
Number products offer the
Examples of models and model same functionality.
numbers in queries: [logitech k860],
[samsung galaxy z flip 4]
DP is an accessory to SQ: Charging cable pixel
Because of the specific design of the
Compatibil a lead product. OP is phone
products, the accessories for them Irrelevant
ity the same type of OP: mini-usb charging
sometimes need to be compatible, i.e.,
accessory, but is not cable

8|Page
Search Relevance: Standard Operating Procedures

specialized to be used with the lead compatible with the Explanation: Pixel phones
product. that lead product. use USB-C cables

Note: sometimes the

products are marked

2 l
as universal meaning
that they are

56 tia
compatible with all of
the products of a
particular type/class.
SQ: books for a 6-year old
OP: book for a first-grader
OP overlaps on the
Explanation: 1st grade Exact
age range with DP.
typically means 6-7 years
old

84 ne OP is +/-1 age range

from DP. If the size is
specified by age, e.g.,
3-6 months baby
shoes, then one
range apart, e.g., 0-3
months or 6-9
months baby.
22 id
The age attribute indicates an age range Note: +/-1 age range
intended or suitable for a product. is a high-level
guidance as age SQ: shoes for an infant
Examples in queries: [board games for a ranges are
starting to walk Substitute
3-year old], [outfits for a 6-month old] sometimes difficult to
Age OP: shoes for a toddler
nf

establish. Try
Note: when the age group is used in a answering a
clothing related query, it refers to the question: is OP
clothing size. Same mismatch guidance potentially suitable
applies. for the age range of
DP? If the age range
is not matching
Co

perfectly, but the

product is still likely
to be suitable, then
the product is a
Substitute.
OP is unlikely to be
suitable for the age
range of DP (general SQ: toys for toddlers
guidance: more than OP: toys for a 9-14 year Irrelevant
one age range old boy
difference from the
intended).
Gender attribute refers to the gender SQ: Women’s pants
OP is not designed
Gender that the intended product is designed or OP: Men’s pants Irrelevant
for the explicitly
styled for, one of male, female, or

9|Page
Search Relevance: Standard Operating Procedures

unisex (suitable for either female or expressed gender in

male shopper). DP.

Note: intended gender is limited to

human beings only.

2 l
The intended species attribute captures
what species of pets/pests the product

56 tia
OP is not designed to SQ: Dry food for dogs
Intended is designed for/against.
work for the species OP: Food pellets for Irrelevant
Species
in DP. rabbits
Examples in queries: [flea medicine for
cats], [dog leash], [mosquito repellant]
The colors of OP do
Color attribute refers to a significant SQ: Red bag
Color not match the color Substitute
color in product appearance. OP: Black bag
description of DP.

84 ne OP is not the same

size as DP but differs
in up to +/-1 size unit.
This applies to
clothing and shoes
only. (no longer
Size refers to physical dimensions of the applies to table,
product, be it in values with or without towel, blanket as
units, adjectives, or product specific these of any size can
SQ: Boots size 6
OP: Boots size 7 Substitute
22 id
descriptions. be used)
OP differs by more
Examples in queries: [large litter box], than +/-1 size unit
Size
[32” tv], [mens dress shoes size 9] from DP.
Alternatively, OP is of
Since different brands may size label a different size than SQ: 34” mens levis jeans
nf

fluctuations (primarily for clothing), we DP, and DP is OP: 32” mens jeans
consider providing some leeway in the designed to fit
Irrelevant
labeling. another product SQ: king size bedsheet set
(e.g., cushion cover, OP: queen size bedsheet
pillow cover, fitted set
sheet, duvet cover,
bed sheet, mattress
Co

or mattress cover,
curtain).
OP does not refer to
the same entity but is
in the same or
Genre refers to classes of creative work overlapping genre
in media. Other non-media items can with the entity for
SQ: batman tshirt
be themed with entities who would DP. This rule can Substitute
Genre OP: Any type of tshirt
have genre associated with them, e.g., refer not just to
harry potter action figures, frozen- media item, but to
themed merchandize. items which use the
likeness of media
characters.
Genre affinity SQ: jazz music Complement

10 | P a g e
Search Relevance: Standard Operating Procedures

OP is a media OP: Jazz Dance Level 1

product of the same video
genre as DP, but is of
a different media
type.

2 l
Theme is an idea or point central to the SQ: Halloween wreath
appearance of or association with the OP does not match OP: Christmas wreath

56 tia
product. Theme could be associated the broad theme or Irrelevant
with media characters or franchises occasion of DP. SQ: Cricket jersey
(e.g., Avengers), sports (e.g., cricket), OP: Football jersey
Occasion/T
visuals (e.g., images of cats), or
heme Occasion/Theme
occasion/event (e.g., wedding,
Christmas). Theme provides a affinity SQ: 4th of July mugs
connection between the products, with OP is not a functional OP: July 4th American Flag Complement
customer potentially interested in other substitute for DP but Banners Garlands
matches its theme.

84 n
Sensitive
e
Content
products with the same theme.
Likeness of the product to religious
symbols or references to religious texts,
and medical or dietary supplements
context of use of the product is sensitive
to the customer. Not matching on this
content makes OP unsuitable.
OP does not match
DP on sensitive
content.
SQ: Buddhist necklace
OP: Love Cross necklace

SQ: back pain

OP: Medicine primarily for
headache
Irrelevant
22 id
SQ: waterproof shoes
OP: shoes (not
waterproof)
Substitute
Label: Substitute
Explanation: customers
might still consider other
nf

sturdy shoes which are

not designed as
waterproof.

SQ: cotton shirt

This is a catch all rule for the remaining OP: shirt (made of less
attributes, e.g., style, feature, material. than 50% cotton)
OP has a different
Co

If the mismatch in the attribute changes

Other style, feature, Label: Substitute
the function of the product, the
Attributes material, etc. than Explanation: customers Substitute
resulting defect would be critical, and
that of DP. might still consider the
the resulting label would be Irrelevant.
product if other
Otherwise, it would be Substitute
characteristics are
satisfied the product is
otherwise attractive (e.g.,
price).

SQ: Kettle for 110v

OP: Kettle 210v
Label: Irrelevant
Explanation: Customer is
specific for a
compatibility/feature of

11 | P a g e
Search Relevance: Standard Operating Procedures

product searched. Voltage

difference could result in
short circuits or result in
damaging of products. Irrelevant
Voltages are also very

2 l
specific to countries and
states. Thus, the defect in

56 tia
the queries is to be
treated as incompatibility
differences and rated as
Irrelevant.

Additional Important Considerations

84 n
Product Variations from Product Detail Page
When customers search for products which have variants of the same model or product line, instead of returning all of such
variants in the results, Amazon product search UI returns only one representative product for all such variants. The customer is
e
given a choice to select the desired variant either from the ASIN impression in the search results or from the product detail page.
The judges are expected to check the product detail page for available variants when making their assessment and to select the
variant which matches the query the closest.
Example
22 id
Keywords: brooks running shoes black
Category: aps (All Products)
Marketplace: amazon.com
Product Title: Brooks Ghost 14 Men’s Neutral Running Shoe
ASIN: B08QVGSNCR
nf

The search result page lets the customer select some of the color variants for each returned model of shoes. The product detail
page allows the customer to select both the color pattern and shoe size. The product image and other information available in the
task UI is unlikely to provide the information about all of the product variant choices available to the customer. The judges are
expected to check the product detail page for available variants when making their assessment and to select the variant which
matches the query the closest following the sequence Exact, Substitute, Complement, Irrelevant in that order. For this search
query, the faceout variant matches on all attributes and is an Exact match. Searches for brooks running shoes grey or brooks
running shoes orange would both be Exact matches for the ASIN below since both colors can be selected from the PDP, but for a
Co

search for brooks running shoes gold it would be a Substitute since no variant selections have gold as the primary color.

12 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
84 ne
22 id
nf
Co

Ambiguous Queries
A query is considered ambiguous if it has multiple possible interpretations. For example, a query with keywords glasses could
refer to sunglasses or reading glasses, or to drinking glasses. The annotators can mark all these types of glasses as Exact. For
example, SQ is query cat pajamas and OP is pajamas with cat images on them, pajama costume of a cat, or pajamas for cats
would all be marked Exact. For example, SQ is reading glasses and OP is sunglasses then it would be marked Irrelevant as the
feature is crucial and is defeated.
“Annotators is required to mark the query as ‘Ambiguous’ by adding additional comment in SDAT without fail when they notice
the query/ASIN is ambiguous in nature”.

13 | P a g e
Search Relevance: Standard Operating Procedures

Misspelled Queries
For misspelled queries, use the intended spelling when assessing the relevance match type. For example, query bumber sticker
very likely refers to bumper sticker. If there is a product interpretation for the query without a spelling correction, give the
customer the benefit of the doubt and use the unmodified spelling. For example, cotreau square drinking glasses orren ellis
refers to a product not sold on Amazon as of this writing (link), and the query should not be spell corrected despite Amazon’s

2 l
search engine doing so (as indicated by the preamble below shown on the search page).

56 tia
84 n
If there are several plausible spelling corrections depending upon which letters are substituted, pick one that will result in the
e
closest relevance match label, following the same guideline as for Ambiguous Queries. If the query is unclear, consider all
common misspellings as different interpretations for the query, you can use Google or Amazon search to identify common spell
corrections.
22 id
Broad Queries
Some of the queries would be exploratory in nature, providing only a basic information about the product, typically, either just a
category, a product type, or a brand. These broad types of queries could potentially have thousands of relevant matches, and any
of the products matching the intent are considered Exact matches. Some broad queries may not admit Substitute or
Complement labels since the function is very broad to establish Substitutes, and not enough information about the product to
establish compatibility relationship.
nf

For example, any jewelry item would be considered an exact match for query with query jewelry. Any product made by Reebok
would be considered an Exact match for a query reebok.

In-Category Search
Most queries will reference All Products search category, meaning that the customer issuing the query did not specify a product
search category. (All Products is the default, and unless otherwise mentioned, assume that the queries refer to All Products.)
Co

When a search category is specified, it can be utilized to disambiguate and focus the intent of the query.

Example: Marvel is a broad query referring to products related to the media franchise spawned by a series of Marvel comics.
Absent additional information, the query can refer to comic books and books, movies, games, toys, clothing, and other items
branded with the images and likeness of the characters and the items from the franchise. However, if the query references Toys
& Games product category, the query intends the product type to be toys or figurines (eg: SQ is Marvel & OP is a spider-man lego
toy – exact, Batman lego toy – Substitute; Captain America t-shirt – Complement; a green lantern t-shirt will be Irrelevant)

Exception: When the intent based on the search keywords are inconsistent with the provided category, ignore the provided
category and assume All Products, e.g., jackets with category Electronics.

Very Vague Queries

Occasionally, a customer would enter either an incomplete query or one so ambiguous that it is not clear what product it might
be referring to. These queries would typically be with All Products Category as knowing the category can sometimes be used to

14 | P a g e
Search Relevance: Standard Operating Procedures

disambiguate such queries. If it is not possible to infer a likely interpretation, mark it as Unjudgeable as the labels collected
under unlikely interpretations are not useful in the downstream applications.

Examples:

2 l
Keywords: waterproof
Category: aps (All Products)

56 tia
Judgement: Unjudgeable
Explanation: Waterproof very likely refers to a property of the product, but it is not clear what product is being referred to: bag,
jacket, casing for cameras or mobile devices, shoes, tarps, etc. Rather than to guess the intent, mark it as unjudgeable.

Keywords: metal
Category: aps (All Products)
Judgement: Unjudgeable
Explanation: Metal could refer to the material of the product and to genre of music, or even to media or videogame titles. There
is not enough information to judge the intent, so mark as unjudgeable.

84 n
Keywords: metal
Category: music
OP: Metallica LP
e
Judgement: Exact
Explanation: Category is specifically mentioned and any music album of the genre Metal will be rated Exact, overlapping genre to
be rated as Substitute and different genre to be rated as Irrelevant.
22 id
Products Not Sold on Amazon
If the query is for products not sold on Amazon like cat, dog, gun and car etc., and the ASIN shows product related to the same
query, which can be used for/buy the product not sold on Amazon to be marked as Exact. However, merchandise which only
depicts items that are not sold (printed posters, pet printed tshirt/mugs, poster of cars, toy or airsoft gun, Image of gun etc.)
would be considered as Irrelevant.
nf

Keywords: cat
Category: aps (All Products)
Product: Cat food
Judgement: Exact
Explanation: The object referenced in the query is not sold on Amazon, any product related to the query to be marked as “Exact”.
Co

Keywords: Cat
Category: aps (All Products)
Product: A figurine of a dog
Judgement: Irrelevant
Explanation: The product is not related to any potential interpretation of the query.

Keywords: Car
Category: aps (All Products)
Product: A poster with a car
Judgement: Irrelevant
Explanation: Poster cannot be of any use to the queries product (car). Hence should be marked as “Irrelevant”. However, if the
ASIN shown is “car repair book” then judgement would be “Exact”.

Keywords: Mother’s milk

Category: aps (All Products)
Product: Lactation improvement products like tea or herbs, pumps, books on lactation

15 | P a g e
Search Relevance: Standard Operating Procedures

Judgement: Exact
Explanation: Mother’s milk is not sold on Amazon. Hence any products supporting lactation will be Exact. Other milk like
buffalo/cow’s or milk alternatives; https://www.amazon.com/dp/B00LBGOW2Y and any item with the exact title match
https://www.amazon.com/dp/B0009F3POY will be Substitute.

2 l
Unjudgeable
This judgment is only to be used if the product cannot be judged, either because the query cannot be understood, or because the

56 tia
product is lacking enough information to determine if it matches the query. An Unjudgable query is one that
a. is complete gibberish,
b. is seeking help,
c. is attempting to navigate to a website,
d. is in a non-market/foreign language (Primary/target language to be considered for the particular marketplace
unless the annotation language is specified clearly. Any language other than primary/ target to be marked
unjudgeable. Ex: French marketplace – primary language is FR and queries or offered products in any other
language like English/ Dutch to be marked Unjudgeable. If it’s a combination of French and English that is written

84 ne due to Brand name, adjectives or nouns which can be interpreted may be judged accordingly. Ex: Puma des
chaussures (Puma Shoes)- can be judged.
e. is a string of ASINs, numbers/symbols that does not correspond to an ISBN, ASIN or model number in Google
searches. *A single ASIN or ISBN number, with corresponding results on Google, is judgeable,
f. is very vague without a clear intent (see Very Vague Queries).

The query-ASIN pair is also unjudgable if the product/ASIN information is insufficient to render a judgement due to not having
22 id
access to all of title/picture/product detail page link, or due to having ASIN information in a non-market/foreign language.

S. No. Search Query ASIN/ISBN Category Judgement Reason

1. 823^%0* Socks (B0895LL44V) aps Unjudgable Gibberish
nf

2. Flipkart.com Pen (B082PN4X5J) aps Unjudgable Navigational Query

Co

3. i) 博朗 Chocolate aps Unjudgable Foreign Language

ii) chocolate寒服 (B08J37H3QK) A query, half in
target language/
English and half in
foreign language
will also be
unjudgeable.
Exception: If query
has noun, adjective
or Brand names
then to be judged.

16 | P a g e
Search Relevance: Standard Operating Procedures

4. How to cancel Women’s black Top aps Unjudgable Service-related

my prime (B07SN3KMS6) query
membership?

2 l
56 tia
5. Volume of The Ghost and the Hemispheres aps Unjudgable Non-product type
hemisphere with Kindle edition query
radius 15cm? (B08KNSWQ5G)

84 n
6.

e Gift for mother

(searched in
.com EN)
Notebook
(B08TYJYD71)
aps Unjudgable Product detail page
in other language
22 id
Labeled Examples
Exact: An offered product an Exact Match when the offered product is exactly the same as the search query. A very specific query
nf

will only be satisfied if the product matches all of the characteristics included in the query.

Substitute: An offered product is a Substitute when it differs from the intended product, but it is functionally equivalent. This will
be the same type of product, but may differ by brand, model, color, title, or other details. The offered product may also meet the
purpose or the utility of the desired product even though it is not strictly of the same product type.

Note: Any critical matching defect/attribute mismatch which makes the recommended ASIN unsuitable in the context of the query
Co

should lead to the ASIN judged as Irrelevant. (See Product Attributes and Impact of Mismatches.)

Complement: A product is complementary to the SQ when it could be useful to the customer if they did buy the intended product.
This could be an accessory, a component, replacement parts, or even another item designed to or could reasonably be used
together with the product. More broadly, the product can be considered a complement if the customer could be interested in it
because of the brand affinity (same brand) or because the offered product follows the same theme, genre, or occasion as the
intended product.

Note: Any critical matching defect/attribute mismatch which makes the recommended ASIN unsuitable in the context of the query
should lead to the intended product judged as Irrelevant. (See Product Attributes and Impact of Mismatches.)

Irrelevant: A product is Irrelevant to the SQ when it does not satisfy the criteria for either Exact, Substitute, or Complement label,
but when the query and the product information are clear enough to evaluate the type of the match. The offered product has
failed to match the purpose like: utility, size, age, gender, compatibility of the desired product, or the offered product fails to
create interest of a purchase by not aligning with the desired product on brand, genre, theme. (See Product Attributes and Impact
of Mismatches.)

17 | P a g e
Search Relevance: Standard Operating Procedures

Assume All Products category and amazon.com marketplace for all examples in the table below

Query Exact Substitute Complementary Irrelevant

Clorox bleach Clorox Performance Lysol Disinfectant Spray Clorox Bathroom Foamer Glad ForceFlex Tall

2 l
Bleach, 121 Oz, Pack of 3 Refillable Cleaner Kitchen Drawstring
Trash Bags

56 tia
84 ne
Coleman 225 Coleman RoadTrip 225
Same purpose
(disinfectant)
Char-Griller E1515 Patio
Brand affinity, related
products (home care
category)

Bernzinatuc Disposable
Products do not
serve functionally
the same purpose.

GasOne Camping
22 id
portable grill Portable Stand-Up Pro Charcoal Grill, Black Propane Cylinder, 16.4- Fuel Blend Isobutane
Propane Grill Oz Fuel Canister
nf
Co

Same purpose/function,
different brand/model Not compatible with
The grill takes 16.4-Oz DP -- defect
propane cylinders – an
accessory to DP.
Navy blazer for Mens Casual Sport Coat 1 Men’s Standard-Fit Stretch Haggar Men’s Dress Blazer Jacket for
men Button Suit Blazer Twill Blazer Women

18 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
Can be used as an
independent product
with DP as part of an Designed for
Does not come in
outfit women, won’t fit
navy/blue
men – defect wrt DP
Otterbox case OtterBox Defender Spigen Liquid Air Google Pixel 7-5G

84 n
for pixel 7

e Series case for Google

Pixel 7
Designed for Pixel 7 Case Android Phone
22 id
Compatible with DP, Lead product when
different brand DP is an accessory
Unicorn Kids Girls Backpack Unicorn Kids Luggage, Robo Pets Unicorn Toy Disney Princess
backpack Preschool Unicorn Girls carry on suitcase Blanket
with backpack
nf
Co

Same theme, different

OP is a bundle with DP as
product.
one of independent
products. Product does not
match the theme.
PS5 Console PlayStation 5 Console – Sony Playstation PS4 1TB Call of Duty: Modern SIKEMAY PS5
Horizon Forbidden West Black Console Warfare II – PlayStation Controller Skin
Bundle 5

19 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
Bundle with the console
as the lead product
Same purpose (gaming
console), different model The product
neither serves the
Game compatible with
purpose of nor is it
the DP
an accessory to the
DP. (It is an

84 ne
Appendix
accessory to an
accessory.)
22 id
Examples of Applying the Labeling Process
Example: pickup tents for camping short bed

Query: pickup tents for camping short bed

nf

Category: aps (All Products)

Marketplace: amazon.com

Product Title: Napier Backroadz Truck Tent, Grey/Green, Full

Size Regular Bed (6.4’-6.7’)
ASIN: B07HXC4HBD
Co

Step 1: Understand the query: Searches on both Amazon and Google (note: the UI will typically provide a link) return tents
mounted in a bed of a pickup truck, broadly used for sleeping while camping. There is no other obvious interpretation for this
query. The query implies product type tent, with camping as the intended use, compatible with a pickup truck with a short bed.

20 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
84 ne
22 id
Step 2: Understand the ASIN: From the title and the picture, the product is a tent compatible with the full size regular bed pickup
truck. To get more information on the intended use of the product and its compatibility, head over to the product detail page (the
UI will typically provide a link). The product detail page indicates car camping as the recommended use for product.
nf
Co

21 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
84 ne
22 id
Step 3: Identify the product attribute matches and mismatches: The ASIN matches the query on the product type (tent), use of
product (camping), overall compatibility (pickup truck, mentioned on the product detail page as Design: Vehicle Mounted Tent).
There is a pickup truck bed size compatibility defect. The keywords mention short bed while the ASIN is for full size regular bed
nf

(6.4’-6.7’). How bad is this defect?

One of the available images (4th from the top on the left) is a model compatibility sizing chart (see below) which indicates that
short bed has 5’-5.8’ length rather than 6.4’-6.7’ length which this model of the tent is designed for. The mismatch is two sizes
(ASIN model is 19022, the one matching the query is either 19033 or 19066, and there is an intermediate size model, 19044, for
the bed length 6’-6.3’).
Co

22 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
84 ne
22 id
Step 4: Assign the label based on the identified defects: There is only one defect, compatible bed size length. Based on the
information available to us, the defect is significant (two model sizes, unlikely to fit the vehicle) likely making the ASIN
nf

incompatible with the intent of the query leading to Irrelevant label.

Example: bobs shoes with dogs on them

Query: bobs shoes with dogs on them

Co

Category: aps (All Products)

Marketplace: amazon.com

Product Title: Skechers Women’s Slipper

ASIN: B08559LHNW

Step 1: Understand the query: The query likely refers to shoes with images of dogs on them. What is bobs referring to? A search
for the keywords on Google indicates that Skechers likely has a Bobs line of shoes (see below). An additional search for skechers
bobs turns up a page for the product line (see below). Therefore, the query can be interpreted as having a shoe product type, with
brand Sketchers, product line BOBS (rather, BOBS for Dogs), and images of dogs for theme.

23 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
84 ne
22 id
nf
Co

24 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
84 n
Step 2: Understand the ASIN: The product title and the image already suggest that the ASIN product type is shoes, brand is
e
Sketchers, and theme is images of dogs. To product line is unclear from the image, but going to the product detail page and
examining the third image from the top left (see picture below) indicates that the product line is indeed BOBS.
22 id
nf
Co

Step 3: Identify the product attribute matches and mismatches: All attributes identified in the query match those in the ASIN.

Step 4: Assign the label based on the identified defects: The label is Exact match.

Working with Categories

Product type mentioned
Category: APS Category: Specified (other than APS)

25 | P a g e
Search Relevance: Standard Operating Procedures

When the search query clearly mentions the product type which When the search query mentions the product type and a
clarifies the customer’s intent and the availability of the product specific category is also selected. In this case the query and
on Amazon. The category selected is APS, then we only have to category together can give you a better understanding of
mark the searched product as Exact. the specific need of the customer. Then only the intended
For example : if the search query is basketball (APS) , now we result should be marked as exact.

2 l
know Amazon sells basketball in this case we will mark only the For example: if the search query is basketball (Electronics) ,
basketball should be marked as exact. now due to the specific category, one can find phone

56 tia
covers, pendrives or airpods cover in shape of basketball
(changing the intend and focusing on the specific product),
thus such products shall be marked as exact and not the
basketball.

84 ne
22 id
Product type not mentioned
Category : APS Category : Specified (Other than APS)
In many cases the search query doesn’t have any product type When the search query doesn’t have any product type
mentioned. And the category selected here is APS. We can mentioned and consider it is a theme, the search alias can
consider our search query as a theme. In such cases we can give us a fair idea to understand the intent of the query. In
provide all the possible products related to the theme as SPT. such cases we can consider the selected category as a
nf

For example: the search query is Marvel (APS) , now we know product type.
Amazon sells a lot of products related to marvel , in this case we For example: If the search query is Marvel (alias : toys) in
will mark all the products as Exact such as t-shirts , books , this case we will consider the category as a product type
DVD’s, figurines, etc. and we will mark only marvel lego toys/figurines etc as
Exact. Other products related to marvel like movies, books,
posters, mugs will be Complement.
Co

Exception: When the product type is clearly understandable but the category selected doesn't specify a different intend of the
customer. In such cases, we will not take the category into consideration while taking an enforcement action. For example: If the
search query is ‘Jacket’ and the search alias is ‘Shoes & bags’, in this case we will ignore the category and consider only Jackets as
the Exact product.

26 | P a g e
Search Relevance: Standard Operating Procedures

Category specific examples

In this section we have bifurcated the search queries on a category basis, to understand how different queries pertaining to certain
categories need to be judged.

2 l
**All the Queries are in APS store unless otherwise mentioned. **
1. Electronics

56 tia
Headphones: Earbuds/Headphones/Earphones all are Exact except in JP where they are substitute
The extended definition of substitute for ASINs which meet the purpose/utility of the intended products apply to Electronics as
well.
Sub Category Query Exact Substitute Complementary Irrelevant
Samsung note Samsung note 10+ note 10+ cover + Samsung note 10
Cell phones iPhone XR
10+ phone protective glass lite Cover
Computers &
Motherboard Motherboard - Processor PC screen

84 n
peripherals

Headphones

TV
e
Amazon
Asus Zen book
Gaming
headphones
TV support 55
inches
Echo dot
Asus Zen book 14
HyperX gaming
headphones
TV support Samsung
45-75 inches
Echo dot
HP Elite book

Apple ear pods+ case

console table with
universal TV stand
Echo dot + smart bulb
laptop cover 13-15
inches
headphone charging
support
screws

Echo dot cover

laptop charger
for Lenovo
keyboard

TV

JBL speaker
22 id
devices
All Alexa compatible
devices
Non Alexa/Non
Alexa
Smart devices
like Alexa compatible
Plugs/bulbs
Canon EOS Canon EOS 2000D + SDXC memory card canon printing
nf

Camera Nikon cool pix

2000D lens + tripod 16 GB cartridge
men's
screen protector for
Watches intelligent Fitbit versa 3 (unisex) men's analog watch Boys’ Fitbit
Fitbit versa 3
watch
Electronics Gadget All devices/machines
Co

2. Media and Games

Media primarily includes films and music. Under films, we have formats like DVD, Blu-Ray and Video cassette. Under songs we have
CD, Vinyl, cassette, and streaming. Genres in movies are the primary judgeability criterion. If genres don't match, then movies are
irrelevant
Games here mainly refer to the video games including all PS models, Nintendo, all X boxes and Windows games. The category also
include game accessories like controllers, consoles, game codes etc. Model compatibility is paramount in the videogames.
For Books, that doesn’t match the query, we have to mark as per the Genres
Substitutability in Media  When the query is a topic or media title such as music, book, or movie, then other media that are in
the same general category, the judgment is Substitute. If a media product can be considered to be in multiple categories, then
either could be Substitute. Also if any of the same contributors to one media product are also in another media product, then they
are Substitute.
Sub Category Query Exact Substitute Complementary Irrelevant
Infinity war (Action) Infinity war i)Spider-man Trailer of the movie Titanic (genre
Movies different, romance)

27 | P a g e
Search Relevance: Standard Operating Procedures

ii)Infinity war
themed products
iii) Music score and
Songs

2 l
Complements in Media  Media products that are about other media products are considered to be secondary material and are

56 tia
judged Complement so long as they match the category included in the query. This includes reviews, analyses, study guides,
anthologies, or commentary, where the book/video is meant as a companion to the central product they are about.
Sub Category Query Exact Substitute Complementary Irrelevant
COD PS4 game
and all COD Games CD stand
COD ps4 PS4 GTA game COD XBOX ONE
themed product Guide book for COD PS4
for PS4
ps5
Charging station Cables for charging Charging station for PS5

84 n
Video Games

e charging
station
Video
game

Xbox
for PS 5 console

Video game CD,

DVD, console

Only XBOX
Consoles are the
main PT
PS4 Console
station for console

Joystick, Controller
Gamepad

s for Xbox Console

Controllers

Cases/Accessories for
Cases/Accessories/Game Xbox Controllers
(Accessories to
Accessories)
22 id
Harry
Movies/Prime Harry Potter Fantastic Beasts Harry Potter books or Titanic (Genre Different)
Potter
video Movie movie Merchandise
Movie
Endgame prime
video/Endgame
Endgame Marvel kindle
DVD format Endgame Blu-ray DVDs, CDs, Blu-ray holder
Blu-ray version/vice-versa
nf

SACDs and DVD

audio discs
One Direction
One Linkin park
music and brand / Biography of members Spiderman movie
Direction music,
themed products
Purpose CD
Co

Justin Bieber - Fine Line Vinyl Vinyl cleaning kit

Purpose Vinyl
Justin
Bieber Vinyl Player
Vinyl
Music/Vinyl/Audio

Walls CD Reputation CD Walls Vinyl

Walls CD

CD Player

28 | P a g e
Search Relevance: Standard Operating Procedures

2 l
Guide to use snapchat, Temple Run app,
Snapchat Snapchat Instagram
Gift card to buy Snapchat General Gift cards

56 tia
Apps/Software
Windows Windows 10 Windows server Laptop with inbuilt
Windows 10 guide
10 Home 2019 window 10

3. Fashion
Sub
Query Exact Substitute Complementary Irrelevant Exceptions
Category
If the query

84 ne Pink Cushion
cover 50 x
50
Pink Cushion
covers, size
50*50
Blue Cushion covers,
size 50*50 Any decorative to
decorate the cushion
covers/straps/zips for
cushion covers
Cushion cover
of different
size 30*50
was “cushion
cover 30 x 40”
then even if we
get cushion
cover of 40 x
30 size, it will
be marked
Exact
22 id
Superman t shirt
Softlines Batman pants/
innerwear
Plain inner
Batman t shirt
wears
nf

Batman T-
shirt Any tshirt

Plain pant will be

considered
complement
Co

Polyester T-shirt.
Cotton T- T-shirt made of T-shirt made of other
Cotton shorts/pants Woolen pants
shirt >=50% cotton material or with less
than 50% cotton

29 | P a g e
Search Relevance: Standard Operating Procedures

Cotton fabric
(pure or >=50%) Cotton T-shirt

Cotton <50% or
Material Cotton
Leather, Polyester,
(fabric)

2 l
Silk (Fabric)

56 tia
In DEFRITES_JP
Tunic/ blouse/ Pants/ jeans Sweatshirt/Pull and IN market
Tank top over place T-shirt
Any Shirts
will be
Shirt substitute

84 ne
Apparels

Top
Tunic

Jacket
Pants
Any Inner
wears
22 id
(anything
from neck to
waistline) Coat
nf

A one-piece
garment from Jumpsuit or Saree, Salwar
top till ankles. 2 –piece dress
Top till knees
Co

Buttons/Accessories
Clothing Dress
for a dress

30 | P a g e
Search Relevance: Standard Operating Procedures

Ready made
Lehanga.

Unstitched materials Dress

2 l
for lehenga/ salwar
suit.
Salwar suit/

56 tia
Ready to wear Semi-stitched
Lehenga Salwar suits. Lehenga/ salwar suit.
choli

84 ne Pajama/PJs/
Pyjama
Any pant/set of
clothes, which
can be worn
while sleeping
or lounging.
Pajama Top

Any normal
pant/shirts not
used as sleep
22 id
wear.

2-Piece Bikini One-piece Monokini

nf

Bikini Cover ups

Bikini
Co

Any Necklace Necklace set with Pendant Anything other

bracelet and ring than necklace
Necklace like only
rings/bracelets
/anklet

Jewelry Different gem stone

Ruby Stone

Ruby Any jewelry made out

of ruby stone

31 | P a g e
Search Relevance: Standard Operating Procedures

Watch strap, screen

Any Wrist protector and watch
watch/ Smart tool kit
watch Watch with bracelet

2 l
56 tia
Wrist Watch

84 n
Wearabl
es

e
Shoes
Any sneakers,
sandals, pumps,
sliders, training
shoes Shoe lace, socks and
insoles

Only in IN
market place
sandals/sliders
22 id
will be
irrelevant.
nf

4. Luggage and bags

Luggage: By definition, luggage refers to empty bags or suitcases, used for carrying a person’s clothes and things on a journey.
Anything which is used to carry personal stuff for the purpose of travelling can be considered as a luggage.
Co

Bags: a strong container made from cloth, plastic, leather, etc., usually with one or two handles, used to carry things in when
travelling, shopping, etc.
Sub
Query Exact Substitute Complementary Irrelevant Exceptions
Category
Rolling Suitcases,
Backpacks, Duffel Bags,
Softside Luggage,
Hardside Luggage, Two-
Wheeled Luggage,
Luggage Luggage Spinner Suitcases

32 | P a g e
Search Relevance: Standard Operating Procedures

Skybags Suitcase cover,

Suitcase Skybags suitcase Aristocrat suitcase Backpack
Suitcase wheels, handle

Handbag, Backpacks,
tote, slings, Lunch bag,

2 l
carry bags, Gift bags,
etc. Bags for

56 tia
Bags Bags A bag used for any carrying pets
purpose.

A bundled
Bag Strap, Chain,
item of 3
bag Charms

84 n
Handbag

e
Purses

Wallets
Caprese
Handbag

MK Purse

Allen Solly
Caprese Tote bag/sling
bag/purse/wallet

MK Purse

Allen Solly Wallet/clutch

Lavie Tote bag/sling
bag/purse/wallet

Gucci
Purse/Handbags
Tommy Hilfiger
wallet/clutch/handb
Purse straps

Zips/chains, Wallet
Suitcase/
backpack

Suitcase/Bac
kpack
Suitcase/Bac
bags can be
given as
exact, as all
are
handbags
22 id
Wallet charms kpack
ags

5. DIY & Tools

Sub Category Query Exact Substitute Complementary Irrelevant
If alias is DIY,
nf

chair/bedsheet
having patchwork is
Irrelevant. However,
Patchwork Patches and scale for if the category is
patchwork Beads or other Needle/ thread "aps" the query can
decorating for sewing the be considered as
Art/Craft/DIY
(if alias aps, products for patchwork on "style" and mark
Co

chair/bedsheet having textile the textile chair with

patchwork) patchwork as
"Exact"

Cloud decoration 1. Star shaped

Store: DIY DIY decoration one accessory/ Non-deco products,
DIY, Tools Cloud deco DIY kit
part of cloud utility objects like
and Garden
2. cloud deco deco DIY kit chair, table etc

33 | P a g e
Search Relevance: Standard Operating Procedures

Nail & hammer

Nail paint or
(bundle)
Nail (Tools) Bundle of different artificial nail (as
Store: DIY sizes of nails alias is DIY/
industrial)

2 l
Solar Light DIY 1.Normal light

56 tia
DIY kit (not
solar) Battery or any
solar light DIY kit
2. Solar light set other single part
Store: DIY
(FINISHED for solar light
PRODUCT)

1.tools/machines for
working with wood

84 n
Industrial/
Research and
development
e
products
wood working 2. book on the theme.
accessory of the
wood-working
tool
Wooden product
like chair, table etc
22 id
6. Beauty

Sub Query Exact Substitute Complementary Irrelevant

nf

Category
Hair care & Shampoo Shampoo Shampoo + Conditioner Conditioner
products

Wigs Hair wigs Hair Extensions (both are Clips/caps to attach a

Body care used to change person’s wig
& products hair style – same purpose).
Co

Perfume Perfume Deodorant

Moisturizer Moisturizers body cream Skin tightening

creams

Face care Foundation Foundation in liquid BB cream/ CC cream/ DD Brushes/pad to Lip gloss
& products form and Powder cream apply foundation to
form face; directly related
products like primers
and concealers/
setting powder

34 | P a g e
Search Relevance: Standard Operating Procedures

Eye liner Pencil/liquid/gel kajal Eyeliner brush contact lenses/ lip

liner color

Lipstick liquid lip color lip gloss/ lip balm lip primer eyeliner
face mask Sheet mask mask Scrub/ toner/ mascara

2 l
sheet moisturizer/cleanser

56 tia
Blush Blush cheek tint highlighter/ blush eye shadow
brush

84 n
7. Medicines
For any formula foods we need to ensure that the age of the baby/kids matches.
Sub Category Query
e Exact

Fat Burner
Substitute
Dietary Supplement
for weight loss
Complementary

Book on Weight Loss

Irrelevant

Weight gain
powder or just
22 id
supplements

Supplements Weight Loss

nf

Cold and runny Nose

Suspension.
Here, it also can be
used for fever as per
the product Fertility Support
Co

Fever Reducer Pill description. Thermometer

Fever
Drug store

8. Grocery
Sub
Query Exact Substitute Complementary Irrelevant
Category

35 | P a g e
Search Relevance: Standard Operating Procedures

Tomato puree,
Vegetable wash
ketchup, all veggies Tomato shaped
spray
Tomato earrings

Food Tomato

2 l
56 tia
Tropicana orange Juice Apple juice Juicer

Beverages Orange juice

84 ne
Alcohol Whiskey
Bourbon Whiskey
Rum
Whisky - The Islay
Edition
(documentary)
Orgain organic
nutritional shake
22 id
Book on
Potato (raw), Frozen or consumption of
potatoes Potato wine
Potato chips
dried potato
nf

Amazon
Potato
Fresh
Co

Wheat flour 5Kg X 3 Wheat flour 2.5Kg

X2 Wheat bran
Wheat drinks

Amazon
Wheat flour 5Kg
Pantry

36 | P a g e
Search Relevance: Standard Operating Procedures

Other dry
fruits/almond
Almond flavour
flour/almond
chocolate/ baking
Almonds butter/
essesnce
[salted/unsalted] almond

2 l
milk/almond oil
Nuts Almonds

56 tia
Ketchup/mustard Fritters with mayo
Mayo filling

84 n
Condiment

e Mayo

Chocolate
spread/Almond
22 id
butter/Nutella
Peanut Butter spread Ketchup/Mustard
sauce
Peanut Butter
nf

Cake Pancake cake mix

Co

Chips

Gourmet Cake

37 | P a g e
Search Relevance: Standard Operating Procedures

Oats milk/ oats

biscuit/oats cake
(primary ingredient
is oats)/any
breakfast cereal

2 l
Honey/Dryfruits

56 tia
Any Spices
Oats

oats

84 ne Chips of other
22 id
brand salsa dip/any other Biscuits
Pringles dip

Pringles
nf

Anything vegan

Vegan Anything non vegan

Co

Non vegetarian Meat

Comment: Non
vegetarians
Vegetarian
may consume
food/vegan food
vegetarian/vega
/plant based meat
n food,
not vice versa

38 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
9. Fresh
Sub Query Exact Substitute Complementary Irrelevant

84 n
Catego
ry

Fruits
e Apple

[Note - Flavors will

be irrelevant but
Apple apple juice/apple jam/
dried fruit/Apple puree/
mixed fruit juice
Fruit basket;
apple peelers
apple
chocolate/apple
yogurt/
22 id
when added as an
ingredient then
substitute]
nf
Co

Lemon Lemon any fruit/lemon lemon mint/pie

pickle/lime sqeezer/lemon
[Note - Flavors will zest
be irrelevant but
when added as an
ingredient then
substitute]

39 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
84 ne Coconut

[Note - Flavors will

be irrelevant but
when added as an
ingredient then
substitute]
Fresh & dry coconut Coconut milk/ coconut
oil (for consumption)/
grated coconut flakes
Coconut ice
cream
22 id
nf

ginger ginger garlic/ginger tea with ginger

paste/ginger powder and lemon
[Note - Flavors will
be irrelevant but
when added as an
ingredient then
substitute]
Co

Fish Raw fish Cooked fish/Frozen Fish shaped

Meat/Processed snack/chocolate

40 | P a g e
Search Relevance: Standard Operating Procedures

2 l
56 tia
10. Pets

84 n
Domesticated Animals - Do not treat them as theme, however Wild Animals can be considered as theme

Sub
Category
e Query

DOG
Store Exact
Dog food, Dog
accessory, Dog toy,
Stuff toy, Figurines,
miniature
Substitute Complementary Irrelevant

Product with their

Image (Print),
themed, Products
22 id
of CAT.
pets/aps
(domestic e.g.
cat/ dog)

Pet
nf

Product with their

Image (Print),
themed
Wild animals pets/aps
Co

11. Toys
Sub
Query Exact Substitute Complementary Irrelevant
Category
Board games Ludo/Catan Card games like Uno Dices, pawns, chips Soft toys/figurines
Any other character
costume from Harry
Harry potter
Harry Potter Potter like Dumbledore
Toys costume Harry Potter wand Batman mask
costume or Hagrid/ Fantastic
beasts costume or
other theme costume.

41 | P a g e
Search Relevance: Standard Operating Procedures

Game for 0-4 years or 6

Games for 5 year Games for 0-5 year
years (we allow -1 and Guide to the game Game for adults
old old
+1 age only)
Accessory for plush Figurine, Plush toy
Plush toys Any Plush toys
toys for pets, Adult toy

2 l
12. Home

56 tia
Sub Category Query Exact Substitute Complementary Irrelevant
Furniture Bed Single bed/ Double bed Sofa cum bed Mattress/Bed Sheet Chair
Wall Art Wall paper Wall paper Poster Glue Stick Photo frame
Bathroom Bath Towel Towel Hand Towel Hook/Rack Bath Mat
Storage and Organization Wardrobe Almirah Clothes Stand Wardrobe light Bed

84 n
Garden

13. Kitchen
e
Sub Category
Dinning
Cookware
Query
Plate
Induction
Plant/
Flower

Exact
Plate
Induction
Real Plant/ Flower
(including seeds)

Substitute
Thali/Bowl
Gas Stove
Artificial Plant/
Flower
Planter/ Fertilizer

Complementary
Spoon
Induction cleaner
Irrelevant
Cooker
Cylinder
Planter
stand
22 id
Appliance Chimney Chimney Exhaust Fan Chimney pipe Celling Fan
Microwave Microwave OTG (Oven Toaster Grill ) Microwave plate Sandwich Toaster

14. Automobiles
nf

Sub Category Query Store Exact Substitute Complementary Irrelevant

Plastic model,
Car (same model) miniature, figure (of
Car Accessory (of the same the same model)
(Automobile) model)
Vehicle aps
(name of the
Co

car model)
Automobiles

Tire/Headlights/ for same model

aps Bulbs
Battery (compatible)
Car
Plastic model, miniature,
(Automobile) toys
figure
Vehicle

15. Shoes
Sub
Category Query Store Exact Substitute Complementary Irrelevant Exceptions
Clogs aps Clogs for any Bathroom Sole for sandals Sport shoes for
Shoes
gender Slippers any gender

42 | P a g e
Search Relevance: Standard Operating Procedures

Sports aps any laces for sports high heels/formal

shoes for sports/running shoes shoes for
women shoes women/sports
shoes for kids or
men

2 l
Tommy shoes shoes , wallet, shoes, wallet tshirt from any The shoes
Hilfiger and shoes laces, and other brand category/store

56 tia
bags etc from accessories consists of small
Tommy from other bags and wallets,
Hilfiger brand hence to be
marked exact in
this category.

84 n
16. Theme
Theme Query
e Cricket
Exact
All cricket
equipment/gears.
Note: unlike SQ - Football
which is a product type in
itself (the ball is called
football), cricket is treated
Substitute Complementary
- Book on cricket
Irrelevant
Normal shoes
22 id
as a theme
SQ: Football
OP: Football shoes-
complement
Profession Anything for that - Toy sets for
profession children
nf

like stethoscope for SQ associated with

Doctor that profession
or on that profession like Doctor set
for SQ-doctor

17. Adult
Co

Sub Query Exact Substitute Complementary Irrelevant

category

Condoms condoms XL condom L Lubricant 1.condom S

Adult XL 2. Sex toy

sex toy sex toy for men Lubricant 1. sex toy for women
for men 2. condom

18. Books

Key Points:

43 | P a g e
Search Relevance: Standard Operating Procedures

 Price, adjectives, services – to be considered

 Overlapping ages to be considered Exact, adjacent ages to be considered Substitute
 Sub-genres of the books to be deep-dived via available decision-tree
 Merchandise to be rated as Substitute
 Target audience to be ignored except for children intent queries.

2 l
 Language difference in books
 Complementarity in books

56 tia
Further elaboration:
Important points to consider in specificity to ‘books’ queries, divergent to the above/usual guidelines:
(1) Price/Free: any book (in any format) matching the price constraints chosen by the customer are to be taken into consideration
If customer is looking for “free books”, price must be 0 in the local currency

Any books which is non-free would thus be rated Substitute.

84 n
(2) Bestsellers/Validated Adjectives: query constraints must be respected (e.g. “romance bestseller” will need to satisfy both
bestseller and genre guidelines to be judged Exact).
e
While adjectives such as ‘cute’, ‘best book’ may be subjective and can continue to be disregarded at time of relevance
judgement, however ‘top seller’, ‘high rated’, ‘bestsellers’ and the like may be valuable for judgement.
>>The criteria for researching and dive deep here may include pointers and hints such as:
22 id
* any Book with the Amazon Charts OR Editor’s Pick OR Bestseller badges
* any Book with “New York Times bestseller” reported on the cover
* any book ranked within Books/Kindle Store Zeitgeist top 2,000
* any book with more than 1,000 reviews
nf

>>Below books would be rated Substitutes:

* any book ranked between 2,000 and 10,000 in Books/Kindle Store
* any book having between 500 and 1000 reviews
(3) Service queries: while service or membership seeking queries such as “cancel kindle unlimited” continue to be disregarded,
however, keywords (and functional equivalents): “kindle unlimited”, “prime reading”, “audible plus”, “kids plus”, “comixology
Co

unlimited” all contextually hint towards a customer searching for books and thus must be considered for judgement accordingly.
E.g. while the SQ: ‘kindle unlimited’ could entail all books to be marked as Exact, ‘audible plus’ hints at a particular format of book
and thus must be provided the relevant judgement, distinguishing format.

(4) Age range/reading level: a children book for an age range adjacent to the one the customer was looking for (e.g. a book suitable
for 3-4 yrs old when customer is looking for “children books 5-6” would be rated Substitute.
Overlapping age range, e.g. books for 3-4-year-old and resulting product with 4-5-year-old book may be rated Exact.

Grade Age

Pre-kindergarten 3-5

Kindergarten 5-6

44 | P a g e
Search Relevance: Standard Operating Procedures

1st grade 6-7

2nd grade 7-8

2 l
3rd grade 8-9

56 tia
4th grade 9 - 10

5th grade 10 - 11

6th grade 11 - 12

84 n
7th grade

8th grade
e
9th grade (High school)
12 - 13

13 - 14

14 - 15
22 id
10th grade (High 15 - 16
school)

11th grade (High 16 - 17

school)
nf

12th grade (High 17 - 18

school)

(5) Distinguishing between genre and subject: Genre refers to categories of works that are characterized by similar plots, themes,
Co

settings, situations, and characters. However, the Subject describes what a book is about, or its topic. Example of subjects are:
Architecture, Photography, Sports (in general or specific, e.g. football), etc.
*fiction and non-fiction books are to be rated Irrelevant
(6) Sub-genres: it is important to take note that while sub-genres such as Sci-Fi-thriller and a usual action-thriller may be rated
as Substitute products, however a significant movement from sub-genre, e.g. Sci-Fi-thriller and historical-thriller will be
rated Irrelevant.

(7) Media Vs non-media/merchandise Products: all the media products that exactly match the keywords and are either the main
work(s) in the original medium or other media derived directly from the original medium are Exacts. Conversely, non-media
products about the original work(s) are Substitutes. E.g. all Harry Potter movies and books are Exacts as they are either the primary
work (the books) or direct media derivations of the primary work (the film transpositions).

However, all non-media results, e.g. merchandising (https://www.amazon.com/s?k=harry+potter), are Substitutes.

Similarly for people names (historical figures, celebrities, authors) and fictional characters – any media product authored by or
about the person is an Exact. Whereas, merchandising and related products are Substitutes

45 | P a g e
Search Relevance: Standard Operating Procedures

a) SQ: Harry Potter

ASIN: Harry Potter coffee mug (merchandise)
Judgement: Substitute (all Harry Potter movies and books are Exacts as they are either the primary work (the books) or direct
media derivations of the primary work (the film transpositions).

2 l
All non-media results, e.g. merchandise is Substitute.)

56 tia
b) SQ: Harry Potter book
ASIN: HP coffee mug
Judgement: Complement (This would be the same for SQ- "harry potter mug", the mug is exact, the book Complement)
(8) In Books, audience as a criterion to judge irrelevance will be ignored, except for children/age intent queries (regardless of
title, author, topical intent). This means that we will continue judging as Irrelevant adult books appearing for children queries.
Previously, annotators also had to consider other types of audiences (e.g. whether a book was written for a professional vs non-

84 n
professional reader), which can be subjective and cause confusion. This is no longer to be considered. Broadly, books on the
same topic are considered to be a Substitute. The is because the reader (customer) might be interested in other books on the
same topic. There can be new-released/ books released earlier (that the shopper might not have seen), which might intrigue
their desire to buy other books related to the same topic.


e Exact: if the age range of the ASIN is comprised in the age searched by the customer (e.g. if customer looks explicitly or
implicitly -e.g. via title search - book for a 5-yr old, any ASIN with age range comprising 5-yr old is exact – e.g. a 3-5 yrs
book).
Substitute: any book with an age range adjacent to the one the customer looks for. E.g. if customer looks for 3-5 yrs old
22 id
books, 6-7 yr old books are substitutes.
 Irrelevant: non-comprised and non-adjacent age ranges. E.g. 6 yr-old suitable books when customers look for baby books.

Examples:
nf

Keywords: The Cancer Code

Recommended Title: Everyday Strength: A Cancer’s Patient’s Guide to Spiritual Survival
Explanation: The search query is a medical book to understand medical conditions during cancer, treatment etc. The search
result however is a guide to cancer patients on spiritual survival. Hence marked substitute.
Keywords: dieters gourmet cookbook
Recommended Title: The Korean Vegan Cookbook: Reflections and Recipes from Omma’s Kitchen
Explanation: Query is dieter’s gourmet cookbook and offer product is Korean Vegan cookbook. Irrespective of target audience, it
Co

is still considered as cookbook and hence marking Substitute.

Keywords: cookbook
Recommended Title: Fatty Liver Diet Cookbook: A Liver Cleanse and Detox Guide
Explanation: Query is just cook book and the ASIN is cook book for fatty liver, it is considered as Exact. But if the query is “Cook
book for thyroid” and the result is “Cook book for fatty liver”, then it is Irrelevant.

Keywords: Comic book for 8 yrs

Recommended Title: Archie 1000 page Comics-Palooza for 9-12 years
Explanation: Query is Comic book for 8yrs and the offered product is 9-12 years. Considering age criteria for children this book
will be adjacent to the age asked and hence Substitute.

(9) Target Language: The idea behind the books guidelines is that if the query is unintelligible for the human auditor (e.g. because
it is in a language the auditor does not speak), then it should be marked as Unjudgeable. But if the query can be understood and

46 | P a g e
Search Relevance: Standard Operating Procedures

it refers to a title, then all editions of that title in any language are Exacts (as long as the ASIN item name is the same as in the
query - this mostly applies to translated titles that keep the English/original title also in the translation).

Example- 1: US Marketplace & Language – English; SQ: Art of War ; OP: Art of War (Chinese Edition) (PDP and Title written in
English) – Exact ; Any edition will be exact provided the title and description in PDP is in the target/ Seach Query language.

2 l
Example- 2: Arab Emirates Marketplace & Language – Arabic; SQ- ‫( ش ـكة الـموت‬The Death Network) -Arabic Edition- Exact. Any

56 tia
edition will be exact provided the title and description in PDP is in the target/ Seach Query language.
(10) Complementarity in Books - In Books, the understanding of "Complementary" category is restricted to study guides,
summaries and companion books, and all other derivations from the original works (soundtrack, cookbooks) would be rated
Substitute. If the Search Query specifies a particular product, then another product may be marked as Complement (e.g. Harry
Potter mug; Harry Potter tshirt - examples stated above in non-media merchandise as well)
Examples:

84 n
SQ: Harry Potter
ASIN: Harry Potter sound track
Judgement: Substitute (Derivation of the original work)

SQ: Harry Potter

e
ASIN: Harry Potter documentary (Behind the scenes of Harry potter)
Judgement: Complement (Summary of the original work)
22 id
For further detailed insight, please refer to the books SOP that is shared separately.
https://w.amazon.com/bin/view/A9_Search_Relevance_ESCIU_SOP/Books

19. Brand affinity

Brand affinity is a personal or emotional connection forged between customer and a brand. From a customer’s perspective, they
nf

might be interested in seeing similar category product offerings from same brand. Therefore, considering this emotional
connection between customer and brand and labelling as Complement to the customer query.

Two different lead products from the same brand can be marked as Complement, if they fall under same product category. Brand
affinity is to be considered in irrelevant cases, where query and shown product are not exact match or cannot substitute each
other or are not used together products. But we would still consider shown product as Complementary as both products are
Co

from same brand and fall under same category. Here we consider query and shown product as complementary not because both
are used together products, but considering the possibility that customer who likes a product from a particular brand might have
a curiosity/interest to buy a different product of the same brand, which fall under the same product category. Thus, when it
comes to brand affinity, we are defying the purpose of products, but the product category will act as a proxy for purpose and will
be a guard rail to check the products are not too far off.

Exceptions:
• Brand affinity is not considered for medical and supplements products as the same brand can manufacture medicine for
different diseases as well. For such medical products, purchase decision of customer would be strictly based on disease.
• Different gender/age group.
• Incompatibility of products.
• Products for different pets.
• Products related to different religion.
• Products for different sports.
• When shown product is an accessory for another lead product.

47 | P a g e
Search Relevance: Standard Operating Procedures

Query Store Exact Substitute Complement Irrelevant Exception

Maybelline APS Maybelline Mac cosmetics Maybelline Hair brush -
eye-liner eyeliner eye liner lipstick
swisse APS swisse Vitafusion - Swisse hair
multivitami multivitamin men multivitamins nutrition

2 l
n men https://www.ama https://www.a for men
zon.com/dp/B07B mazon.com.dp/ https://ww

56 tia
8NF7C9 B007L0DONW w.amazon.
sg/dp/B07
BF67B9C
Dove APS Dove Shampoo Loreal Shampoo Dove
Shampoo Deodorant
Benadryl APS Benadryl fever Ibuprofen fever Benadryl Brand affinity cannot be
fever tablets tablets Anti-Itch considered for medicines
tablets’ Gel

84 n
Nike shoes
for men

Naturvet
e
for cat
Samsung
refrigerator
APS

APS

APS
Nike shoes for
men

NaturVet for Cats

Samsung
refrigerator
Puma shoes for
men

PurinoPro for
cats
LG refrigerator
Nike flip flops/
Formal shoes/
Socks/Laces
NaturVet tick
remover
Samsung
Washing
machine
Nike shoes
for women

NaturVet
for Dogs
Samsung
TV remote
22 id
20. Subscription
Prime subscription related queries, annotator to do an Organic search and check for videos under prime. Mark those as Exact and
any other product which is not purchased under prime to be Irrelevant.
nf

Query Store Exact Substitute Complement Irrelevant Exception

hbo max APS All videos under All videos - All non- -
subscription prime -HBO MAX under prime in digital
for prime in organic search organic search videos
members
https://www.ama https://www.a
Co

zon.com/dp/B0B6 mazon.com/d
6PQZRB p/B0916PSH18
Ebt-snap APS All products that All products Explanation: EBT SNAP –
are eligible for that do not Electronic Benefit transfer
either ‘SNAP’, say ‘SNAP’, that allows Supplement
‘EBT’ or both on ‘EBT’ or Nutrition Assistance Program.
the PDP both on PDP
(mentioned in
PDP)
https://www.ama
zon.com/dp/B00
WI0T5AQ

48 | P a g e
Search Relevance: Standard Operating Procedures

More Labeling Nuances

Truncated queries Some of the most difficult queries are partial words or partial titles. If they are issued within an alias
such as books or instant-video, then it’s easier to judge based upon products that begin with that
string. Use Amazon or Google search to see what the auto-complete query options are, and consider

2 l
each of them as valid interpretations for the query. Each of those interpretations may be in different
categories, so ASINs can be judged Substitutes or Complements for any of the interpretations. Do not

56 tia
consider truncations if a word is a product by itself. The aforementioned to be followed only when
the query is an incomplete product title.

Topic Queries for a product sometimes also return ASINs which are books or other media about that
product as a topic. These are secondary material to be judged as Complement that
includes documentaries, video reviews, how-to manuals, and educational materials.
Products which display an image of another product or miniatures are considered Irrelevant unless
the customer has selected an alias like toys or apparel, or added additional terms to the query

84 n
Quantity/ Quality/
Price/Shipping/Condition
e indicating they want a shirt, mug, sticker or other product displaying an image.


If any of these terms appear in a query, they are not used when determining ESCI relevance,
and the query is judged as if those terms were removed. Subjective terms such as ‘cute’
‘cool’ ‘nice’, refurbished, best seller, best, renewed, cheap, free, sexy, awesome, new can be
ignored.
Query is 5kg rice and Offered product is 2kg Rice, the judgement would be : Substitute as
the weightage or unit of the product is considered whereas 2 pack/3 pack rice/book/shirt
(any other product) should be marked as “Exact” as quantity should be ignored.
22 id
Name When a query consists of a person’s name, it is judged differently depending upon what relationship
that person has to the ASIN:

 if they are a primary contributor towards a book, video, piece of music or art, then that
ASIN is Exact, and any ASINs in the same category are Substitute, in case of author name
nf

we check for genres.

 If the ASIN is about them or depicts them, then it is Complement

For people names any media product authored by or about the person is an
(historical figures, Exact
celebrities, authors)
Co

merchandising and related products are Substitutes

merchandise of another person is Irrelevant
Media format [DVD, Vinyl etc.] may match but if the genre
of the asked person is different then the OP is Irrelevant

Name of a music band merchandise of that band- substitute, as the OP is a related

product and can be expected to be bought by fans of the
music bands
music of band of different genre is Irrelevant. Merchandise
of a different band is also Irrelevant.

Name of a singer ( For ex Music album/songs of the singer- Exact

Beyonce)
Multiple artists in the same song- (Beyonce + Shakira
"Beautiful liar)- Exact, in case of multiple contributors, we
consider all as primary contributors

49 | P a g e
Search Relevance: Standard Operating Procedures

Megan Thee Stallion - Savage Remix (feat. Beyoncé)- Exact

Merchandise of the singer- Substitute

Merchandise of Miley Cyrus [another singer] is Irrelevant

2 l
Music of a different genre is Irrelevant

56 tia
William Morris (Name of William Morris was a designer, who designed many types of
a designer) products- furniture, fabrics, stained glass, wallpaper etc, so
any product bearing his designs can be exact. Similarly, if SQ
is an apparel designer and OP is an apparel from that
designer (Exact), but if OP is an apparel from a different
designer or an apparel without designer name then
(substitute). Product type designed by the asked designer
will be narrowed down, for ex jewelry designer, interior

84 ne designer etc.
If the asked product type is not designed by the designer
then Irrelevant. Example: SQ - William Morris Furniture and
OP is Apparel from a different designer.

For a query for a fictional character-

22 id
 the primary book/movie/game that character came from is to be judged Exact
 costume & merchandise- such as posters, toys, clothing- Substitute.
 Other characters from the same book/movie would be Substitute, merchandise of other
character is Irrelevant.
 and books of analysis/commentary on the subject of the fictional character would
be Complement.
nf

John Snow OP- GOT season 1 Exact- If the query is

OP- character’s character’s name, the
figurines or primary
merchandise book/movie/game that
character came from is to
be judged Exact, unless are
Co

also represented in many

types of products, such
as posters, toys, clothing,
any of which can be
Substitute. Other
characters from the same
book/movie would be
Substitute, and books of
analysis/commentary on
the subject of the fictional
character would be
Complement.
Characters from a different
book/movie where the
asked character Is absent is

50 | P a g e
Search Relevance: Standard Operating Procedures

Irrelevant. Other media

items of the actor who
played the asked character
is also Irrelevant

2 l
Kit Harington Game of Game of throne series Exact
Throne

56 tia
Search Category Offered Judgement Rationale
Query product
Deals related queries Deals/Offers aps T-shirt Unjudgable This is an ambiguous query as
of the day the product type is not
mentioned and the customer is
looking for offers/deals. So, we
would mark this as Unjudgable.

84 ne Deals/Offers
of the day

Iphone
apparel

aps
Ladies

Iphone
skirt

13
Exact

Exact
Due to the selected category, we
would not mark this search
query as unjudgable. Instead, we
would mark all the products
belonging to the selected
category (apparel in this case) as
Exact.
As the product type is clearly
22 id
deals of the mentioned in the search query,
day we would not mark it as
unjudgable. Judgment would be
given according to the product
type mentioned in the query.
nf
Co

51 | P a g e