2/3/2021 Guidelines for Search - Video Hints — BaseLine

Search - Video Hints

Guidelines for Search - Video Hints

Introduction
In this document we will explain how to evaluate video search hints.

What is a search hint?

A hint is a suggestion that the search engine is offering to the user as they start typing their query.
The hint is not a search result, It is a query by itself.
Examples:
Input: 'ab' -> Hint: 'abraham'
Input: 'the' -> Hint: 'the morning'
Input: 'the' -> Hint: 'the simpsons'

3 l
Input: 'dis' -> Hint: 'disney'

25 ia
Hint types
For video search we offer several types of hints:

09 nt
Content title
A search hints that suggest a specific movie or TV show
Input: 'cin' -> Hint: 'cinderella'
Input: 'lit' -> Hint: 'little einsteins'
20 ide
In case of a series the hint suggested can point to part of the title and not a specific title.

Input: 'st' -> Hint: 'star wars'

Person - Cast and Crew

A search hints that suggest a a person. It could be an actor or actress or any other cast and crew member.
nf

Input: 'leo' -> Hint: 'leonardo dicaprio'

Input: 'que' -> Hint: 'quentin tarantino'

Other hint types

Co

A search hints that suggest video related terms like: Brand, Genre, Franchise Name and Award

Input: 'an' -> Hint: 'animation'

Input: 'os' -> Hint: 'Oscar'

A search hints that suggest a combination of video related terms like: Brand + Genre, Brand + Content Type,
Genre + Content Type, Brand + Genre + Content Type

Input: 'an' -> Hint: 'animation'

Input: 'os' -> Hint: 'oscar'

The purpose of video search hint evaluation

The purpose of this evaluation is to evaluate the relevance of a search hint for a given input.
We want to ensure that the hints add value to the users in their search journey and the hint is relevant to the
input given by the user.

Bear in mind that the hints are given in the context of video search and are supposed to help users find video
content.
For example:
Input: 'dis' -> Hint: 'disney' - This is a a relevant hint as Disney is video related query.
Input: 'dis' -> Hint: 'disapprove' - While this is a relevant hint, 'disapprove' has no video context,
hence, this is not considered a good hint.

A video search hint should help the user find a TV show, a movie, an actor/actress or any other video related
content.

https://baseline.apple.com/evaluations/97/guidelines 1/4
2/3/2021 Guidelines for Search - Video Hints — BaseLine

Use your best judgment and try to assess if the hint can used as the primary or secondary intent for the given
input in the context of video search.

To help you find out if a hint has video context and can be the primary or secondary intent of the input we
added links to the task page that will point you to popular search and video content websites.

E3696F23-639A-437D-BB52-07C48FF21CF7_4_5005_c.jpeg · 22.6 KB

For example:

3 l
Input: 'bar' -> Hint: 'bar rescue'

25 ia
Clicking on the IMDB link https://www.imdb.com/find?q=bar+rescue will reveal that there is a TV series
from 2011 called 'Bar Rescue', Hence this hint has a relevant video related context.
Input: 'bar' -> Hint: 'barkskin'
Clicking on the Google play link https://play.google.com/store/search?q=barkskins&c=&gl=us&hl=en will
reveal that there is a TV series from called 'Barkskin', Hence this hint has a relevant video related context.
09 nt
Rating Scale
20 ide
The grades on the relevance scale are as follows:
Perfect
Good
Acceptable
Unacceptable: Extra Language
Unacceptable: Spelling
Unacceptable: Other
Problem: Other
nf

When choosing your rating take into consideration the following

Relevancy to the input
Recency of the video content hinted
Popularity of the video content hinted
Co

Primary and Secondary intent

The primary intent of a query is the most likely intent based on relevancy, recency and popularity on the
content in the market you’re evaluating.
A secondary intent, on the other hand, is less likely, less popular and/or less recent intent that is relevant to
the input.

Examples for different ratings

Perfect
A suggestion that very likely satisfies a common primary intent of the input.
Perfect suggestions complete the query, are mainstream popular, add value, and 'make sense' given the length
of the string.
Examples of Perfect hints/top hits:
'grea' → 'grease': High relevancy. Popular content. Very likely to satisfy a primary intent.
'grea' → 'grease 2': High relevancy. Popular content. Very likely to satisfy a primary intent.
'grea' → 'the greatest showman': High relevancy. Popular content. Very likely to satisfy a primary intent.
'star' → 'star wars: the clone wars': Popular and very likely to satisfy a primary intent.

You may encounter cases where the hint is identical to the input, The rating for these cases should be perfect

'glee' → 'glee': Popular and very likely to satisfy a primary intent.

Good
https://baseline.apple.com/evaluations/97/guidelines 2/4
2/3/2021 Guidelines for Search - Video Hints — BaseLine

A suggestion that is relevant and likely satisfies a secondary intent of the input.
These hint are pointing to
Not popular and / or not recent content
Slightly specific - The hint suggested are specific for highly ambiguous input.
For example : 'be' → 'Beverly Hills 91201': Very specific for the input 'be' as the intent could be a lot of other
content.

Examples for hints rated Good

'under' → 'under the dome': Popular but not recent. Likely to satisfy a secondary intent.
'atlanta' → 'the real housewives of atlanta': Long reality TV series. Not recent and so popular. Likely to
satisfy a secondary intent.
'defend' → 'Defendor': Strong relevance to the input query. Popular but not recent. Likely to satisfy a
secondary intent.
'food' → 'food paradise': Popular but not recent. Strong relevance to the input query. Likely to satisfy a
secondary intent.
'I lo' → 'I love Lucy': Classic, Popular but not recent. Strong relevance to the input query but very specific.

Acceptable

3 l
A suggestion that may be incomplete, too vague, too specific and unpopular

25 ia
Too Specific/Unpopular
Suggestions that are not recent, unpopular, incomplete, too specific or don't have strong relevance to the input
query.

Examples of Acceptable hints/top hits:

09 nt 'archer' → 'judge archer': Low relevancy to the input. Not recent and popular content.
'atlanta' → 'little women Atlanta': Low relevancy to the input. Not recent and so popular. Not likely to satisfy
primary or secondary intent.
'defend' → 'range defendor': Low relevancy to the input. Not recent and not popular. weak relevance to the
20 ide
input query.
'faceb' → 'why I'm not on Facebook': Low relevancy to the input. Not recent and not popular.

Unacceptable: Extra Language

A suggestion that includes extra language that is obvious or redundant.

'grey' → 'greyhound movie': The word movie is redundant and is not required as part of the hint.
nf

Unacceptable: Spelling
A suggestion that contains spelling errors, spacing errors, punctuation errors, etc.
Co

Should you be unsure if a suggestion should fall into this category, it is better to be more critical and use this
rating than not.

Note: This case in video search is very rear as most hints are using movie and TV shows names.

Unacceptable: Other

Output is irrelevant to the input

Any other reason that makes the hint not acceptable.

Problem: Other
Use this rating only if there is a problem with the task itself in BaseLine. For example, there's no output to
judge.

https://baseline.apple.com/evaluations/97/guidelines 3/4
2/3/2021 Guidelines for Search - Video Hints — BaseLine

Copyright © 2021 Apple Inc. All rights reserved. Apple Confidential.

3 l
25 ia
09 nt
20 ide
nf
Co

https://baseline.apple.com/evaluations/97/guidelines 4/4