A Guide to Determine if a webpage relevant to a query

ATTENTION! This task requires the MANDATORY use of a US VPN. Since some pages may not
open without this. If while working you come across a page that does not open (page not found,
error 404 and others), try opening the page using a VPN.
Remember, we carefully monitor the quality of our projects and check expert assessments.
Only if the page does not open with or without VPN, you should mark the task as Page loading
error.

Your task is to decide if the webpage is relevant to the query and

what is a degree of relevance.
How to perform the task

1. Review the user query. Make sure you understand what they were looking for. If the query is
unclear, use the Google it button
2. Check if the query is time-sensitive (such queries require the most updated and recent answer)
3. If the query is time-sensitive, check the webpage:

4.

a. Follow the link to view the webpage that will be assessed. Please review it carefully and
make your rating decision based on these instructions only.
b. If the webpage is up-to-date, contains detailed, relevant information and the website is
trustworthy, then choose Excellent
c. If the webpage is up-to-date, relevant but has a lack of details or the website is not
trustworthy, then choose Good
d. If the webpage is outdated or irrelevant, then choose Not relevant.
5. If the query is not time-sensitive, check the webpage:

6.

. Follow the link to view the webpage that will be assessed. Please review it carefully
and make your rating decision based on these instructions only.
a. If the webpage contains detailed, relevant information and the website is trustworthy,
then choose Excellent
b. If the webpage is relevant but has a lack of details or the website is not trustworthy, then
choose Good
c. If the webpage is irrelevant, then choose Not relevant.
7. If the webpage doesn't load then choose Page loading error (Only if the page does not open
with or without a US VPN)

How to assess web page relevance

Answers options
There are 4 answer options that you can choose to access webpage relevance to the search query.

Excellent
Generally, to be Excellent the webpage must provide accurate and detailed information and the site must
be trustworthy. Firstly you need to assess the time-sensitivity of the query and then the relevance,
actuality and trustworthiness of the answer.
A really relevant webpage should contain the following attributes:

 Detailed Explanations: a relevant web page should offer comprehensive and detailed
explanations related to the query. It should go beyond surface-level information and provide
in-depth analysis, examples, and relevant details that help answer the query thoroughly.
 Balanced and Objective Viewpoints: look for a balanced presentation of different viewpoints
or perspectives on the topic. Reliable content often acknowledges varying opinions and
provides a fair analysis without undue bias or manipulation.
 Directness to the topic: the content is highly relevant to the search query, addressing the
topic directly.
 Matching the title of the web page to the topic: the title of the web page closely matches the
search query.

Remember that no single attribute can guarantee the accuracy and detail of a web page, and at the same
time, the absence of individual attributes does not always mean that the description on the page is not
accurate. It is necessary to critically evaluate the information and make a decision taking into account the
described factors.
Good
You must answer Good if ANY of the following is True:

 The web page provides information relevant to the search query but does not meet all the
criteria for an Excellent result
 The webpage lacks authority, expertise or reliability.

The two major differences between Good and Excellent answers are:

1. Excellent articles contain a detailed response to the query and are trustworthy. Excellent results give
us full results and allow us to stop looking for more information.

2. Good articles can contain the answer, but not all the detailed information or not come from reliable-
looking sources to convince the reader (say to prevent from even looking for more information).
You can also ask yourself the question: "Would I be satisfied if I received this page for such a query?".

 If you answer "Exactly yes, this is the best page to answer my query", then answer
"Excellent".
 If you answer "The page is good, but it could be better", then answer "Good".

Not relevant
You must answer Not relevant if AT LEAST ONE of the following is True:

 The webpage does not contain information related to the query and provide off-topic content
 The query is time-sensitive and webpage is NOT up-to-date

One of the most probable reason for irrelevance is out-of-dateness of the webpage:

 If the query is not time-sensitive, you should consider the webpage as up-to-date.
 If the query is time-sensitive you must check if the webpage is up-to-date and still relevant.
  If the query is time-sensitive and the webpage is not up-to-date and already not relevant to
the query, you should select " Not relevant".

Page loading error

If the page doesn't load, try to open it using a VPN set to the United States. Choose 'Page loading error'
only if it still doesn't load with a US VPN. This step is important for quality checks; skipping it could limit
your access to tasks.
Avoid giving 'good' or 'excellent' ratings to non-loading pages without proper verification.

Examples

Query Web Right Comment

Article answer

how to get an emotional Web This is a highly relevant article with detailed explanation from a
Excellent
support animal cat license article link The article is in the blog section, but on the website of a well-kn

Web
jonas name meaning Excellent This is a highly relevant page with a lot of useful details on a tru
article link

Web
george clooney tequila Excellent This is a highly relevant page with a lot of useful details on a tru
article link

Web This is a relevant article but the website is not trustworthy, inclu
men how to measure shirt size Good
article link products.

how many chill hours in Web Article has relevant info (possible answers), but no clear answe
Good
sacramento article link for more information.

how to avoid carbon buildup on Web

Good Article has relevant but there are a lot of advertisements and the
lionel fastrack article link
when does year of the dragon Web
Good Article is not fully related to the topic and the website doesn't loo
start article link

why is the stock market up Web Not The query is time-sensitive and requires a fresh up-to-date answ
today article link relevant last update was on JUL 25 2023.

Web Not The article doesn't answer the question "how old is jon stewart"
how old is jon stewart
article link relevant an issue about Joe Biden's age, not his own.

who is playing in tunnel in Web Not Article is absolutely irrelevant. Tunnel in milan is a club and that
milan article link relevant schedule, website of this club, but results is a wiki page about tv

Some important definitions

Trustworthy websites
The website is trustworthy if we can trust the information that website contains.
For trustworthiness, we have to use the general knowledge about reputation of sources for different
topics, e.g., news websites may not be the best source for a medical topic, whereas WebMD is.
If the general knowledge based check is not possible (because we may not know which websites are
good for all the topics), then we can study the page:

1. Too many ads on a website, especially those blocking content, can suggest it's unreliable.
However, a few banners that don't interrupt browsing don't affect a site's trustworthiness.
Websites with non-intrusive ads still can be evaluated as "Excellent".
2. Readability of text also indicates to us the quality of a website.

Examples of trustworthy sites:

 BBC News (https://www.bbc.com/news), The New York Times (https://www.nytimes.com),

National Geographic (https://www.nationalgeographic.com) , etc. - widely recognized and
respected media.
 Wikipedia (https://www.wikipedia.org/) - a widely used online encyclopedia that contains a
vast amount of information on a wide range of topics.
 Stanford Encyclopedia of Philosophy (https://plato.stanford.edu) - online encyclopedia that
provides authoritative and scholarly articles on various philosophical topics. Its association
 with Stanford University contributes to its reputation as a trustworthy source for philosophical
information.
 https://www.masterclass.com, https://www.allrecipes.com, https://www.travelandleisure.com
- niche, but good-looking and trustworthy sites.

Examples of not trustworthy sites:

 User-Generated Content Platform (blogs, forums, Twitter, other social media, etc) - certain
subreddits are known for spreading misinformation or conspiracy theories. Since anyone can
contribute content, the credibility of the information varies greatly.
 Pseudoscientific Websites - for example websites that promote alternative health practices
without sufficient scientific evidence.
 Unreliable Personal Blogs or Unknown Websites - personal blogs may lack credibility and
authoritative expertise. Authors may claim to be for example a health expert but do not
provide verifiable credentials or references to support their expertise. But not all blogs are
not reliable. For example, this is a personal blog and we can't trust the information that the
author gives us. But another example is the blog of a well-known recognized company and
we can trust information in that blog.
 Satirical News Websites - these websites publish satirical articles that are meant to entertain
rather than provide factual news. While they may resemble real news stories, their content is
intentionally exaggerated or fabricated for comedic purposes.
 Misinformation and Conspiracy Websites - These websites are known for spreading
conspiracy theories, misinformation, and unsubstantiated claims. They often promote
sensationalized narratives and engage in fear mongering without reliable evidence or
credible sources.

Time-sensitive queries
Time-sensitive queries are queries that require the most updated and recent answer. In other words,
time-sensitive queries require the most fresh and last answer.

Examples of time-sensitive queries:

 "Stock market trends" - this query is time-sensitive because stock market trends change
frequently. The relevance and accuracy of information regarding stock market trends are
highly dependent on the current market conditions and can vary significantly from day to
day.
 "Local events" - this query is time-sensitive as the relevance of the information is time-bound
because events occur on specific dates and often have limited availability.
 "News update on XYZ incident" - this query is time-sensitive because it involves obtaining
real-time updates on a recent incident or event. News queries require immediate and up-to-
date information.
 Requests about the weather, stock prices, various posters and other information that
requires dynamic updating.