Needs Met Ratings
Full Meets (FullyM)
-The query interpretation and user intent is specific, clear, and unambiguous -There is a specific website, a specific webpage, a specific know-simple fact or other specific result that all or almost all users in the rating locale and user location are looking for -All or almost all users in the rating locale and user location would be completely satisified y the result
Slightly Meets
A ______ Meets result is also assigned to a helpful result for an unlikely query interpretation or intent or be related to the users information need without directly addressing the core of the need.
Moderately Meets (MM)
A helpful result for any dominant, common, or reasonable minor query interpretation/user intent
Slightly Meets (SM)
A less helpful result for a dominant, common or reasonable minor interpretation/user intent OR a helpful result for an unlikely minor query interpretation/user intent
FailsM
A page can have such low Page Quality that it is unhelpful or even harmful, such as gibberish pages or scams. These types of extremely low quality pages should be rated ______ because the Page Quality issues interfere with helpfulness.
Fails to Meet (FailsM)
A result that completely fails to meet the needs of all or almost all users. For example, the result may be off-topic for the query or address a no chance interpretation of the query
Fully Meets (FullyM)
A special rating category, which one applies to queries with clear intent to find one specific result and the corresponding specific result the user is using for
Highly Meets (HM)
A very helpful result for any dominant, common or reasonable minor query interpretation/user intent
landing page
Assign the Did Not Load flag based on the ________, not the result block.
SCRB- the content inside this block should always play a large role in your rating Some Special Content Result Blocks may have links to landing pages. Think whether user would click on link to satisfy their needs -If most users would not click, rate the SCRB based on that block content alone -If some of many users would slick, you consider the helpfulness of both the landing page(s) in addition to the content in the black. In this case, both need to be helpful to justify a high rating Web Search Result Block- A click is required, and you should evaluate the landing page to assign a rating
Difference between a Special Content Result Block (SCRB) and Web Search Result Block
Highly Meets
For most queries _____ Meets is the highest possible rating a result can get
1st example Needs Met : Even though the article is topical, the information is not trustworthy and is potentially misleading or harmful. This result is unhelpful for the query. Page Quality : This is a YMYL topic. This page has no evidence of medical expertise/authority and is not trustworthy. 2nd example Needs Met : This is a very helpful result for the query. Page Quality : This is a YMYL topic. This page is on a highly authoritative medical website and has a lot of reliable and accurate MC. This page is very trustworthy.
For the following Result block, what is the Needs Met and Page Quality Rating for this Query?
Needs Met : This block shows the number of octaves on a piano, not a guitar. Even though this block has a link to a high quality landing page about pianos, the information shown in the block is inaccurate with respect to the query. This result Fails to Meet the user need. Page Quality : There is a lot of helpful MC on the landing page. The page has medium to high E-E-A-T. Medium+ to High is an appropriate rating. Copyright
For the following Result block, what is the Needs Met and Page Quality Rating for this Query?
Both are examples of Highly Meets Explanation the first Query is a Know Simple query. The LP for this web result provides the answer to the query along with a lot of information about the mountain. While the answer can be found in the description of the web result, it is not displayed prominently. the second Query shows a Highly meets result. The LP for this web search offers many Broadway tickets for sale. This is a helpful result
For the following examples, what are the Needs Met Ratings?
FailsM
If the landing page truly doesn't load, assign the Did Not Load flag and rate the page _______ . True Did Not Load pages are useless.
landing
If there is a Page Quality slider, please assign a Page Quality rating based on the _______ page
FailsM
In most cases, pages you flag as Foreign Language should be rated _______, because they cannot be understood by most users in your locale and are therefore useless.
Yes. Moderately Meets - Moderately Meets + Helpful interpretation and intent to know what states border Virginia. However not very helpful as it lacks visual information (e.g., map)
Is this an example of Moderately Meets?
● The task language ● A language that is commonly used by a significant percentage of the population in the task location ● English
Please do not assign the Foreign Language flag when the language on the landing page is one of the following: 3 instances
FailsM and yes it does
Query: [toys] User intent : find toys to buy Landing page: Warning this a porn page What Results Need rating does this get and does it get a porn flag?
The Needs Met rating
The ____ met rating is based on both the query and the result
Highly Meets rating (HM)
The _____ rating is not appropriate if a page is untrustworthy or has any other undesirable characteristics, such as outdated or inaccurate information. There are very high standards for the _____ rating.
Highly Meets
The following characteristics (in image) are an example of what Needs Met?
Fails to Meet
The following types of content should be rated _______ to meet if it is clear that the user is not looking for such content -Harmful to self or other individuals -Harmful to Specified Groups -Harmfully Misleading info -Untrustworthy -Spammy -Porn
Moderately Meets
These examples are what type of Needs Met?
False This is a Moderately Meets. Its a cute short video of a baby emperor penguin walking to a fully grown emperor penguin This is a helpful result for the reasonable interpretation and intent to find cute photos/videos of emperor penguins. However the result is not very satisfying or very helpful for the query.
This is an example of Highly Meets results. True or False
True
True or False Pages that provide a poor user experience, such as pages that try to download malicious software, should also receive low ratings, even if they have some images appropriate for the query
T SM results exist in all formats (video, audio, image, text) and of many types of websites: professional websites, corporate websites, forums, social media, etc
True or False A query can have many Slightly Meets results
False A query can have many Highly Meets results
True or False A query cannot have many Highly Meets results
False This would be a rating of Slightly Meets
True or False A rating of Moderately Meets is also assigned to a helpful result for an unlikely query interpretation or intent or be related to the users information need without directly addressing the core of the need.
True.
True or False Accuracy is required for Highly Meets informational results. Highly Meets information pages on YMYL must be accurate and trustworthy.
False flags like "Porn," "Foreign Language," and "Did Not Load" are based on the content of the result itself and are independent of the query. These flags assess whether the result contains inappropriate content, is in a different language than required, or fails to load. They do not consider the user's query or intent, focusing solely on the nature and functionality of the result provided.
True or False All flags (Porn, Foreign Language, and Did not Load flags) should be based on the result alone and do not depend on the query
False FailsM results may be off topic for any query interpretation
True or False Fails to Meet results are never off topic for certain types of query interpretations
True
True or False Fails to Meet results have some aspect that causes them to be unhelpful for the query , such as lack of attention to an aspect of the query (or user location) that is critical for satisfying the user intent
False
True or False For these instances, use a Did not Load Flag ● Malware warnings, such as "Warning - visiting this site may harm your computer!" ( example ). ● Pages that have removed or expired MC (e.g., expired classified listing, removed social media post, products or services unavailable). ● Pages that are inaccessible because you need a subscription to view the MC.
True
True or False If the user intent is clearly not porn-seeking, a landing page that has a porn for its MC should be rated Fails to MEet
True
True or False If you are unable to evaluate the Page Quality rating of a Foreign Language result, you do not need to assign a Page Quality rating and can leave the slider at N/A .
False When rating BCNM, landing page content should not factor into the BCNM ratings.
True or False It's ok to click on the landing pages when rating Block Content Needs Met
True
True or False Moderately Meets are NOT out of date or inaccurate
True **most queries do not specify a single website or webpage. Most queries are not specific enough to have a single, know-simple fact as a comprehensive answer
True or False Most queries cannot have a Fully Meets result
True Slight Needs results may simply be less helpful "fit" for the particular query in the rating task, but there is nothing "wrong" with SM results
True or False Often there is "nothing" wrong with Slightly Meets results
False Do not rate all porn queries as MM or HM. Even though the query is porn and the result is porn, the page must fit the query and be helpful to get a high Needs Met rating
True or False Rate all porn pages for porn queries as MM or HM
True
True or False Slightly Meets results may have some aspect that prevents a higher rating, for example they may have outdated information
True If a result block has no Page Quality rating slider, you do not have to give a Page Quality rating.
True or False Some results don't have a Page Quality slider.
False The Page Quality rating slider does NOT depend on the query. Do not think about the query when assigning a Page Quality rating to the LP.
True or False The Page Quality rating slider depends on the query
False Always assign the Porn flag to all porn pages, whether the query is porn seeking or not. The Porn flag does not depend on the query or the user intent
True or False The Porn flag should only be assigned when the query is not porn seeking.
False They are both too broad of information queries and it is impossible to know exactly what the user is looking for
True or False The example query [knitting] or [barack obama] can still qualify for a Fully Needs Met (FullyM) rating.
False This would be a Highly Meets - Highly Meets+ rating. The intent is to find coffee shops near the specific user location, represented by the blue dot on the map. The result block has a very satisfying list of nearby, popular, and prominent options within a few miles of the specific location. This kind of block is especially helpful for users who want to visit the business in person. Also, the blocks are interactive. Click on the coffee shop will give options to call the business, get directions, read reviews, etc. Note: This result is not Fullym even though this is a very helpful result block for users seeking nearby coffee shops, it is unlikely that every user would be fully satisfied with this specific list given the abundance of coffee shops near the user location
True or False This is an example of a FullyM rating.
True
True or False Use in-between ratings if you think the rating of a result falls between two labels
False For many queries, a single user may need to see many different results
True or False Users never need to see many different results
False. If some or many users would slick, you can consider both the landing page(s) and the content in the block to justify the users need rating.
True or False You never need to consider the helpfulness of the landing pages(s) in addition to the content in the SCRB.
True
True or False All result blocks must be given a Needs Met rating
True
True or False If you are unable to evaluate the Page Quality rating of a Did Not Load result, you do not need to assign a Page Quality rating and can leave the slider at N/A
FailsM
Useless results should always get a _____ meets rating, even if the landing page has a high Page Quality rating
Fully Meets (FullyM)
What Needs Met Rating is this describing? All or almost all users in the rating locale or user location who search for this query would be looking for the same, specific fact or piece of information. The Result block provides the information thoroughly, accurately, and clearly, and is from a trustworthy source acceptable for all or almost all users
FailsM The translation in the result block is completely wrong and inaccurate. Mañana means "tomorrow," not "wedding."
What Needs Met Rating would this query and result get?
Fully Meets (Fully M)
What Needs Met Rating would this query and result get?
Fully Meets (FullyM)
What Needs Met Rating would this query and result get?
FailsM A news result is completely unhelpful for this query. This result Fails to Meet the needs of the user.
What Needs Met rating does this query and result get?
FailsM Users issuing this query want to go to the Amazon website. Showing information about how to visit or call the corporate office for an online company would not be helpful. At the time this example was written, the Amazon corporate office was not open to the public; very, very few people ever go to an Amazon corporate office. If someone needed to go to the office for an interview or business visit, they would need to get that information from someone at the company or would use a more specific query.
What Needs Met rating does this query and result get?
FailsM The LP is a humorous blog post about a wife helping her husband to buy a suit. The page mentions "American Express," but is insufficiently related to the query to be helpful for users, so it Fails to Meet the user intent. Note: Even if this blog post were more recent, it would still fail to meet the user intent. This LP has gibberish text. Read this sentence: "How do you make a cigarette symbol on the keyboard? In." This result is unhelpful for the query.
What Needs Met rating would these two examples get?
FailsM The answer in this block is about the cost of education, not salary, which is misleading and doesn't answer the user's query.
What Needs Met rating would this query and result get?
Fully Meets (Fully M)
What Needs Met rating would this query and result get?
Fully Meets (FullyM)
What Needs met rating would this query and result get?
Fully Meets (Fullym)
What Needs met rating would this query and result get?
Highly Meets The user intent is too broad to have a Fully Meets result. However, this is Celine Dion's official website and has a lot of content including recent news, show information, music, photos, videos, etc. This is a very helpful result
What is the Needs Met Rating for the following?
Highly Meets + Explanation: This is the official website for the Trader Joes, which has information on the businesses' stores, promotions, recipes, news, etc. Note: This result block is not FullyM because users who want information to visit the store have to do a bit of work to find that information- it is not immediately available
What is the Needs Met Rating for the following?
Highly Meets + The result block shows information about the museum, and is especially helpful for users who want to visit the museum note: this result block is not FullyM, even though there is a website link, it is not very satisfying for users who want to go to the website because it is not as informative as a web search block with a title, snippet, and URL
What is the Needs Met Rating for the following?
Highly Meets + Explanation -The result shows a complete list of all three locations in the Charlotte area, with info that is very helpful for users who want to visit the store. -Note: This result block is not Fully Meets because users who want to go to the website to see coupons, promotions, etc. would have to see additional results
What is the Needs Met rating for the following?
Slightly Meets (SM) The LP of this web result is about the movie "Hot Dog," an obscure film which came out in 1984. Even though the landing page has good content, this is an unlikely minor interpretation of this query. This result block does not contain enough information to be fully satisfying and mentions only one type of shark. This is not a very helpful result.
What is the Needs Met rating for these two examples?
Fails to meet + or Slightly Meets
What is the Needs Met rating for this query and result?
Slightly Meets (SM) Although the LP has comprehensive information from a reputable source, the information is about the 2010 Honda Odyssey. This information would be considered stale for the query today, making this result less helpful for users.
What is the Needs Met rating for this query and result?
Slightly Meets (SM) The LP of this web result has a 2006 article about Britney Spears filing for divorce. This is very old, stale news, making this result less helpful for users.
What is the Needs Met rating for this query and result?
Special Content Result Block (SCRB)
What kind of result block is this?
FailsM This LP has information about tooth loss in pike fish and has the words "five years old" on the page. This result Fails to Meet the user intent because it has keyword matches only and is unhelpful for the query.
What needs Met Rating does this query and result get?
FailsM This is a broad query and it is unclear exactly what the user is looking for. This result block shows visit-in-person information for two businesses that contain the name Mike. However, the query is very broad and it is very unlikely these businesses are what users seek given the query.
What type of Needs Met is this query and result?
FailsM This result completely fails to satisfy the user intent, which is to find weather information for Paris, Texas, and not Paris, France.
What type of Needs Met is this query and result?
FailsM This result shows information for a church, an event venue, and the closest airport. However, users clearly want to know details about a specific flight, and there is absolutely no information in the block about the flight. This result Fails to Meet the needs of the user.
What type of Needs Met is this query and result?
FailsM This result block shows a go kart arcade in Warrington, Pennsylvania, and two other go kart arcades in the surrounding area. These arcades are places to go ride go karts, not purchase them, so the result completely Fails to Meet the user intent .
What type of Needs Met rating does this query and result get?
Special Content Result Block
What type of result block is this?
Web Search Result Block
What type of result block is this?
1st example FullyM and yes flag it as porn 2nd example FailsM and yes flag it as porn
What would the Needs Met rating be for both examples and should they get a porn flag?
MM to HM yes assign a porn flag
What would the Needs Met rating be for both examples and should they get a porn flag?
lower
When in doubt(whether to use FullyM), use a _______ rating
False These are non-porn intent queries
[boys speedos] [girls] [gay people] [pictures of girls] [cheerleaders] These are examples of porn intent queries True or False
Landing Page Needs Met
_____ Page Needs Met should be based exclusively on the landing page(s) links in the result block.
Fails to Meet (FM)
______ Meet rating should be assigned to results that fail to meet the needs of all or almost all users.
Slightly Meets
______ Meets assigned to results that are less helpful for a reasonable query interpretation or user intent.
Moderately Meets (MM)
______ meets have fewer valuable attributes than Highly meets results
Highly Meets (HM)
_______ Meets results are highly satisfying and a good "fit" for reasonable interpretations and intents.
Moderately Meets (MM)
_______ meets is assigned to helpful results for any reasonable interpretation or user intent of the query, including reasonable minor interpretations and intents
Block Content Needs Met ratings
_______ ratings should be based exclusively on the content inside the result block. This applies to both SCRBS and Web Search Result Blocks.
Did not Load flag
________ flag is used to indicate technical problems with the webpage that prevent users from viewing any LP content
Fails to Meet (FailsM)
________ to meet results may be helpful for a no chance interpretation. They may also be off-topic for any query interpretation.
FailsM and yes it does get a porn flag
query: [breasts] user intent: find anatomy or health information about breasts landing page goes to a porn page What type of Needs Met rating does this get? Does it get a porn flag?
Highly Meets (HM) and no it does not get a porn flag
query: [breasts] user intent: find anatomy or health information about breasts landing page goes to page showing anatomy of breasts What type of Needs Met rating does this get? Does it get a porn flag?
