A team of over 10 thousand people, active worldwide and engaged in a specific task: evaluate the results provided by Google in the light of the guidelines provided by the company, to find out whether the SERPs actually meet the criteria reported and whether, therefore, users receive an effective and positive service. Today we try to find out more about Google Quality Raters and the documents they must refer to, to understand what they really do, debunk some myths and, no less importantly, understand how their work affects Research and what you need to know about the concept of EAT.

Who are the Google Quality Raters

It was at least the 2005 when Google launched the quality raters program, involving a small army of testers, evaluators and quality reviewers in the evaluation work of its final product, namely the SERPs shown to the public, the pages containing the search results.

According to rumors, today the company indirectly employs about 10 thousand quality raters worldwide, paid per hour (13.5 dollars, it seems), through a network of contracting companies. These external collaborators of Google are hired through temporary employment contracts, which can be renewed but which, in general, never last very long.

How can you become a google evaluator?

There is obviously a good deal of secrecy around this program, and there is no way to submit your quality rater application or an office or email to which you can send a resume: as far as you can expect, it is Google that directly contacts the people it identifies as potential evaluators considering them up to the task (sometimes bloggers scattered around the world), or delegates in subcontract the search to specialized agencies.

The short duration of the contracts has a very specific ratio: it prevents the quality raters from being able to interfere in some way with the research system going beyond their duties or to take advantage of their position.

In these 15 years, the Google program has employed millions of Qrs, which have been the external and human eyes to monitor the quality of the Serps and the results provided.

The duties and functions of Google Quality Raters

So, what do these Google evalutors do? We can think of them as a group of quality auditors, who evaluate a company’s results on the basis of criteria, principles and documents provided by management.

In the case of the search engine, GQRs are in charge of analyzing and studying the information contained in the query results using the famous Google guidelines, the General Guidelines of the Search Quality Rating Program that are published online, publicly available and constantly updated.

Google quality raters do not determine penalties, downgrades or bans

It is good to immediately clarify one aspect: Google’s project contributors have no access or control to any component of its algorithms and do not directly determine penalties, bans or drop of rankings for sites.

Their function is not to decide the positions of the results in SERP, but only to verify that the product – that is to say the search algorithm – is working in the expected way and according to the established rules.

That said, with their judgment the quality raters can still indirectly influence the organic positioning of the pages, as they analyze the quality of the contents and assign a human assessment that will then be interpreted and elaborated by the appropriate Google teams.

If we want to summarize it, we can say that they do not influence the ranking of the sites they evaluate, but they influence the rankings of each site compared to the queries being evaluated.

The work of human evaluation of the SERPs

Each quality rater has the task of verifying the quality of a group of Serps and then controls a series of web pages (usually, those best placed for the most delicate queries); its task is to judge, based on a precise check-list, if the document complies with the guidelines on the informative and technical quality of the pages.

Once completed this review, the quality rater assigns a quality rating, on a minimum to maximum rating scale: it is not a subjective or personal assessment, but the margin of discretion is very limited and the reviewer must strictly comply with the aforementioned guidelines drawn up by Google. These data are then provided to machine learning systems, which use them to improve algorithms based on known factors.

Thanks to this work, Google can identify disinformation elements that can escape automated algorithmic systems. The mission of quality raters is therefore to contribute, through a human judgement, to make the pages of research results useful and high quality, reducing the presence of misleading results and content not up to the standards.

Google’s guidelines for quality rater

At this point it becomes important to understand what these guidelines are and what they refer to, a document that to the latest version (currently updated to December 5, 2019, as we analyzed in this article) consists of 168 pages that scan everything that means quality to Google.

As we read at the beginning of the document, “the general guidelines concern mainly the evaluation of the quality of the page (PQ) and the assessment of the needs (NM)”: in practice, the quality raters have the task of expressing an opinion and a vote on the way in which the search engine responds to search intent (needs) and required quality levels.

The interpretation of the usefulness of results

In a recent study of Searchenginejournal, Dave Davies gives us some useful information about the work of quality raters and in particular about the evaluation process, which is probably based on the question “How useful and satisfactory is this result?”.

During the test, an evaluator can visit a single web page or focus on an entire SERP, analyzing each result placed. In both cases, it will send signals to Google about the structure of the site, the device, demographic differences and location results, plus a number of other factors that apply to the classification of each result.

These data will guide changes to improve results and algorithmically determine which factors or combinations of factors are common to results with higher rankings. The evaluation of needs met requires at least decent page quality and is based on both the query and the result.

Human work is also useful in the interpretation of ambiguous queries, such as those that have multiple results: in these cases, the NM score must give more weight to the pages that satisfy the highest and most sought-after intents, so as to prevent the high positioning of pages dealing with topics that do not correspond to the general search intent and the sending of wrong signals to algorithms, which therefore can focus on the right signals for most users.

The evaluations of the page quality

In fact, and as it was easy to guess, the indications to assess the quality of the pages provided to human testers do not differ much from the best practices that classically refers to the sites: the evaluations are based on a series of factors, all linked to each other, and the weight attributed to each factor is based on the type of site and queries.

There are some types of topics and categories that are under a special magnifying glass, and in particular the most thoughtful one is the sector Your Money or Your Life (YMYL) that, Google’s word, “potentially affects a person’s future happiness, health, financial stability or security”. In this case, raters need to take more care and are advised to give more weight to the EAT.

Google’s indications on YMYL pages

The guidelines divide the YMYL content into 7 groups of sites, depending on the main type:

  1. Current news and events, including news on important topics (such as international events, business, politics, science, technology), except those that do not concern YMYL issues (generally, sport, entertainment and everyday topics are not considered sensitive).
  2. Civic, government and law, which refers to important information to maintain an informed citizenship and therefore concerns information about voting, government agencies, public institutions, social services and legal issues (such as divorce, children custody, adoption and so on).
  3. Finance, for all content related to financial advice or information on investments, taxes, pension planning, loans, banking or insurance, and in particular for web pages that allow people to make purchases or transfer money online.
  4. Shopping, which groups information or services related to the search or purchase of goods / services, in particular Web pages that allow people to shop online.
  5. Health and safety, that is advice or information on medical problems, drugs, hospitals, emergency preparedness, hazardous activity etc.
  6. Groups of people (a topic introduced with the September 2019 update), for all information or statements related to groups of people, including but not limited to those grouped according to race or ethnic origin, religion, disability, age, nationality, veteran status, Sexual orientation, gender or gender identity.
  7. Other, macro-category for all other topics related to important decisions or important aspects of people’s lives that can therefore be considered YMYL, such as fitness and nutrition, home information, choice of a college, job search and so on.

The division of the site’s contents

According to the guidelines, the sections of a website can be classified into three main categories:

  • Main content (MC): any part of the page that directly helps the page achieve their purpose.
  • Supplemental content (SC): additional content that contributes to the user’s good experience on the page, but does not directly help the page to achieve its purpose. The example provided by the document are navigation links: an essential element for the site, but not necessary to meet the needs of visitors.
  • Advertisements (Ads): Advertising or monetization (Google Ads) are content and/or links that are displayed on page for the purpose of monetizing, or receiving money.

The ease of access and the volume of the main content do their part in the calculations on the quality of the page: it is what helps the rater to evaluate not only if the needs/ intent are met, but also if and how easy it is to access the additional content, if you wish to.

The focus on the E-A-T

The section on the EAT paradigm is one of the most complex and discussed, and Googlers have often intervened to provide clarification and guidance on these parameters (such as Gary Illyes in the linked article).

The first point to understand is that Expertise, Authoritativeness, Trustworthiness, the acronyms of EAT are not ranking factors on Google.

These are the parameters that quality raters look for and use to orient themselves in the evaluation of websites and to understand if Google’s systems work well in providing good information, but they are not part of any algorithm.

The operation is therefore as follows: raters use EAT principles to judge websites and Google uses their ratings to adjust its algorithm. Then, at the end the algorithm will align itself with EAT principles, which may be useful as a guiding principle in the design of the site, in the creation of content and in the support to external signals.

There is no specific optimization we can do for these parameters, but we can still work to improve the (complex) way in which Google sees, interprets and evaluates our site and our pages.

Call to action