We Asked ChatGPT to Audit Google’s Search Quality Guidelines – This is What it Said.

For those of you who are not familiar, Google has a 170-page “Search Quality Evaluator Guidelines” PDF. They started to publicly release this document in 2014. In 2018, they made some major updates to the original document, and since then, Google has been making minor updates every couple of years. The latest version is dated March 5th, 2024.

For a little more background, Google used to hire human Search Quality Raters to “score” web page content. Naturally, these folks needed a guide and system to do their jobs, and this PDF was the reference book for employees to do their jobs objectively.

Fast forward to 2024, and Google no longer really needs humans to grade and score content. AI learning and chatbots have taken the place of these humans, but their guidelines still remain the same. So, I thought it would be interesting to give ChatGBT a copy of the PDF and share its responses to some direct SEO questions.

Q1: What do “fully meets” quality rating pages have vs. pages that don’t meet the rating?

ChatGBT: Pages that receive a “Fully Meets” rating versus those that do not meet the rating scale differ significantly in how well they satisfy the user’s query intent. Here’s a breakdown of the differences:

“Fully Meets” Rating:

• Perfectly Satisfies User Intent: Pages rated as “Fully Meets” provide the exact information the user is seeking with no need for additional results. These pages are the best possible match for the query and fulfill the user’s needs completely.
• High Relevance: The content is highly relevant to the search query and directly answers the user’s question or provides the exact service or product they are looking for.
• User Satisfaction: A user who lands on a “Fully Meets” page should feel that their search is over. There is no need for them to click on any other result.
• Clear and Accessible Content: The content on these pages is usually well-organized, easy to read, and free from distractions like excessive ads or irrelevant links.
• Credibility and Trustworthiness: The information provided is from a reliable source, and the page demonstrates a high level of E-E-A-T (Experience, Expertise, Authoritativeness, and Trustworthiness).

Pages That “Don’t Meet” the Rating Scale:

• Fails to Satisfy User Intent: Pages that do not meet the rating scale fail to deliver the information or service that the user was searching for. They may be off-topic, incomplete, or irrelevant.
• Low Relevance: The content on these pages might be somewhat related to the query but doesn’t directly answer the user’s question or satisfy their need. The user will likely need to continue searching.
• User Frustration: Users landing on these pages might feel disappointed or frustrated because the page does not provide what they expected based on their query.
• Poor Content Quality: These pages might have low-quality content, such as thin copy, uninformative text, broken links, or a poor user experience due to excessive ads or misleading information.
• Lack of Credibility: Information may come from less credible or unreliable sources, and the page may not demonstrate E-E-A-T.

In essence, the difference between a “Fully Meets” rating and a “Doesn’t Meet” rating lies in the page’s ability to precisely and completely satisfy the user’s search intent with high-quality, relevant, and trustworthy content.

Q2: How is credibility established to demonstrate E-E-A-T?

ChatGBT: Credibility, in the context of demonstrating E-E-A-T, is established through several key factors that evaluate the quality and reliability of the content, its creators, and the website hosting the content. Here’s how each component contributes to establishing credibility.

1. Experience:

• First-Hand Knowledge: Content that shows the author has personal experience or direct involvement with the subject matter. For example, a travel blog written by someone who has visited the locations they describe.
• User Reviews and Testimonials: If the content includes reviews or testimonials from real users or customers, this adds to the credibility by showing that the content reflects real-life experiences.

2. Expertise:

• Qualifications of the Author: The author’s qualifications, such as academic degrees, professional certifications, or years of experience in a relevant field, establish their expertise. For example, medical advice written by a licensed doctor is more credible.
• Depth of Content: The content should provide in-depth and accurate information on the topic, demonstrating the author’s deep understanding and expertise.

3. Authoritativeness:

• Reputation of the Website: Websites with a strong reputation in their field, such as well-known news outlets, academic institutions, or government sites, are considered authoritative.
• Citations and References: Content that cites credible sources, such as scholarly articles, official statistics, or expert opinions, helps establish the authority of the page.
• Recognition in the Industry: If the author or website is recognized by others in the industry, such as through awards, media mentions, or backlinks from reputable sites, this contributes to their authoritativeness.

4. Trustworthiness:

• Transparency: Trust is built by clearly presenting information about the author, such as their name, biography, and contact information. A trustworthy website often includes an “About Us” page, privacy policies, and clear editorial standards.
• Accuracy and Honesty: Content that is factually accurate, well-researched, and updated regularly adds to its trustworthiness. Honest content does not exaggerate, mislead, or present information in a deceptive way.
• Secure and Reliable Website: Trustworthiness is also enhanced by technical factors like a secure (HTTPS) connection, lack of intrusive ads, and a well-maintained website that is free of errors or spammy content.

By focusing on these factors, content creators and websites can establish credibility and effectively demonstrate E-E-A-T, which in turn leads to higher ratings in search quality evaluations.

Q3: What is the one “SEO thing” people tend to ignore (or not reference) from this pdf when citing ways to improve their website?

ChatGBT: One important SEO aspect that people tend to overlook or not reference from the PDF when discussing ways to improve their website is “User Experience (UX) as part of E-E-A-T.”

While many focus on traditional SEO factors like keywords, backlinks, and content optimization, the guidelines emphasize that user experience is crucial in establishing trustworthiness and meeting user needs. This includes ensuring that the website is easy to navigate, mobile-friendly, and free from intrusive ads or distractions that could frustrate users.

Improving the overall user experience contributes significantly to how a page is rated, especially in terms of its Trustworthiness and how well it meets the user’s intent. This aspect, though critical, is often not highlighted enough in common SEO strategies.

Q4: How would a human interpret this PDF vs. an AI algorithm?

ChatGBT: The interpretation of the PDF document, “Search Quality Evaluator Guidelines,” would differ significantly between a human and an AI algorithm due to their distinct cognitive processes and abilities.

Human Interpretation:

1. Contextual Understanding:
   A human reader, especially one familiar with search engines or content evaluation, would grasp the purpose and overall message of the document quickly. They would understand the importance of concepts like E-E-A-T and how they apply to evaluating web content.
2. Nuanced Judgments:
   Humans can make nuanced judgments based on the content. For example, when evaluating the quality of a webpage, a human can recognize subtle cues in language, cultural relevance, and the credibility of information that might be difficult for an AI to discern.
   They can also apply subjective reasoning, such as determining whether a piece of content feels trustworthy or if it aligns with societal values, which can vary from person to person.
3. Handling Ambiguities:
   When faced with ambiguous instructions or conflicting information in the guidelines, humans can rely on their experience, intuition, and understanding of context to make informed decisions. They might seek clarification from colleagues or supervisors when in doubt.
4. Holistic View:
   A human can understand the broader purpose behind the guidelines and how different sections connect to each other, enabling them to apply the rules more flexibly in complex or edge cases.

AI Algorithm Interpretation:

1. Literal and Structured Processing:
   An AI algorithm would process the document in a structured and literal manner, analyzing the text based on predefined rules and patterns. It would focus on specific keywords, phrases, and the organization of content without necessarily understanding the underlying intent.
2. Consistency and Scalability:
   AI would apply the guidelines consistently across large volumes of data, making it ideal for tasks that require uniformity. However, it might struggle with content that doesn’t fit neatly into the predefined categories or that requires a more nuanced evaluation.
   AI can efficiently process and analyze large datasets, making it useful for generating insights at scale, but it might miss out on subtleties that a human would catch.
3. Limited Contextual Understanding:
   While modern AI can be trained to understand context to some extent, it would not have the deep cultural and contextual awareness that a human possesses. For instance, AI might struggle with sarcasm, cultural references, or the significance of certain types of content in different regions.
4. Handling Ambiguities:
   AI would rely on probabilistic models to handle ambiguities, making decisions based on patterns it has learned from training data. This can lead to incorrect interpretations if the document contains contradictory or unclear information.
5. Dependence on Data Quality:
   The AI’s interpretation would heavily depend on the quality and diversity of the data it was trained on. If the AI wasn’t trained with a wide variety of examples, it might misinterpret certain guidelines or fail to recognize exceptions to rules.

Summary:

Humans excel in interpreting complex, ambiguous, or culturally nuanced content, using intuition and experience. AI, on the other hand, can process large volumes of information quickly and consistently but might struggle with subtleties and context. Ideally, combining both human insight and AI efficiency could lead to the most effective application of the guidelines.

Conclusion

Even though Google has transitioned from human raters to AI algorithms, the Search Quality Evaluator Guidelines remain a valuable resource for understanding the principles behind high-quality content.

The old SEO mantra of focusing on user experience, E-E-A-T, meeting user intent, and creating content that engages your readers still aligns with Google’s standards. Regardless if humans or machines are scoring your content, this advice is still good for improving your website’s search rankings.

While I have little doubt that Google employees value the speed and scale of AI-scored content, even AI knows that human signals are perhaps the most valuable metric when it comes to grading what people really want.