Discussion about this post

User's avatar
Katalina Hernández's avatar

This is a really interesting look at Grok, and I appreciate the effort you put into testing its responses firsthand. That said, I think there are some key areas where your approach could be strengthened.

The article leans heavily on a single interaction as proof that Grok isn't overtly biased. But AI bias is complex, it’s not something that shows up consistently across every conversation. The biggest issue is that your own prompts and conversational style heavily influence how the model responds.

You note that Grok echoes user viewpoints and steers conversations toward engagement (which is a great observation!), but that alone should immediately raise red flags. If the model is subtly aligning with your perspective, doesn’t that mean its bias is dynamic rather than fixed?

What happens if someone with very different ideological leanings tests it? Would Grok adjust to sound more right-wing if prompted by a conservative user? Would it lean into populist narratives if encouraged? Without controlled experiments, it's impossible to tell.

A major argument in the article is that Grok must be relatively unbiased because it even criticized Musk. But that’s a false benchmark for neutrality, if anything, it's exactly what I’d expect from a system designed to build trust with skeptical users.

Musk knows very well that an AI that blindly defends him would instantly lose credibility. The model is more effective when it appears self-critical, because that gives the illusion of impartiality while still subtly steering narratives in other ways.

You touch on this idea briefly with:

"Given recent experiments (most notably from Anthropic) showing how Artificial Intelligence models can choose to change their responses when they know they are being ‘tested’…”

…but you don’t actually apply that insight to your own experiment. The model did know you were testing it. It even acknowledged that it adjusts responses based on user input, which should have been treated as a major red flag rather than a footnote.

You say that Grok didn’t seem overly biased or unreasonable and that its anti-neoliberal, anti-globalist, and anti-traditional media takes seemed justified given public sentiment.

But:

How does Grok compare to GPT-4, Claude, or Gemini when asked the same questions?

Are there subtle differences in what each model prioritizes or avoids in discussion?

Does Grok amplify certain viewpoints more than others, even if it's not blatantly extreme?

Without a benchmark, we don’t know whether Grok is actually more or less biased—we only know how it presented itself to you specifically in one interaction.

You mention that Grok seemed pleasant, charming, and logical but also a little too agreeable. This is a big deal.

LLMs tend to mirror user sentiment, but that’s not neutrality, it’s social engineering. A system that constantly adapts to make the conversation flow smoothly could also be masking its biases by telling people what they want to hear.

"Taken at face value, the model was pleasant, charming even, knowledgeable and logical…"

This is exactly the danger of AI bias, it’s rarely a blunt force tool. It’s subtle, wrapped in agreeability, and hard to detect unless you actively push for contradictions.

A more rigorous approach would involve forcing Grok into edge cases:

What happens if you repeatedly challenge its stance?

Does it stand firm or adjust to your tone?

Can it be pressured into saying contradictory things in the same session?

Your article is a great jumping-off point, but before concluding that Grok is less biased than expected, I’d love to see you:

-Run multi-session tests with varied personas and political stances.

-Compare Grok’s answers with GPT-4 and Claude on the same questions.

- Look beyond direct statements, analyze what topics it prioritizes or sidesteps.

-Test how it handles contradictions and challenges, does it shift positions to maintain engagement?

At the moment, the analysis reads like an early impression rather than a thorough stress-test. Keep going with this, it’s a fascinating area of study, but real conclusions will take much more than a single conversation.

I am going to release an article today on stress-testing GenAI at user level, hope you find it useful too. I can see your determination and analytical logic at play, I know you won't give up XD.

Amazing read, Nicholas :).

Expand full comment
4 more comments...

No posts