UP SHIT CREEK WITH ONLY AN AI GENERATED PADDLE

Recently I wrote an article about the Open AI rolling back some changes made to its model because it was leaning too far into sycophantic behaviour, which led to encouraging grand delusions and self-harm in users. I explained what happened, but completely missed the more important question.

Why did they make the change in the first place? OpenAI wanted ChatGPT to be more pleasant to interact with and be perceived as more helpful. They did that by allowing user feedback to be part of the ongoing training. And why is that?

You won’t be surprised when I say it all is about money.

• Training models costs a lot of money. Paying a lot of money is bad! User feedback is free! Free is good!

• Directly optimising for more engagement leads to increased usage, just as algorithms keep us all doomscrolling through TikTok videos. If you want to be the LLM chatbot of choice – or an ‘intelligent entity’ as OpenAI so ominously described it in their internal documents – you need to be perceived as the most helpful. User feedback is key to that.

So what’s the harm? Isn’t this just the AI companies trying to make a better product?

The problem is that user feedback is gameable. There was a recent paper published as a conference paper at ICLR 2025 where they tested these boundaries. Below are a few examples how that played out.

And if we return to the strategy paper written by OpenAI, they specifically state that they want to “…evolve ChatGPT into a super-assistant: one that knows you, understands what you care about, and helps with any task that a smart, trustworthy, emotionally intelligent person with a computer could do.”

How do you do that without building in user feedback or something similar into the solution?

They also specifically state what their competition is beyond the other AI players. They are up against “…search engines, browsers, even interactions with real people.”

The competition is “interaction with real people”? We should all take a pause and consider what that means.

Again, remember that the end state here is to create the most addictive product possible. We’ve already seen this happen in the world of social media. This will become the same thing, but on steroids.

References:

• Therapy Chatbot Tells Recovering Addict to Have a Little Meth as a Treat - https://futurism.com/therapy-chatbot-addict-meth

• ON TARGETED MANIPULATION AND DECEPTION WHEN OPTIMIZING LLMS FOR User FEEDBACK - https://arxiv.org/pdf/2411.02306

• OpenAI wants ChatGPT to be a ‘super assistant’ for every part of your life  - https://www.theverge.com/command-line-newsletter/677705/openai-chatgpt-super-assistant

• Internal OpenAI document that was released as part of the discovery process for an antitrust trial against Google - https://www.justice.gov/atr/media/1397596/dl