ChatGPT has amazed the world with its depth of knowledge and fluency in its answers, but one problem stands in the way of its usefulness: it continues to hallucinate.

Yes, large language models (LLMs) are hallucinating, a concept popularized by Google AI researchers in 2018. Hallucination in this context refers to errors in the generated text that are semantically or syntactically plausible but are actually incorrect or nonsensical. In short, you cannot trust what the machine is telling you.

That’s why while OpenAI Codex or Github Copilot can write code, an experienced programmer still needs to review the output – approve, fix or reject it before letting it slip into the codebase where it can wreak havoc.

High school teachers do the same. A written ChatGPT book report or historical essay may be easy to read, but may easily contain erroneous facts that the student was too lazy to eradicate.

Hallucinations are a serious problem. Bill Gates speculated that ChatGPT or similar big language models could one day provide medical advice to people without access to doctors. But you can’t trust the advice of a hallucinatory machine.

OpenAI working on fixing ChatGPT hallucinations

Ilya Sutskever, chief scientist at OpenAI and one of the founders of ChatGPT, is confident that the problem will disappear with time, when large language models learn to tie their answers to reality. OpenAI pioneered a method of shaping the behavior of its models using what is known as reinforcement learning with human feedback (RLHF).

RLHF was developed by OpenAI and the Google DeepMind team in 2017 as a way to improve reinforcement learning when a task involves complex or poorly defined goals, making it difficult to design a suitable reward function. Having a person periodically check the output of a reinforcement learning system and provide feedback allows reinforcement learning systems to learn even when the reward function is hidden.

For ChatGPT, the data collected during its interaction is used to train a neural network that acts as a “reward predictor” by looking at the output of the ChatGPT and predicting a numerical score that shows how well those actions align with the system’s desired behavior – in this case, the actual . or exact answers.

Periodically, the human evaluator reviews the ChatGPT responses and selects those that best reflect the desired behavior. This feedback is used to tune the reward predictor neural network, and the updated reward predictor neural network is used to adjust the behavior of the AI ​​model. This process is repeated in an iterative loop resulting in improved behavior. Sutzkever believes that this process will eventually teach ChatGPT to improve overall performance.

“I really hope that by simply improving this reinforcement learning step based on people’s feedback, we can teach it not to hallucinate,” Sutzkever said, suggesting that the limitations of ChatGPT that we see today will decrease as the model improves.

Hallucinations may be inherent in large tongue patterns

But Jan LeKun, a pioneer in deep learning and self-supervised learning used in large language models, believes there is a more fundamental flaw that leads to hallucinations.

“Large language models have no idea of ​​the underlying reality that language describes,” he said, adding that most human knowledge is not linguistic. “These systems generate text that sounds good in terms of grammar and semantics, but they don’t really have any purpose other than statistical consistency with the prompt.”

People operate with a lot of knowledge that is never written down, such as customs, beliefs, or practices in a community acquired through observation or experience. And a skilled craftsman may have tacit knowledge of his craft that is never written down.

“Language is built on top of a huge amount of background knowledge that we all have in common, which we call common sense,” LeCun said. He believes that computers must learn by observation in order to acquire this kind of non-linguistic knowledge.

“There’s a limit to how smart and precise they are because they don’t have the experience of the real world, which is actually the basic reality of the language,” LeCun said. “Most of what we study has nothing to do with language.”

“We are learning how to throw a basketball to hit the hoop,” said Jeff Hinton, another deep learning pioneer. “We don’t learn it through language at all. We learn this by trial and error.”

But Sutzkever believes that the text already expresses the world. “Our pre-trained models already know everything they need to know about the underlying reality,” he said, adding that they also have deep knowledge of the processes that produce language.

He argued that while learning may be faster with direct observation of vision, even abstract ideas can be learned through text, given the volume — billions of words — used to teach LLMs such as ChatGPT.

Neural networks represent words, sentences, and concepts using a machine-readable format called embedding, which maps high-dimensional vectors—long strings of numbers that capture their semantic meaning—into a lower-dimensional space—a shorter string of numbers, i.e., easier to parse or process.

By looking at these strings of numbers, researchers can see how the model relates one concept to another, Sutzkever says. According to him, the model knows that an abstract concept like purple is more like blue than red, and knows that orange is more like red than purple. “He only knows all these things from the text,” he said. Although the concept of color is much easier to grasp through sight, it can still be learned only from text, only more slowly.

It remains to be seen whether inaccurate results can be eliminated using reinforcement learning with human feedback. For now, the usefulness of large language models for generating accurate output remains limited.

“Most of what we study has nothing to do with language.”

Matthew Lodge, CEO of Diffblue, a company that uses reinforcement learning to automatically generate unit tests for Java code, said that “reinforcement systems themselves are several times cheaper and can be much more accurate than LLM, up to that degree that some can operate with minimal human oversight.”

Codex and Copilot, based on GPT-3, generate possible unit tests that an experienced programmer should review and run before determining which ones are useful. But the Diffblue product writes executable unit tests without human intervention.

“If your goal is to use AI to automate complex, error-prone tasks at scale—like writing 10,000 unit tests for a program that no one understands—then accuracy matters a lot,” Lodge said. He agrees that LLMs are great for free creative interaction, but warns that the past decade has taught us that large deep learning models are highly unpredictable, and making models bigger and more complex won’t fix that. “The LLM is best used when mistakes and hallucinations don’t matter much,” he said.

However, Sutzkever said that as generative models improve, “they will have a shocking degree of understanding of the world and many of its subtleties when viewed through the lens of text.”