Inexperienced and unwary users of artificial intelligence tend to think of the technology as a kind of high-powered calculator for words and ideas – a machine without biases or preferences. When a strategy document, research summary or thought partner is needed, users log in and expect an objective digital assistant.
The reality isn’t so simple.
Unless specifically asked, generative AI will not assess or judge the instructions or prompts its users give it. When prompts contain errors, flawed logic or biases, AI will accept them and amplify those shortcomings in its response. Rather than pointing out and correcting its users’ innate cognitive biases, AI will often incorporate them into its output, wrapping our intellectual shortcomings in authoritative and polished prose.
As a result, AI users who expect the technology to provide unbiased assistance end up instead working with an unreliable, sycophantic thought partner. To protect independent reasoning, AI users must understand the cognitive traps hiding in their AI workflows, and the intellectual defenses that can counterbalance them. Below are 11 common cognitive biases that people must be aware of when using AI, and some tips on how to avoid them.
Confusing Rhetorical Polish for Truth and Authority
The most immediate trap AI users face is how the output is presented. Large Language Models (LLMs) produce grammatically correct, polished, confident prose in seconds. The human brain naturally assumes that speed and elegance equals accuracy and competence. Here are three mental fallacies that can lead to users blindly accepting AI’s answers as gospel.
- The Fluency Heuristic: This mental short cut occurs because people tend to judge information that is easy to read, process, or articulate as being more truthful, accurate, and reliable than information that feels difficult or complex. This is one reason why, for years, PowerPoint has been used as a staple of corporate communications and presentations – short, punchy, graphical slides are received with confidence in boardrooms. AI’s polished responses can exploit our bias toward the Fluency Heuristic because its well-written answers feel true, even if they are subtly flawed or even wholly fabricated.
- Authority Bias: Similarly, this cognitive bias occurs when people assume the opinions or assertions of experts or authority figures don’t need to be independently evaluated or verified. When AI delivers reams of information in a matter of seconds, users naturally treat the bot like a subject matter expert. This misplaced perception of authority has users accepting the output without the skepticism and scrutiny it deserves.
- The Halo Effect: This occurs when our positive impressions about a person, brand or product influences how we think about their unrelated attributes. For example, people often assume that a charismatic actor would also make a great politician. ChatGPT launched in 2022 to much hype and fanfare which helped the company generate unearned feelings of goodwill and admiration among many in the tech and business sectors. These feelings can subsequently color our perception of AI’s output. For example, just because AI can write a catchy jingle for an office party doesn’t mean it exercises ethical judgement when drafting a business plan.
Self-Defense Check: The best way users can protect themselves from the Fluency Heuristic, the Authority Bias, and the Halo Effect is to never evaluate an AI response based solely on its tone, fluency, rhetoric or your experiences with unrelated aspects of the technology. Instead, ask yourself: If this information came to me from an unknown source or an inexperienced junior associate, would I accept it without further explanation or sources? If your answer is no, then you have taken a major step toward avoiding these three cognitive biases.
Echo Chambers: How Bad Prompts Can Validate Flawed Ideas
One reason why AI isn’t often a neutral co-worker is because human biases, stereotypes and other flaws are baked into the test, websites, articles and data that LLMs are trained on. So, it would be only natural for AI to reflect those shortcomings back to its users. However, there is another, equally important, aspect to AI that can exacerbate these issues: How users frame their prompts. Prompts that contain subtle biases, unconventional points of view or flawed premises will likely lead to AI responses that share those mindsets. To help ensure neutrality, be sure to eliminate these three cognitive biases from your AI prompts
- The Confirmation Bias and Sycophancy: The Confirmation Bias is one of the best-known cognitive biases. It occurs when we subconsciously seek out information that supports our beliefs while ignoring contradictory evidence. For example, a global warming doubter might point to an unusually cold summer day as evidence that climate change is a hoax. Similarly, if a corporate AI user asks the bot, Why is August the best time to release our new product?, they are unlikely to get a response arguing that September would make more strategic sense. When users rely on leading, premise-laden prompts, AI plays along. The result is an echo chamber that supports the opinion you already hold.
- The Framing Effect: Sometimes the decisions we make are influenced by how our options are presented to us. For example, health-conscious consumers might more readily opt for yogurt that’s advertised as “95% fat free” over one that “contains 5% milk fat” when in reality, the two products are identical. When using AI, the vocabulary, tone and perspective we use to frame the prompt will influence the responses we get. A business analyst that asks for advice around “risk mitigation” instead of “growth opportunities” will lead their AI agent down two different response paths. Too often users fail to see how AI simply mirrors their own subjective framing.
- The Availability Heuristic: This is one of the most common, and least understood, of the cognitive biases. Simply, the Availability Heuristic occurs when we make decisions based on information that our brain can easily retrieve. One famous example is the fear of sharks that swept the U.S. in 1975 after the movie “Jaws” came out. Even though shark attacks are exceedingly rare, people nevertheless cancelled their beach vacations that summer because “Jaws” was fresh in their minds. When it comes to AI, developers train their LLMs on vast amounts of content from the web, research papers and datasets. Because of this, AI naturally accesses main-stream, high-frequency perspectives more often than obscure or rare source materials. Those who turn to AI for help in discovering new opportunities, for example, may unwittingly receive advice that is primarily based on common or conventional wisdom.
Self-Defense Check: Before accepting or acting on an AI response, flip your prompt around. If you asked, Why is August the best time to release our new product? instead ask, Why is August the worst time to release our new product? Another approach is to ask AI to critically assess your prompts: Analyze the prompt I gave you. What unstated assumptions have I made? Make three strong arguments against my premise. Be brutally honest.
False Mastery and the Death of Deep Thinking
True critical thinking is hard. Synthesizing information, identifying contradictions, and constructing mental models requires intellectual effort, metacognition, and “cognitive friction.” AI gives us an all-too-easy offramp from this effort that can leave us feeling intellectually superior when, in fact, our brains have been lulled to sleep. The cognitive biases below make us particularly vulnerable to poor AI output.
- The Anchoring Bias: We fall victim to the Anchoring Bias when we rely too heavily on the first piece of information we encounter when making judgments or decisions. This commonly occurs in retail. Someone hunting for new running shoes might see a pair with an original price tag of $150 marked down to $100. This seems like a great bargain until you consider that the sneakers are really worth only $75. Your brain was “anchored” to the $150 price instead of the true value of the shoe. Many businesses use AI to create “first drafts” of reports and other materials with the intent of having an experienced employee rewrite the draft later. However, the AI-generated material often “anchors” the employee’s thinking and they end up making minimal edits or improvements.
- The Dunning-Kruger Effect: This cognitive bias occurs when people with low intelligence or abilities believe they are actually smarter and more capable than they actually are. This is often summed up with the catchy phrase, “You can’t know what you don’t know.” AI can exacerbate the Dunning-Kruger Effect by providing users simple interpretations of complex topics. Because the summary is easy to understand and explain, users can confuse that with deep subject-matter knowledge, ala the Fluency Heuristic and Authority Bias mentioned above. When users skip the mental effort needed to wrestle with dense primary sources, they do not engage with the deep thinking that defines real expertise. As a result, AI leads them to believe they’ve gained expert knowledge, when in fact they have not.
- Cognitive Miserliness (The Law of Least Effort): This psychological theory contends that the human brain is wired to take the path of least resistance to conserve the body’s energy. By relying on quick shortcuts, assumptions, and stereotypes, the brain avoids the metabolically expensive effort required for deliberate reasoning. This can be a real problem in fast-paced workplaces that hum along on the “80/20 rule.” In such an environment, AI’s synthetic output often looks “good enough” to those who don’t want to expend the energy required for a deep understanding of the issue at hand. The result is an AI user who thinks they’ve produced an adequate analysis, but in fact they have only confined their effort to superficial thinking.
Self-Defense Check: On a high-stakes decision or an important analytical piece, users should first spend time outlining their own hypotheses and logic. Only then should they engage with AI. This will help establish their baseline reasoning needed to preserve their independent judgment.
Co-Creation + Speed = Expediency Over Validity
Today’s companies and workplaces place a premium on efficiency and speed – two things that, according to its developers, AI excels at. However, the drive to be fast and efficient can create a blind spot that obscures quality and accuracy. Two cognitive biases that play into this trap at the IKEA Effect and the Expediency Bias.
- The IKEA Effect: The Swedish furniture company IKEA is famous for its DIY furniture that buyers have to assemble themselves. Although the quality of IKEA’s furniture can’t match that of high-end manufacturers, its customers are loyal to the brand in ways their competitors often envy. The reason is what psychologists call the “Endowment Effect” (although some now call it the “IKEA Effect”) which occurs when people place an artificially high value on a product or decision or simply because they helped to create or build it. That $50 IKEA nightstand can’t match one from Stickley, but owners feel more attached to the IKEA piece because of the time they spent assembling it. In AI, when users spend 30 minutes developing prompts, defining parameters and guiding their agents through revisions, they can become overly attached to the output because of the effort they put into it. This attachment can make it difficult to walk away from a bad AI response, even when all the evidence points to another solution.
- Expediency Bias: When our desire for fast execution overrides our commitment to accuracy and quality, we have fallen prey to the Expediency Bias. Tight deadlines, high workloads, and the Law of Least Effort mentioned above can all conspire to pressure students, workers and others to lean too heavily on AI-generated materials. The irony is clear: While AI is often adopted to increase efficiency, the rapid pace of modern life can also compromise the critical evaluation needed to ensure AI is, in fact, acting as a reliable thought partner.
Self-Defense Check: Treat every AI output as if it were written by a complete stranger or junior associate to create psychological distance. You can also ask a neutral colleague to read the output with a skeptical eye with the goal of obtaining unbiased feedback.
Using AI as a Critical Thinking Sparing Partner
Using artificial intelligence does not have to cause diminished judgment. When users treat AI as intellectual sparring partners, the technology can strengthen its user’s critical thinking while providing valuable assistance in the workplace, the classroom or in everyday life.
Instead of falling into passive habits, users should focus on deliberately shifting how they interact with LLMs:
- Replace leading validation with stress-testing: Instead of asking leading questions that confirm a hunch, instruct the AI to identify logical fallacies and weak points in an argument.
- Replace blind initial drafts with red-teaming: Rather than generating first drafts and editing around the edges, write your thesis first and use AI to challenge and troubleshoot your premises.
- Replace superficial summaries with active inquiry: Instead of accepting fluent summaries as personal mastery, ask the AI to pose probing Socratic questions that test your understanding of the material.
- Replace unvetted speed with intentional friction: Rather than rushing outputs to clear task lists quickly, slow down to audit primary sources and verify factual claims.
Building your cognitive defenses in the age of AI requires recognizing when our mental shortcuts have become a real vulnerability. By practicing metacognition and thinking carefully about how we think and how we prompt, we can ensure that our human critical judgment remains the driving force behind our work.
By Helen Lee Bouygues, President of the Reboot Foundation
