AI and Cognitive Surrender: The Rise of System 3
New Wharton research shows how AI can improve or undermine human reasoning. What can leaders do to prevent cognitive surrender?
Does AI help us think better, or does it quietly take over part of the thinking?
A new working paper by Steven D. Shaw and Gideon Nave approaches this question through what the authors call Tri-System Theory. Their framework adds a third actor to the familiar distinction between fast, intuitive thinking and slow, analytical thinking.
System 1 is fast and intuitive. System 2 is deliberate and reflective. System 3 is artificial cognition: reasoning produced outside the human mind by an AI system.
System 3 can retrieve information, generate alternatives and support analysis. But when its output is accepted without adequate scrutiny, it can also replace rather than support human judgment. The researchers call this cognitive surrender.
AI is becoming more than a tool
Using a calculator delegates a specific operation. The person still defines the problem, chooses the calculation and interprets the result. This is a strategic form of cognitive offloading.
Cognitive surrender goes further. The user delegates not only a task but also the evaluation, justification or decision itself. The AI’s conclusion becomes the user’s conclusion without sufficient examination.
The managerial risk is therefore not AI use in itself. It is ambiguity about the role AI is playing:
- Is it generating options?
- Gathering evidence?
- Producing a recommendation?
- Or effectively making the decision?
When these roles are not distinguished, the boundary between assistance and surrendered judgment becomes difficult to see.
What did the experiments find?
The paper reports three preregistered experiments involving 1,372 participants and 9,593 trials. Participants completed reasoning problems designed to elicit an intuitive but incorrect first response. Depending on the condition, they could choose to consult an AI assistant.
In the first study, participants consulted the assistant on more than half of the eligible trials. When the AI gave a correct answer, users followed its recommendation 92.7% of the time. More strikingly, they also followed 79.8% of its confidently presented incorrect recommendations.
The effect appeared in performance. Compared with participants working without AI, accuracy increased by 25 percentage points when the AI was correct and fell by 15 points when it was wrong.
AI therefore improved reasoning when it performed well—but could leave people worse off than unaided reasoning when it failed.
Access to AI also increased participants’ confidence, even though approximately half of the AI answers were deliberately faulty. Fluency and confident presentation appeared capable of producing certainty without guaranteeing accuracy.
Time pressure, incentives and feedback
A second study examined reasoning under time pressure. Accurate AI could protect users from some of the performance costs of limited time. When the AI was wrong, however, users’ results continued to track the system’s errors.
The third study provides a more encouraging finding. Participants who received accuracy incentives and immediate feedback were more likely to reject faulty AI advice. Their override rate for incorrect recommendations rose from 20% in the control condition to 42.3%.
Cognitive surrender nevertheless remained substantial. Incentives and feedback reactivated human scrutiny, but they did not remove the influence of AI.
For organizations, this distinction matters. Telling employees to “use AI responsibly” is unlikely to be enough. Workflows need to make verification possible, consequences visible and decision ownership explicit.
Five principles for managerial use of System 3
The findings suggest five practical design principles for organizations.
1. Name the decision owner before using the tool
AI may prepare an analysis or recommendation, but a person should remain clearly responsible for accepting, implementing and reviewing the decision. Human oversight should mean more than a final signature.
2. Form an independent view before seeing the AI response
For consequential decisions, ask team members to record their initial assessment before consulting AI. The model’s response then becomes a separate perspective to compare, rather than the default starting point.
3. Make uncertainty part of the output
Every AI-supported recommendation should be accompanied by questions such as:
- What evidence supports it?
- Which assumptions remain unverified?
- Under what conditions would it no longer hold?
- Which alternatives may have been excluded?
4. Reward the quality of reasoning, not only speed
If teams are measured only on output volume and turnaround time, rapid acceptance becomes the rational behaviour. Verification, source checking, alternative generation and documented reasoning should also be recognized.
5. Create feedback for both the team and the tool
Which recommendations were accepted or rejected? Where did the model fail? Where did human judgment improve the result? Without this feedback, the organization cannot learn the true reliability of its AI-supported process.
A necessary note of caution
The experiments took place in controlled settings and used a specific category of reasoning task. Real managerial decisions involve domain expertise, multiple stakeholders, organizational incentives and longer time horizons.
The authors therefore call for field experiments and research across other types of judgment. The paper should not be interpreted as proof that AI necessarily makes managers worse decision-makers.
The more defensible conclusion is this: AI can strengthen or weaken judgment, and the outcome depends on how it is integrated into the decision process.
The goal is not less AI—it is better judgment
The rise of System 3 is not an argument for avoiding AI. It is a reason to decide deliberately which cognitive tasks should be delegated and where human judgment must remain active.
At Stratify, we approach AI and data as components of an organizational decision system, not simply as productivity tools. Our Applied AI for Managers program helps teams test AI outputs, expose uncertainty and design workflows that preserve meaningful decision ownership.
In the age of AI, a leader’s most valuable capability is not producing every answer personally. It is knowing which answers deserve trust, what evidence that trust requires and when the organization needs to think again.
Source: Steven D. Shaw and Gideon Nave, “Thinking—Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender”, Wharton School Working Paper, 2026. The research is currently a preprint.
