第 7 章 · 20 分钟

Reverse audit: checking the evidence of chapters 1–6

The audit covers chapters 1 to 6, including right conclusions reached wrongly.

概念地图

Reason audit

A right conclusion reached by wrong reasoning is booked as a defect.

深色为本章概念，浅色为其他章

陪练模式

不想只是读？让它带你走一遍。

导师把这一章拆成小步，每一步先问你，再讲；你用自己的话答，它先认下对的部分再纠偏。用你自己的 AI，对话只存在这台设备上。

对话只存在本机

Locus01

延伸阅读

Meehl · 1950s · statistical versus clinical prediction

Compared simple statistical rules with expert judgement and found the rules no worse on most tasks. He argued against treating clinical experience as irreplaceable; the cost is that the comparison covers only tasks with a clear outcome variable, and judgements without one still cannot be compared this way.

Foundational

Simmons and colleagues · 2011 · researcher degrees of freedom

Demonstrated that entirely permissible analytic choices can make a non-existent effect significant. This turned questionable practice from a matter of ethics into a matter of process, and led directly to preregistration.

Turn

Kahneman · 2017 · public statement on the priming section

The author stated that the parts of the book resting on priming research were on weaker evidence than he had believed when writing. Rare for a popular book, and it gives this chapter's graded audit the author's own backing.

Revision

Chapters 1 to 6 of this course

The subject of the audit. The third block (derivation) and the fifth block (cracks) of each chapter are the main entry points.

Under audit

机制02

Evidence grade does not travel with the text

Two sentences printed side by side can rest on evidence an order of magnitude apart: anchoring replicated across many labs at large samples, some priming effects on a single lab's small sample. A reader cannot see the difference in the text, because the register and the sentence shapes are the same. This course books that as grade not travelling with the text. It is not the author's fault but a property of the form: popular writing requires a uniform register.

机制A uniform prose register, through flattening differences in evidential strength, leads readers to assign one credence to every claim.

可迁移性测试

Move it to a financial report: management discussion describes signed contracts and letters of intent in the same phrasing, and a reader who skips the notes cannot tell which money is committed. The isomorphism breaks in that a report has mandatory notes and a book has none.

机制03

A mechanism demoted to a label

'That is anchoring', 'that is availability' — once a mechanism becomes a popular phrase, it tends to be used to name an outcome afterwards rather than to predict one beforehand. That step turns a falsifiable mechanism into an unfalsifiable label, which is the same problem chapter 1 identified in filing System 1 and System 2 under exposition. The test is simple: did this use rule anything out before the result was known?

机制A mechanism, through being used only to name outcomes after the fact, loses its power to predict in advance and decays into a label.

可迁移性测试

Move it to code review: 'that is technical debt' has content when it blocks a change in advance, and says nothing when it explains after the fact why the system is slow. The isomorphism breaks in that technical debt can be itemised and costed, while the list of cognitive biases has no boundary.

Derivation04

Counting categories and evidence in chapters 1–6

This aggregates the tag of every derivation step in the first six chapters and grades them by evidential structure, deriving one conclusion: the book's empirical risk sits on few steps, and those few are not equally well supported.

Chapters 1 to 6 have 4 derivation steps each, 24 in total
Counting rule: only the steps inside each chapter's third block, not cracks or debate.
Thirteen steps are tagged [identity]
These hold under any data and carry no empirical risk.
Five are tagged [assumption], six are tagged [empirical]
Assumptions are chosen; empirical claims can be overturned, but not all with equal support.

推导 · 0 / 4

裂缝05

Cracks in this framework

争议地形06

本书主张

This course's position: claims in a popular book must be graded one at a time by evidential structure, and never accepted or rejected as a whole.

另一种看法

Another line holds that for a general reader the cost of checking each claim far exceeds the benefit, and that a book pointing broadly in the right direction improves judgement even if individual claims are overstated. Demanding grades imposes an academic standard on general reading.

分歧扎在

The disagreement is rooted in what readers will do with the claims. If only to shift general attitudes, the cost argument holds; if to design processes, evaluate people or invest, the strength of an individual claim decides the consequences directly.

什么证据能裁决

What would settle it is a survey of use: track where readers actually cite these claims in the six months after reading and count the share used in concrete decisions. A low share supports the cost argument; a substantial share entering process design and personnel evaluation makes grading necessary.

The evidence points to a substantial share entering real decisions: loss aversion and framing are widely written into product design and incentive schemes. What is missing is direct measurement of the consequences — being cited often is not the same as being used wrongly.

Falsification07

A mechanism demoted to a label, as a preregistrable test

The mechanism under test: a mechanism, through being used only to name outcomes after the fact, loses its power to predict in advance.

Data source: published commentary and internal post-mortems, in the passages where the book's mechanism names appear
Both kinds are needed, because public and internal usage may differ.
Sample: every matching passage in a fixed period, with no filtering by the writer's identity
Filtering by identity turns the result into a description of one population.
Window: one year, covering at least one complete business cycle
Advance use clusters in planning periods and after-the-fact use in review periods; the window must contain both.
Threshold: the share of passages using the name to explain after the fact rather than to predict in advance is above a pre-registered seventy per cent
Seventy per cent is set in advance and not adjusted afterwards.
Failure condition: the share falls below seventy per cent, or inter-coder agreement is below the acceptable level
The second voids the test.

推导 · 0 / 3

接口08

Which slot it hangs on

挂在哪个槽位The conclusion you already hold is probably 'having read this book, I understand my own biases better' — a judgement about the return on reading. This chapter goes at how that return is realised.

Does this chapter (a) replace 'reading it means understanding it', or (b) constrain it to the case where you have used a mechanism to make an advance prediction?

慢变量Register two slow variables: how many advance predictions you have made using a mechanism from this book, and how many of those you have checked. Look at the first one in a few days — if it is zero, the book is still only vocabulary to you.

小结09

本章小结

01Of 24 derivation steps in chapters 1–6, thirteen are identities, five are assumptions, and six carry empirical risk.

02Claims within one book can differ in evidential strength by an order of magnitude, and the text does not carry that information.

03A mechanism used only to name outcomes afterwards decays into an unfalsifiable label.

04'Not covered by replication' is not 'failed replication'; the two need separate tiers.

05Combining evidential strength into one score needs weights, so this course lists the components instead.

提取练习 · 合上书，先自己答一遍。

?Is 'steps at risk equal total minus identities minus assumptions' (a) an identity or (b) an empirical claim?

?Is explaining something that already happened with 'that was availability' (a) an advance prediction or (b) an after-the-fact label?

Forced choice10

B · Identify the tag

'Steps carrying empirical risk equal total steps minus identities minus assumptions' — what kind of step is that?

二选一

Forced choice11

A · Identify the mechanism

Which of these is a mechanism?

二选一

读到这里，把它记为已读。