13_calculated_risks.md

Calculated Risks

"Overcoming innumeracy is like completing a three-step program to statistical literacy. The first step is to defeat the illusion of certainty. The second step is to learn about the actual risks of relevant events and actions. The third step is to communicate the risks in an understandable way and to draw inferences without falling prey to clouded thinking."

Related Reading

• Simple tools for understanding risks: from innumeracy to insight

Illusion of Certainty

• "often wrong but never in doubt"

• Perceptual systems:

  • "automatically transform uncertainty into certainty, as depth ambiguities and depth illusions illustrate."

    • Depth illusion illustrated by Roger Shepard's "Turning the Tables"
    • Necker cube --- ambiguous depth

  • The perceptual system analyzes incomplete and ambiguous information and “sells” its best guess to conscious experience as a definite product. ... Look back at the two tables; they will still appear to be different shapes. Even if one understands what is happening, the unconscious continues to deliver the same perception"

  • "Our perceptual systems assume that a line of a given length on the retina that extends into depth is actually longer than one that does not and corrects for that"

  • "human perceptual system constructs representations of three-dimensional objects from incomplete information"

  • "Helmholtz coined the term “unconscious inference” to refer to the inferential nature of perception."

Miscommunication of Risk

Three major forms of risk communication that invite miscommunication:

1. Use of single-event probabilities. e.g. there is a 30% chance it will rain tomorrow + cannot be proven wrong/right + people can interpret the statement as 30% of the area will get rain, 30% of the day it will rain, 30% of the days like tomorrow will get rain

2. Relative risks

   • "People with high cholesterol can rapidly reduce . . . their risk of death by 22 percent by taking a widely prescribed drug called pravastatin"
     • "Studies indicate that a majority of people think that out of 1,000 people with high cholesterol, 220 of these will be saved"
       • Of the 1000 people who took p for 5 years, 32 died in treat vs. 41 died in control.
   • Fix = absolute risk reduction or NNT (here, it is 9 in 1000 or 1 in 111)

3. Conditional probabilities

   • Fix = use natural frequencies

Natural Frequencies

"The probability that a woman of age 40 has breast cancer is about 1 percent. If she has breast cancer, the probability that she tests positive on a screening mammogram is 90 percent. If she does not have breast cancer, the probability that she nevertheless tests positive is 9 percent. What are the chances that a woman who tests positive actually has breast cancer?"

vs.

"Think of 100 women. One has breast cancer, and she will probably test positive. Of the 99 who do not have breast cancer, 9 will also test positive. Thus, a total of 10 women will test positive. How many of those who test positive actually have breast cancer?"

• Create a tree

  Of the 1000:

  • 992 no cancer
    • 70 positive
    • 922 negative
  • 8 cancer
    • 7 positive
    • 1 negative

Other Issue with Understanding of Risk---Prosecutor's Fallacy

• "The probability that this match has occurred by chance is 1 in 100,000." Vs. "Out of every 100,000 people, 1 will show a match."

• p(positive test|breast cancer) is not the same as p(breast cancer|positive test)

Case Studies

Fingerprinting

• Galton/1st cousin of Darwin

  • Est. 1 in 64B match if you use all the 35 to 50 points.
  • Current system
    • Declare a match if 8--16 points match.
    • two problems with fingerprints:
      • incomplete
      • latent: need to be treated with chemicals
  • To est. reliability of fingerprint evidence in a robbery case where 2 latent fingerprints were matched, FBI sent the evidence to 53 labs --- 35 responded --- 8 failed to find a match for 1, 6 for the other. 14/70 or 1 in 5 tests failed.

• Communicating Risk:

  • You have X% of getting Y != X out of every 100 will get Y. This is because people interpret X% of their events will have Y. Latter makes the unit clear.

Breast Cancer

• Mass breast cancer screenings introduced in Germany in the 1930s

  • Age > 30, biennial, monthly breast exam conveyed as a 'moral duty'

• American National Cancer Institute recommends screenings post 40. US Preventive Taskforce and Canadian Task Force ---> 50. American Cancer Society recommends breast self-exams starting 20.

• Clarifications for Screening:

  • Screening is for asymptomatic people
  • It doesn't reduce the incidence of breast cancer (Duh!)
  • Mortality reduction is not guaranteed
  • Not all cancers progress. Most cancers in young women are ductal carcinomas and half don't progress.
  • Early detection is not always beneficial. For instance, if cancer doesn't progress

• RCT estimates:

  • Overall: Of the 1000 women in treatment, 3 died in 10 years. Of the 1000 women in control, 4 died. NNT = 1,000

  • Women who participate in breast cancer screening from 50 to 69 live on average 12 more days

  • Same effect as driving 300 miles less each year.

  • 40--49: No RCT shows benefit for women b/w ages 40--49 for 9 years following first screening. 9 out of 10 RCTs show no benefit for 10--14 years after the 1st screening.

  • 50+: Of the 8 RCTs, 3 sig., 4 non-sig., 1 ~ near zero. For every 270 women who start screenings at 50 and continue for 20 years, 1 is saved.

    • Absolute reduction is risk ~ 4 in 1,000
    • No evidence that self-exam helps in addition to screenings
    • yearly or biennial screenings ---> same reduction
    • Cost ~ $21k per year of life saved.

  • Costs:

    • FP
      • Only 1 in 10 women who were flagged to have cancer on their first screening had breast cancer within 13 months of the screening.
      • Of 1,000 women who have breast cancer screening b/w 40 and 50, 70 FP and 7 TP
      • In a series of 10 mammograms, 1 in 2 women will receive at least one FP
        • 3--4M screenings in Germany each year. About 300k FP. About 100k of those have invasive surgery.

  • Three sources of Costs:

    • Nonprogressive cancer --- Few women have heard of non-progressive breast cancers.
    • Radiation-induced breast cancer
      • Of every 10k women starting screenings at 40, 2--4 will develop radiation-induced breast cancer
    • False Negatives
      • 5 to 20 percent. Higher in younger women.
    • Funny person: "some women probably die on the way to the screening"

  • Misinformation

    • Women think the positive effects for 40+ is 60 per 1000 saved.
    • Source --- leaflets. Slaytor and Ward --- absolute risk reduction never mentioned in the 58 leaflets they analyzed. Neither is the false positive rate.
    • Media
      • 1 in 10 chance refers to cum. chance by age of 85. Most women die before that. And others will die of other causes.
        • 1 in 3 women who develops breast cancer will die of it.
      • Of 1,000 45-year-olds, 5 will die of breast cancer in the next 10 years. But when you ask people, they think 100 will die.
      • American Heart Association survey: only 8% flagged that heart disease is the most common cause of death among women
      • Prostate cancer > Breast cancer. Kills as many people. Prostate cancer represented as old people disease when profile very sim. to breast cancer.
      • Colorectal cancer is the 3rd most deadly cancer among women but like lung cancer, doesn't make headlines as often
      • 85% of the stories featured women under 50---atypical cases

  • Effect of prophylactic mastectomy --- decline from 5 in 100 to 1 in 100

Problems with Medical Communication

• Zero communication of false positives
• Some doctors think that
  • patients want the illusion of certainty. They may prefer such doctors.
  • don't have the mental capacity to make the decisions
• In medicine, unlike aviation, no system of reporting errors without being punished.
• Referrals drop if the doctor informs patients---no more guaranteed operations, etc.
• Informed Consent:
  • Incentives of doctors not same as patients --- reason why the information should be conveyed
  • 1957, Bray's judgment coined "informed consent"

Misc.

• 'Solving a problem simply means representing it so as to make the solution transparent.' --- Herb Simon

• "Franklin’s law": Nothing is certain but death and taxes.

• In the 17th century, buying life insurance meant making a bet on prominent person's life, e.g., mayor of Paris will die in the next 3 years.

• Heroism is self-deception and the greatest obstacle to progress

• Placebo effect 'goes away' if you discuss treatment efficacy in NNT.

• "Math is hard. Let's go shopping."---Barbie

• Americans are not alone. "In a survey, 1,000 Germans were asked what “40 percent” means: (a) one-quarter, (b) 4 out of 10, or (c) every 40th person. About one-third of the respondents did not choose the right answer."

• Theory of mathematical prob. only developed in 1654 --- letter b/w Pascal and Fermat.

• "[a] little girl ... was scheduled to receive an injection from her pediatrician. Upset that her father signed a consent form stating that he understood that 1 out of 10,000 children experience a serious allergic reaction, she insisted on speaking to the doctor. “I want to know,” the little girl asked, “what number you’re on.”"

• The nineteenth century saw a struggle among three different visions of the physician: the artist, the statistician, and the determinist. The French medical professor Risueño d’Amador, as an artist who relies on medical “tact” and his intuitions about the individual patient. His rival Pierre-Charles-Alexandre Louis ... rejected bloodletting as a medical treatment. By collecting data, he showed that slightly more people who were bled died than people who were not. The French physiologist Claude Bernard rejected the images of both the physician-artist and the physician-statistician. For Bernard, science meant certainty. In the nineteenth century, statistical data were still considered antithetical to the scientific method. Whereas science was about certainty, statistics was about uncertainty; therefore, statistics was not a proper scientific tool. "physician Ignaz Semmelweis's statistical studies of childbed fever and scurvy are as legendary as the reluctance of the relevant authorities enact the preventive measures that his statistics suggested."

• Out of some 175 accredited law schools in the United States, only one requires a course in basic statistics or research methods.

• 40k Americans die each year in accidents. Average say 75 years and 300M per year. You are looking at a 1 in 100 chance of dying in an accident. In the U.K., the number is ~ 1 in 220. Portugal and Greece ~ 1 in 50.