Selection bias and paradoxes

Bias from "non-randomly selected samples" is a form of omitted-variable bias, and it can be corrected with a two-step estimator. This is part of the work behind his 2000 Nobel ("for his development of theory and methods for analyzing selective samples").

Train a model only on approved or closed deals and it inherits the selection that produced them.

When outcomes are seen only for cases a human approved (bail, loans), you can still evaluate a model by using the differences between lenient and strict decision-makers ("contraction").

You never see outcomes for the files you declined or the leads your reps skipped. Differences between your underwriters or reps are your natural experiment.

Credit scorecards are built only on approved applicants. "Reject inference" tries to guess how the rejected would have performed, and the 1993 paper asks whether it can ever work. The 1997 paper is the standard review of scoring methods.

It's the lending industry's name for "you only see the loans you made."

"a lease winner tends to be the bidder who most overestimates reserves potential." Thaler presents evidence that "the winner's curse may be a common phenomenon."

If you win a contested lead, keyword or media auction, it's bad news about your own estimate. Your A/B-test winners suffer the same curse (see Lee and Shen, 2018).

Berkeley's combined admissions showed "a clear but misleading pattern of bias against female applicants." Department by department, the data showed "a small but statistically significant bias in favor of women." Women had applied more to the departments that admitted fewer of anyone.

A blended rate like churn, conversion or approval can move the opposite way from every segment inside it, because the mix changed.

Studying only hospital patients can create "a spurious negative association" between a disease and a risk factor (as summarized secondarily; the paper has no abstract).

If you only analyze the deals you won, you'll find trade-offs that don't exist in the market.

An exposition of Wald's wartime memos on estimating aircraft vulnerability "using data obtained from survivors."

Your surviving customers can't show you why the others left.

The first controlled experiment on the winner's curse. 419 M.B.A. students in 12 microeconomics classes at Boston University each took part in four sealed-bid auctions, 48 auctions in all. "Unknown to the subjects, all commodities had a value of $8.00": 800 pennies, 160 nickels, 200 large paper clips valued at four cents or 400 small paper clips valued at two cents (p. 624). The highest bidder paid the bid and received the $8.00, and "a $2.00 prize was given for the closest estimate to the true value in each auction" (p. 625).

• The bidders guessed low: "the average value estimate was $5.13 ($2.87 below the true value)" (p. 628).
• The winners still overpaid: "The mean AWB for the 48 auctions was $10.01," and "the average auction resulted in a loss of $2.01 to the winning bidder." In 23 of the 48 auctions the average winning bid was over $9.00 (p. 628).
• The abstract names two causes, "the degree of uncertainty concerning the value of the item up for bid" and "the number of competing bidders." Either one widens the spread of estimates and bids, "making it more likely that the winning bidder will overestimate the true value of the commodity and thus overbid" (p. 618).
• Bidders didn't adjust. "Neither UNC nor K significantly affected the value of the mean bid" (uncertainty and the number of bidders, pp. 628–629), while average winning bids rose with both (equation 4, p. 629).

The winner's curse occurs when "a successful buyer finds that he or she has paid too much for a commodity of uncertain value" (abstract, p. 618).

The subjects were students and the prizes were $8.00. The average winning bid "is sensitive to idiosyncratic bidding behavior (i.e., a handful of grossly inflated bids)", and equation 4 "explains only about 18% of the variation in AWB" (pp. 629–630). The authors assume the low estimates are "specific to our task" (p. 628).

The curse doesn't need an optimistic room. Here the room guessed low and the winners still overpaid, because winning picks the top of the spread. More uncertainty and more competitors make it worse: more variants in a test, more bidders on a lead or a keyword.