Fascinating applications in real-world scenarios, such as hiring, dating, real estate purchases…

Optimal Stopping Problem #

How do we arrive to 37% when tackling an Optimal Stopping Problem? #

The “the optimal stopping problem”, also known as the “marriage problem”, “the sultan’s dowry problem” or “secretary problem” is a famous problem in decision theory and optimal stopping theory. The problem can be framed as follows: You are interviewing a series of candidates (e.g., for a secretary position). You must decide immediately after each interview whether to hire that person, and once rejected, a candidate cannot be reconsidered. The goal is to maximize the probability of selecting the best candidate. The question is, after interviewing how many of the candidates should you start considering them for selection?

To understand why this strategy works, consider the balance between two risks: the risk of picking a candidate too early (and missing out on better ones later) and the risk of waiting too long (and having the best candidates in the initial rejected set). The strategy of rejecting the first 37% candidates and then choosing the next best one balances these risks optimally.

In more technical terms, this strategy maximizes the probability of selecting the best candidate. The probability that the best candidate is not in the first n/e candidates is 1/e, and given that the best candidate is not in the first group, the probability that this strategy will lead you to the best candidate approaches 1 as n increases. Hence, the overall probability of success converges to 1/e, or approximately 37%, as n becomes large.

This problem is a seminal example in the study of the theory of optimal stopping, which explores the tension between the costs of continued search and the benefits of making an optimal choice. The 37% rule has fascinating applications in real-world scenarios, such as hiring, dating, real estate purchases, and other situations where a decision must be made with incomplete information and no possibility of recall.