Decision Weights

A decision maker assigns decision weights to stated probabilities through a weighting function, whereas “the decision weights are not probabilities” in that “they do not obey the probability axioms and they should not be interpreted as measures of degree or belief.”

Kahneman and Tversky, 1979
Probability1%2%5%10%20%50%80%90%95%98%100%
Decision Weight5.5%8.1%13.2%18.6%26.1%42.1%60.1%71.2%79.3%87.1%100%

You can see that the decision weights are identical to the corresponding probabilities at the extremes: both equal to 0 when the outcome is impossible, and both equal to 100 when the outcome is a sure thing. However, decision weights depart sharply from probabilities near these points. At the low end, we find the possibility effect: unlikely events are considerably over-weighted. For example, the decision weight that corresponds to a 2% chance is 8.1. If people conformed to the axioms of rational choice, the decision weight would be 2%—so the rare event is over-weighted by a factor of 4. The certainty effect at the other end of the probability scale is even more striking. A 2% risk of not winning the prize reduces the utility of the gamble by 13%, from 100 to 87.1. (Kahneman)

Data Science Curriculum

Mathematics

  • Single Variable Calculus (prerequisite: Algebra and Trigonometry) 101
  • Multi Variable Calculus 101
  • Linear Algebra 201
  • Differential Equations 201
  • Statistics 301
  • Probability 301

Computer & Data Science

  • Python
    • Introduction (basics, iPython, Jupyter Notebooks, Git) 101
    • Numpy (computation) 201
    • Pandas (data manipulation) 201
    • Matplotlib (visualization) 201
    • Scikit-learn (machine learning) 401
    • Keras (neural networks) [Tensorflow prerequisite] 401
  • Databases 301
  • Machine Learning 401
  • Neural Networks 401

Reference

https://towardsdatascience.com/data-science-curriculum-from-scratch-2018-part-1-35061303c385

Resources

Intuitive Predictions

A corrective procedure for prediction

This is the process by Kahneman and Teversky for the Elicitation and correction of intuitive predictions.

Intuitive predictions do not take regression to the mean into account, and this method of correcting errors by de-biasing your predictions incorporates regression into your forecast.

Any significant activity of forecasting involves a large component of judgment, intuition and educated guesswork.

Step 1: selection of a reference class

Step 2: Assessment of the distribution for the reference class

In particular, the expert should provide an estimate of the class average, and some additional estimates that reflect the range of variability of outcomes. Sample questions are: “How many copies are sold, on the average, for books in this category?” “What proportion of the books in that class sell more than 15,000 copies?”. You are trying to determine the base rate for the reference class you selected in step 1.

Step 3: Intuitive estimation

This is where you provide your intuitive estimate. Your intuitive estimate is likely to be non-regressive. The objective of the next two steps of the procedure is to correct this bias and obtain a more reasonable estimate.

Step 4: Assessment of Predictability

You should construct a rough scale of predictability based on computed correlations between predictions and outcomes for a set of phenomena that range from highly predictable (e.g., temperature) to highly unpredictable (e.g., stock prices). you would then be in a position to locate the predictability of the target quantity on this scale, thereby providing a numerical estimate of p.

An alternative method for assessing predictability involves questions such as “if you were to consider two novels that you are about to publish, how often would you be right in predicting which of the two will sell more copies?”An estimate of the ordinal correlation between predictions and outcomes can now be obtained as follows: If p is the estimated proportion of pairs in which the order of outcomes was correctly predicted, then T = 2p -1 provides an index of predictive accuracy, which ranges from zero when predictions are at chance level to unity when predictions are perfectly accurate. In many situations T can be used as a crude approximation for p.

Step 5: Correction of the intuitive estimate

To correct for non-regressiveness, the intuitive estimate should be adjusted toward the average of the reference class. If the intuitive estimate was non-regressive, then under fairly general conditions the distance between the intuitive estimate and the average of the class should be reduced by a factor of p, where p is the correlation coefficient. This procedure provides an estimate of the quantity, which is hopefully free of the non-regression error.

For example, suppose that the expert’s intuitive prediction of the sales of a given book is 12,000 and that,on the average, books in that category sell 4,000 copies.Suppose further that the expert believes that he would correctly order pairs of manuscripts by their future sales on 80% of comparisons. In this case, T = 1.6 -1 = .6, and the regressed estimate of sales would be:

Corrected estimate = Base rate + correlation(intuitive prediction – base rate)

4,000 + .6(12,000 -4,000) =8,800.

Reference

INTUITIVE PREDICTION: BIASES AND CORRECTIVE PROCEDURES
by Daniel Kahneman and Amos Tvertsky https://apps.dtic.mil/dtic/tr/fulltext/u2/a047747.pdf

Brier Score

Use the Brier Score to measure your forecasting ability. Scores range from 0 to 1, the lower your score the more accurate your predictions are. A score of .50 is no better than a coin toss, so your average result should be lower (better) than 50%. Your goal should be to have an average forecasting score of .20 or less.

Formula
Brier Score =(probability − outcome )^2 +(probability − outcome )^2 / N

Where outcome = 1 if win or 0 if loss

N = number of items

For example an 81% win forecast versus a 19% loss where the underdog wins

(win probability – 0)^2 + (loss probability – 1)^2
(.81 – 0)^2 + (.19-1)^2 / 2 = .66

If the result would have been a win for the favorite, then your brier score would be:
(.81 – 1)^2 + (.19-0)^2 / 2 = .036

Resources

https://en.wikipedia.org/wiki/Brier_score