ML Performance Measures for Diagnosis

https://www.youtube.com/watch?v=uULhuuSjBww

6 flashcards · Computer Science · by charlie42

1. What does sensitivity measure, in words

   Probability that the test is positive if the person has the disease

2. What does specificity measure, in words

   Probability that the test is negative if the person does not have the disease

3. Is what study setting sensitivity and specificity are appropriate measures?

   In case-control studies, when disease is upsampled (50% sick, 50% controls). It's not appropriate in cohort studies where positive case is low.

4. Why sensitivity and specifiicity are confusing for clinicians?

   Because clinicians don't care about the probability of the test being positive, they care about the probability of a person having a disease

5. What's a proper scoring rule?

   A rule that incentivizes the forecaster to always predict correct probabilities (a calibrated model)

6. In calibration curve, how true probabilities are calculated?

   It's a proportion of true positive cases in each bin (if using 3 bins prob_true will have 3 values)