[PDF] Reverse Engineering of Receiver Operating Characteristic Curves

View - Download Reverse Engineering of Receiver Operating Characteristic Curves

Previous PDF

Next PDF

Reverse Engineering of Receiver Operating

Characteristic Curves

by

Catherine Medlock

Submitted to the Department of Electrical Engineering and Computer

Science

in partial fulfillment of the requirements for the degree of Masters of Engineering in Electrical Engineering and Computer Science at the

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

September 2017

c ○Massachusetts Institute of Technology 2017. All rights reserved. Author ................................................................ Department of Electrical Engineering and Computer Science

September 1, 2017

Certified by............................................................

Alan V. Oppenheim

Ford Professor of Engineering

Thesis Supervisor

Accepted by ...........................................................

Christopher J. Terman

Chairman, Masters of Engineering Thesis Committee

2 Reverse Engineering of Receiver Operating Characteristic

Curves

by

Catherine Medlock

Submitted to the Department of Electrical Engineering and Computer Science on September 1, 2017, in partial fulfillment of the requirements for the degree of Masters of Engineering in Electrical Engineering and Computer Science

Abstract

Receiver operating characteristic (ROC) curves have played a crucial role in the design and evaluation of radar systems for many decades. More recently, their use has spread to a variety of fields including clinical decision-making and machine learning. The common thread in all of these fields is an interest in binary hypothesis testing problems, in which the objective is to use an observation of a random variable of interest, sometimes referred to as a score variable, to infer the answer to a yes-no question. The standard progression of a binary hypothesis testing system from an observation of a score variable, to a set of parameterized decision rules with a binary outputs, to an ROC curve that characterizes the performance of those decision rules is well-understood. Thus, it comes as no surprise that an ROC curve only contains partial information about the problem for which it was designed. In this thesis, a key objective is to find ways of "reverse engineering" ROC curves in order to infer as much information as possible about the underlying binary hypothesis testing problems. We focus specifically on ROC curves that were or could have been constructed using likelihood ratio tests on an actual score variable, which we refer to as LRT-consistent ROC curves. For example, a specific LRT-consistent ROC curve does not uniquely determine the conditional distributions of the score variable used to generate it. A main result is a method for starting with an LRT-consistent ROC curve and using it to construct the conditional distributions of an unlimited number of score variables that could have been used to produce it. One interpretation of the result is as a characterization of the family of score variables that lead to the same ROC curve. This approach is extended to the similar problem of characterizing the family of score variables that lead to the same a set of LRT decision rules.

Thesis Supervisor: Alan V. Oppenheim

Title: Ford Professor of Engineering

3 4

Acknowledgments

I would first like to express my immense gratitude to my research advisor, Alan Op- penheim. I feel extremely lucky to have taken the Discrete-Time Signal Processing course the last time he lectured it in Fall 2015, and even more lucky that an interest- ing project (that would eventually turn into the spark for this thesis) happened to be available at that time. Al has done far more for me than simply providing academic guidance and encouragement. He has been an incredible mentor who showed, and continues to show, me what it means to be a creative researcher and effective teacher. Our twice-weekly discussions about ROC curves were sometimes good-natured argu- ments, usually less than two-and-a-half hours, and always the highlight of my day. I admire Al"s ability to identify promising research directions and explain complex technical concepts in addition to his willingness to bend over backwards to help his students and the people around him. I can"t wait to "wander in the wilderness" with

Al over the next few years.

I have also had the good fortune of working with many members of the Digital Signal Processing Group (in alphabetical order): Tom Baran, Petros Boufounos, Dan Dudgeon, Meir Feder, Hsin-Yu Jane Lai, Tarek Lahlou, Pablo Martinez-Nuevo, Maya Said, Guolong Su, and Jim Ward. Our discussions in the two group meetings related to this thesis led to important technical contributions. Three members of the group had a more direct impact on the thesis. To Guolong: Your identification of the constraints imposed on a score variable by a specific set of LRT decision regions was extremely helpful. Thank you also for explaining your method of designing a deterministic decision rule to achieve an arbitrary achievable operating point to me. To Tom: Your realization that Guolong"s constraints could be discretized and implemented as a linear program ultimately became a main result of the thesis. Thank you for taking the time to meet with me to discuss further ideas about the linear programming approach, and for giving me writing tips. To Jim: Your perspective as an expert in radar was valuable for the thesis and impacted the way I think about ROC curves. Thank you for taking the time to discuss the thesis with me throughout 5 the year, and especially for always caring about and asking how I"m doing, even outside of 6.341 or DSPG. I first became exposed to ROC curves as part of a summer internship with Digital Cognition Technologies, Inc. (DCT). I am grateful to my direct supervisor, William Souillard-Mandar, for being an attentive mentor, a good friend, and a constant source of valuable suggestions for the project. I would also like to thank my other supervisors at DCT (in alphabetical order): Randy Davis, Bruce Musicus, and Dana Penney for their guidance and for helping me to understand the difficult process of taking an idea from a research environment into the real world. Last but not least, I would like to thank my family and friends for their support. To my parents: Thank you for listening so patiently while I tried to work out ideas about ROC curves on the fly, even though they didn"t usually make sense. Your medical expertise provided a valuable perspective on binary hypothesis testing in the clinical setting. To my mother: Thank you for all the times you surprised me with blueberries, baloney sandwiches, M&M"s, and coffee right before a class, or in my office, or sitting on a street corner that you know I pass between classes. It always made my day. To my father: Thank you for giving me so many rides to and from campus over the years, riding in the car with you has become one of my favorite things to do. To Mariko, Kimiko, and Keiko: The fact that all three of you are ready and willing to support me at any time, whether it be through a lunchtime walk along the river or a late night phone call, means a lot. Thanks, and expect a lot more walks and calls in the years to come! To Thomas: Thank you for telling so many goofy jokes, for wearing your hat, for cooking me so many meals, and for letting me listen to songs on repeat. To Miles: Gagu says thank you for our long and enlightening conversations about "gucks." 6

Contents

1 Introduction 15

1.1 Outline and Objectives of the Thesis . . . . . . . . . . . . . . . . . . 18

2 Background 21

2.1 Overview of Binary Hypothesis Testing . . . . . . . . . . . . . . . . . 22

2.1.1 Deterministic vs. Random Decision Rules . . . . . . . . . . . 23

2.2 Minimum Probability of Error (MPE) Decision Rules . . . . . . . . . 24

2.3 Neyman-Pearson Decision Rules . . . . . . . . . . . . . . . . . . . . . 25

2.4 ROC Curves Constructed Using LRTs . . . . . . . . . . . . . . . . . . 27

2.4.1 Necessary Condition for LRT-Consistency . . . . . . . . . . . 27

2.4.2 Sufficient Condition for Equivalence of LRTs and Score Variable

Threshold Tests (SVTs) . . . . . . . . . . . . . . . . . . . . . 29

2.5 The Ideal ROC Curve . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.6 Area Under an ROC Curve . . . . . . . . . . . . . . . . . . . . . . . 32

2.7.1 The Patient and The Two Doctors . . . . . . . . . . . . . . . 39

3 Achievable Operating Points Using Deterministic Decision Rules 43

3.1 "Addition" and "Subtraction" of Achievable Operating Points . . . . . 45

3.2 Method for Achieving an Arbitrary Operating Point in the Achievable

Region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

7

4 Family of Score Variables Consistent with a Given ROC Curve 57

4.1 Sufficient Condition for LRT-Consistency . . . . . . . . . . . . . . . . 58

4.2 Construction of Score Variables Consistent with a Given ROC Curve 60

4.2.1 Special Case: The Likelihood Ratio as a Score Variable . . . . 62

4.3 Alternatives to the AUC . . . . . . . . . . . . . . . . . . . . . . . . . 63

4.3.1 Conditional Expectations of the Likelihood Ratio . . . . . . . 64

4.3.2 Area Under a Geometrically Weighted ROC Curve . . . . . . 66

4.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5 Family of Score Variables Consistent with a Given Set of LRT De-

cision Regions 73

5.1 Linear Programming Formulation . . . . . . . . . . . . . . . . . . . . 75

5.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

6 Summary and Future Work 81

8

List of Figures

1-1 The central component of a binary hypothesis testing system is a de-

cision rule that infers the answer to a yes-no question. . . . . . . . . . 16

1-2 Each operating point on an ROC curve represents the rates of detection

2-1 Formalization of the components of a binary hypothesis testing system. 22

1= 1) and unit variance along with (b) the LRT ROC curve, which

is identical to the SVT ROC curve. . . . . . . . . . . . . . . . . . . . 30

2-3 (a) Two conditional Gaussian PDFs with zero mean and different vari-

curves. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2-4 (a) Example of the ideal scenario in which the supports of the condi-

tional PDFs of the score variable are non-overlapping. (b) All points on the ideal, step function curve can be achieved by including appropriate If only LRTs are used, there are only three achievable operating points. 32

2-5 Achievable region for the Gaussian conditional PDFs in Figure 2-3. . 37

9

2-6 Empirical ROC curves of (a) Doctor A and (b) Doctor B. Each dot rep-

resents the relative frequencies of false alarm and detection achieved by SVTs on the Doctors" respective score variables. The chosen op- erating points are marked by×"s and are both located at the point frequencies of the operating points on the endpoints of the dashed line frequencies of the desired operating point. If he tests Patient Z, he will

2-7 Decision regions of Doctor A and Doctor B. If Patient Z"s heart rate

whether to deem him healthy or ill, i.e., whether to include that heart

3-1 The unique line of slope 1 through a desired operating point (⋆). . . . 47

3-2 The unique pair of points on the curve that have the same change in

3-3 (a) Conditional PDFs of the score variable used in this section and (b)

the associated ROC curve. The '×" marks the desired operating point. 51

3-4 (a) Unique line of slope 1 through the desired operating point, as well

expected, they are consistent with the desired operating point to within a fraction of a percent. . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3-6 The desired operating point could also be reached using randomization

10 falls in the cross-hatched region then the rule decides bility 0.495 and were rounded to tthree significant digits. . . . . . . . . . . . . . . . . 55

4-1 (a) Example of the ideal scenario in which the supports of the condi-

tional PDFs of the score variable are non-overlapping. (b) All points on the ideal, step function curve can be achieved by including appro-

1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4-2 Depiction of an ROC curve as the projection (dashed line) of a three-

curve. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 along with (b) its associated LRT-consistent ROC curve. As expected,

4.2.1, along with (b) its associated LRT-consistent ROC curve. As

Figure 4-3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4-6 Empirical validation that the conditional PDFs of the likelihood ratio

are as shown in Figure 4-5. . . . . . . . . . . . . . . . . . . . . . . . . 71 11

4-7 (a) Five ROC curves, each constructed using LRTs on a score variable

whose conditional PDFs are Gaussian distributions with means 0 and vs. AUC. The monotonic relationship between the two metrics shows consistency between them in the case of ROC curves that do not in- AUC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

5-1 Abstraction of the implementation of the set of LRT decision rules for

used to establish the likelihood ratio constraint. (b) Solution of the linear program in Equation 5.9 ( distribution between 0 and 1. The resulting vector was normalized to have unit length. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 used to establish the likelihood ratio constraint. (b) Solution of the linear program in Equation 5.9 ( distribution between 0 and 1. The resulting vector was normalized to have unit length. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 12 used to establish the likelihood ratio constraint. (b) Solution of the linear program in Equation 5.9 ( distribution between 0 and 1. The resulting vector was normalized to have unit length. The PDFs have been truncated to a range in which their ratio lies in a small range around 1, since we observed thatquotesdbs_dbs33.pdfusesText_39

[PDF] le vademecum ct

[PDF] exemple de roc svt 1ere s

[PDF] mise en situation oral concours fonction publique

[PDF] exemple récit historique

[PDF] cours méthodologie 1ère st

[PDF] étapes démarche de projet

[PDF] etapes de la demarche detude st2s

[PDF] exemple de motivation pour loral daide soignante

[PDF] fiche revision concours aide soignante oral

[PDF] sujet oral concours aide soignante 2016

[PDF] sujet oral aide soignante corrigé

[PDF] préparer loral du concours daide soignante

[PDF] synthese escp 2009 corrigé

[PDF] rapport de jury synthèse escp

[PDF] correction synthèse escp 2015