Area under the receiver operating characteristic (ROC) curve – A versatile tool, by Jackson Smith*
The method used in classic studies [1, 2] to quantify the discrimination sensitivity of middle temporal (MT) neurones in a two-alternative, forced-choice (2AFC) task has since become an important technique of behavioural neurophysiology. The key question is whether a neurone fired more spikes in one condition than in another. However, traditional parametric methods for answering this question place restrictive assumptions on the statistics of neural activity; for example, neurones do not always resemble a Poisson process .
Figure 1. Area under the receiver operating characteristic (ROC) curve. A, Hypothetical example of two spike-count distributions from trials grouped by conditions X and Y. Spike counts range between 0 and the maximum value, cmax. B, The curved line, located above the dashed ‘chance’ line, represents the ROC curve that is constructed from the distributions in Panel A by classifying their values with the ideal observer (see Appendix). Classification performance is tested for every possible value of the classification criterion, c, which includes all possible spike counts between 0 and cmax. Thus, each value of c corresponds to a point in the ROC curve; the arrow shows how increasing values of c are mapped. The grey region is the area under the ROC curve. C, Behavioural sensitivity (or stimulus sensitivity) are defined as the area under the ROC curve that compares a distribution of failed-trial (or noise) spike counts (grey) versus a distribution of correct-trial (or signal) spike counts (open). The area under the ROC curve quantifies the difference between the two distributions.
Receiver operating characteristic (ROC) curves provide an unbiased, non-parametric way of quantifying the difference between any two distributions [for mathematic derivation, see 4] – in this case, the number of spikes fired by a neurone on one set of trials versus another. Indeed ROC analysis can be used to evaluate such diverse applications as medical imaging, materials testing, weather forecasting, information retrieval, polygraph lie detection, and aptitude testing , to name a few. Figure 1 illustrates how an ROC curve is used to quantify the difference between two distributions of spike counts (Figure 1A).
Faced with the problem of classifying a randomly sampled spike count as being from either distribution X (open) or Y (filled), the strategy adopted by the ideal observer is to choose a criterion level of c and assign any spike count less than c to X, and any spike count above c to Y. In other words, the decision rule used by the ideal observer is to assign spike count s (randomly drawn from either X or Y) to distribution X if s < c, or to distribution Y if s > c. All possible values of c between 0 and the maximum spike count (cmax) must be tested to find the optimal criterion that makes correct classifications the most often.
To do so, an ROC curve (Figure 1B) is built by plotting the probability that spike counts sampled from X are greater than c (false positives) against the probability that spike counts sampled from Y are greater than c (true positives). When c = 0, all spike counts are greater than c, thus the beginning point of the ROC curve is always (1,1). As c is increased (Figure 1B, arrow) the performance of the ideal observer using each criterion level is plotted. When c reaches cmax, no spike count is less than c and the end point of the curve is (0,0).
The area under the ROC curve (Figure 1B, grey shading) is the probability that the ideal observer will correctly classify any given spike count, randomly drawn from either distribution, and ranges between 0 and 1 accordingly; this probability is 0.75 for example distributions X and Y from Figure 1A. Therefore, when X and Y are completely distinct from each other, the ideal observer correctly classifies 100% of all spike counts (area = 1, Figure 1C, right). On the other hand, if there is no distinction between X and Y, then the ideal observer has a 50% chance of correct classification – a coin toss (area = 0.5, left). If X and Y from Figure 1A switch positions then the difference between them remains the same; this is reflected by the area under the ROC curve, which is an equal distance below 0.5 after the switch (0.25 = 0.5 – 0.25) as it was before (0.75 = 0.5 + 0.25).
Example 1 – A neurone’s stimulus sensitivity
The classic studies of Newsome and colleagues demonstrated the power of a careful comparison between neural activity and perceptual behaviour [1, 2]. Experiments were performed to carefully measure the discrimination sensitivity of MT neurones from monkey subjects performing a 2AFC motion-discrimination task. The subjects had to report whether the coherent motion in a patch of randomly moving dots was in the preferred or null (preferred + 180°) direction of an isolated MT neurone. It was critical to match the direction, speed, and location of dot motion to the neurone’s RF preferences. This ensured that the subject was responding to the same stimulus as the neurone. But more importantly: it maximised the chance that spikes recorded from the neurone were used by the subject to perform the task.
The direction of motion was randomly drawn on every trial so that the subject would have to watch the coherent dots carefully in order to make a correct choice. However, the strength of coherent motion was also varied from trial to trial by changing the percentage of dots that moved together. This varied the difficulty of the task and therefore the subject’s performance, which provided a frame of reference. The neurone’s ability to discriminate the direction of coherent dot motion at any one difficulty level could be directly compared against the performance of the subject.
A receiver operating characteristic (ROC) analysis (Figure 1) was used to quantify the discrimination sensitivity of MT neurones in the 2AFC task. For this, two distributions of spike counts were compared against each other, the distribution of counts from trials when the coherent motion was in the neurone’s preferred direction (distribution Y in Figure 1A) versus the distribution of counts from trials with coherent motion in the null direction (distribution X in Figure 1A). The resulting ROC areas (Figure 1B) described the probability that an ideal observer could tell which direction had been presented to the subject, based on the distribution of MT spike counts. This was computed separately for each level of coherent motion strength and compared directly against the subject’s performance. It was found that the average MT neurone could account for the subject’s discrimination sensitivity – at least under the particular conditions of the experiment [see 6].
Example 2 – A neurone’s behavioural sensitivity
The classic studies of Newsome and colleagues highlighted the large variation in the choices made by subjects and in the number of spikes fired by MT neurones. In response to statistically identical stimuli with low signal to noise ratios, subjects would sometimes report the wrong direction and their neurones would sometimes fire as if the opposite direction had been shown. However, this variation presented an exciting opportunity – because the ROC curve is a versatile tool and can be used to compare any two distributions of neural activity. Celebrini and Newsome  performed a ground-breaking analysis: they measured the correlation between the number of spikes fired by a neurone and the choice that the subject was about to make.
They began by grouping trials based on the ‘preferred’ or ‘null’ motion discrimination report made by the subject. Then they computed the ROC curve comparing the distribution of null-trial spike counts (distribution X in Figure 1A) versus the distribution of preferred-trial spike counts (distribution Y in Figure 1A). The area under this ROC curve is the probability that the ideal observer could correctly predict which direction of motion the subject would choose, using spike counts. This kind of ROC metric was named ‘choice probability’ when it was later used to analyse MT neurones . However, we will refer to this ROC metric, and other like it, as ‘behavioural sensitivity’, because it measures how much the neural response predicts perceptual behaviour. It is important to keep in mind that behavioural sensitivity does not measure the correlation between spike counts and perception itself – only the perceptual report, which may not always be faithful to what was actually perceived.
Similar to stimulus sensitivity, a behavioural sensitivity of 0.5 shows that there was no difference in the number of spikes fired prior to either choice (Figure 1C, left). If more spikes were fired prior to choices coinciding with the neurone’s preferred direction, then behavioural sensitivity would rise towards 1, to indicate a positive correlation (Figure 1C, middle and right). On the other hand, if more spikes were fired prior to null direction choices, then behavioural sensitivity would sink towards 0, to indicate a negative correlation. On average, MT neurones had a weak but significant, positive correlation with the subject’s upcoming choice of motion direction .
Since then, behavioural sensitivities have been observed between MT spike counts and the subject’s upcoming choice when discriminating coherent dot motion direction [9-11], speed [12, 13], disparity [14, 15], and cylindrical rotation [16-18]. Similar behavioural sensitivities have been observed between a subject’s discrimination performance and spike counts from cortical areas V2 [19, 20] and MST [7, 21, 22], and even spike counts from somatosensory cortex [23, 24]. Using behavioural sensitivity, correlations have also been observed between MT spike counts and the subject’s ability to detect a change in coherent motion strength [6, 25] and speed , while similar behavioural sensitivities have been observed between a subject’s detection performance and spike counts from cortical areas V1 , V4 [28, 29], and VIP .
This blog is adapted from a chapter in the open access book ‘Visual Cortex’  in accordance with its Creative Commons Attribution 3.0 Unported licence.
1. Newsome, W.T., K.H. Britten, and J.A. Movshon, Neuronal correlates of a perceptual decision. Nature, 1989. 341(6237): p. 52-4.
2. Britten, K.H., et al., The Analysis of Visual-Motion – a Comparison of Neuronal and Psychophysical Performance. Journal of Neuroscience, 1992. 12(12): p. 4745-4765.
3. Kara, P., P. Reinagel, and R.C. Reid, Low response variability in simultaneously recorded retinal, thalamic, and cortical neurons. Neuron, 2000. 27(3): p. 635-646.
4. Green, D.M. and J.A. Swets, Signal detection theory and psychophysics. 1966, New York: Wiley. 455.
5. Swets, J.A., Measuring the accuracy of diagnostic systems. Science, 1988. 240(4857): p. 1285-93.
6. Cook, E.P. and J.H.R. Maunsell, Dynamics of neuronal responses in macaque MT and VIP during motion detection. Nature Neuroscience, 2002. 5(10): p. 985-994.
7. Celebrini, S. and W.T. Newsome, Neuronal and Psychophysical Sensitivity to Motion Signals in Extrastriate Area Mst of the Macaque Monkey. Journal of Neuroscience, 1994. 14(7): p. 4109-4124.
8. Britten, K.H., et al., A relationship between behavioral choice and the visual responses of neurons in macaque MT. Visual Neuroscience, 1996. 13(1): p. 87-100.
9. Purushothaman, G. and D.C. Bradley, Neural population code for fine perceptual decisions in area MT. Nature Neuroscience, 2005. 8(1): p. 99-106.
10. Cohen, M.R. and W.T. Newsome, Estimates of the Contribution of Single Neurons to Perception Depend on Timescale and Noise Correlation. Journal of Neuroscience, 2009. 29(20): p. 6635-6648.
11. Law, C.T. and J.I. Gold, Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nature Neuroscience, 2008. 11(4): p. 505-513.
12. Liu, J. and W.T. Newsome, Correlation between speed perception and neural activity in the middle temporal visual area. J Neurosci, 2005. 25(3): p. 711-22.
13. Price, N.S.C. and R.T. Born, Timescales of Sensory- and Decision-Related Activity in the Middle Temporal and Medial Superior Temporal Areas. Journal of Neuroscience, 2010. 30(42): p. 14036-14045.
14. Uka, T. and G.C. DeAngelis, Contribution of area MT to stereoscopic depth perception: Choice-related response modulations reflect task strategy. Neuron, 2004. 42(2): p. 297-310.
15. Sasaki, R. and T. Uka, Dynamic Readout of Behaviorally Relevant Signals from Area MT during Task Switching. Neuron, 2009. 62(1): p. 147-157.
16. Dodd, J.V., et al., Perceptually bistable three-dimensional figures evoke high choice probabilities in cortical area MT. J Neurosci, 2001. 21(13): p. 4809-21.
17. Parker, A.J., K. Krug, and B.G. Cumming, Neuronal activity and its links with the perception of multi-stable figures. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences, 2002. 357(1424): p. 1053-1062.
18. Krug, K., B.G. Cumming, and A.J. Parker, Comparing perceptual signals of single V5/MT neurons in two binocular depth tasks. Journal of Neurophysiology, 2004. 92(3): p. 1586-1596.
19. Nienborg, H. and B.G. Cumming, Macaque V2 neurons, but not V1 neurons, show choice-related activity. Journal of Neuroscience, 2006. 26(37): p. 9567-9578.
20. Nienborg, H. and B.G. Cumming, Decision-related activity in sensory neurons reflects more than a neuron’s causal effect. Nature, 2009. 459(7243): p. 89-U93.
21. Gu, Y., D.E. Angelaki, and G.C. Deangelis, Neural correlates of multisensory cue integration in macaque MSTd. Nat Neurosci, 2008. 11(10): p. 1201-10.
22. Gu, Y., et al., Perceptual learning reduces interneuronal correlations in macaque visual cortex. Neuron, 2011. 71(4): p. 750-761.
23. de Lafuente, V. and R. Romo, Neuronal correlates of subjective sensory experience. Nature Neuroscience, 2005. 8(12): p. 1698-1703.
24. Hernandez, A., et al., Decoding a Perceptual Decision Process across Cortex. Neuron, 2010. 66(2): p. 300-314.
25. Bosking, W.H. and J.H. Maunsell, Effects of stimulus direction on the correlation between behavior and single units in area MT during a motion detection task. J Neurosci, 2011. 31(22): p. 8230-8.
26. Herrington, T.M. and J.A. Assad, Neural Activity in the Middle Temporal Area and Lateral Intraparietal Area during Endogenously Cued Shifts of Attention. Journal of Neuroscience, 2009. 29(45): p. 14160-14176.
27. Palmer, C., S.Y. Cheng, and E. Seidemann, Linking neuronal and behavioral performance in a reaction-time visual detection task. Journal of Neuroscience, 2007. 27(30): p. 8122-8137.
28. Cohen, M.R. and J.H.R. Maunsell, A Neuronal Population Measure of Attention Predicts Behavioral Performance on Individual Trials. Journal of Neuroscience, 2010. 30(45): p. 15241-15253.
29. Cohen, M. and J. Maunsell, Using Neuronal Populations to Study the Mechanisms Underlying Spatial and Feature Attention. Neuron, 2011. 70(6): p. 1192-1204.
30. Smith, J.E.T., N.Y. Masse, and C.C. Zhan, E. P., Linking neural activity to visual perception: separating sensory and attentional contributions, in Visual Cortex, S. Molotchnikoff and J. Rouat, Editors. 2012, InTech.