资源 || MedCalc作ROC曲线

今天学习用MedCalc软件做ROC曲线，MedCalc处理ROC曲线方面，操作简便，具有一定的优势。
1.数据录入
MedCalc的数据录入界面与SPSS类似，只是它没有变量视图，只有一个数据录入界面，这是与SPSS区别的地方，那么我们在第一行直接命名变量标签即可。需要指出Diagnosis变量指的是以金标准（如病理诊断）判断是否患病的结果。Test1,2,...3指得是不同诊断方法所得到的结果。

---

---

2. 参数选择
2.1 单一诊断试验评估
在标签栏中选择Stastistics——ROC curves——ROC curve analyze，出现以下操作界面，那么在Variable一栏中选择我们研究的变量（也就是诊断方式）；Classification variable一栏中选择诊断结局（是否患病等）；Select一栏中选择我们要进行的亚组分析，如性别，年龄段等。其他一些框框大家可以自己勾选看看是什么效果

ROC图是以灵敏度（真阳性率）为纵坐标，1-特异度（假阳性率）为横坐标，我们可以得到某诊断试验的最佳诊断标准，以及在该标准下该诊断试验的灵敏度和特异度（如图所示）。在该图中，最接近左上角（0,100）的点所对应的诊断标准即为最佳诊断标准。

2.2 多个诊断试验的比较

多诊断试验的比较与单一诊断试验的区别在于ROC curves选项下选择Comparison of ROC curves，然后进行variable和Classification variable的输入，这里Variable需要将每一个诊断试验指标逐一选入（如图）。

最后是结果的解读，在第一个表里，我们可以看到Test1-3的AUC(曲线下面积)，ROC曲线下面积反映诊断试验的价值大小，（0.50，0.70]，表示诊断价值较低；（0.70，0.90]，表示诊断价值中等； 0.90以上表示诊断价值较高。第二张表列出了诊断试验两两比较是否有差异。

贴张自己做的最终效果图

还是很不错的，图形格式调整等等很方便的。

---

---

Medcalc官网ROC分析文档：

ROC curve analysis in MedCalc

---

Command:

Statistics ROC curves ROC curve analysis

Description

Allows to create ROC curve and a complete sensitivity/specificity report. The ROC curve is a fundamental tool for diagnostic test evaluation.

In a ROC curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points of a parameter. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. The area under the ROC curve (AUC) is a measure of how well a parameter can distinguish between two diagnostic groups (diseased/normal).

Theory summary

The diagnostic performance of a test, or the accuray of a test to discriminate diseased cases from normal cases is evaluated using Receiver Operating Characteristic (ROC) curve analysis (Metz, 1978; Zweig & Campbell, 1993). ROC curves can also be used to compare the diagnostic performance of two or more laboratory or diagnostic tests (Griner et al., 1981).

When you consider the results of a particular test in two populations, one population with a disease, the other population without the disease, you will rarely observe a perfect separation between the two groups. Indeed, the distribution of the test results will overlap, as shown in the following figure.

For every possible cut-off point or criterion value you select to discriminate between the two populations, there will be some cases with the disease correctly classified as positive (TP = True Positive fraction), but some cases with the disease will be classified negative (FN = False Negative fraction). On the other hand, some cases without the disease will be correctly classified as negative (TN = True Negative fraction), but some cases without the disease will be classified as positive (FP = False Positive fraction).

Schematic outcomes of a test

The different fractions (TP, FP, TN, FN) are represented in the following table.

	Disease
Test	Present	n		Absent	n		Total
Positive	True Positive (TP)	a		False Positive (FP)	c		a + c
Negative	False Negative (FN)	b		True Negative (TN)	d		b + d
Total		a + b			c + d

 

The following statistics can be defined:

Sensitivity	a a + b		Specificity	d c + d
Positive Likelihood Ratio	Sensitivity 1 - Specificity		Negative Likelihood Ratio	1 - Sensitivity Specificity
Positive Predictive Value	a a + c		Negative Predictive Value	d b + d

• Sensitivity: probability that a test result will be positive when the disease is present (true positive rate, expressed as a percentage). 
  = a / (a+b)

• Specificity: probability that a test result will be negative when the disease is not present (true negative rate, expressed as a percentage). 
  = d / (c+d)

• Positive likelihood ratio: ratio between the probability of a positive test result given the presence of the disease and the probability of a positive test result given the absence of the disease, i.e. 
  = True positive rate / False positive rate = Sensitivity / (1-Specificity)

• Negative likelihood ratio: ratio between the probability of a negative test result given the presence of the disease and the probability of a negative test result given the absence of the disease, i.e. 
  = False negative rate / True negative rate = (1-Sensitivity) / Specificity

• Positive predictive value: probability that the disease is present when the test is positive (expressed as a percentage). 
  = a / (a+c)

• Negative predictive value: probability that the disease is not present when the test is negative (expressed as a percentage). 
  = d  / (b+d)

The ROC curve

In a Receiver Operating Characteristic (ROC) curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. A test with perfect discrimination (no overlap in the two distributions) has a ROC curve that passes through the upper left corner (100% sensitivity, 100% specificity). Therefore the closer the ROC curve is to the upper left corner, the higher the overall accuracy of the test (Zweig & Campbell, 1993).

How to enter data for ROC curve analysis

In the spreadsheet, create a column DIAGNOSIS and a column for the variable of interest, e.g. TEST1. For every study subject enter a code for the diagnosis as follows: 1 for the diseased cases, and 0 for the non-diseased or normal cases. In the TEST1 column, enter the measurement of interest (this can be measurements, grades, etc. - if the data are categorical, code them with numerical values).

Required input

Complete the ROC curve analysis dialog box as follows:

Data

• Variable: select the variable of interest.

• Classification variable: select or enter a a dichotomous variable indicating diagnosis (0=negative, 1=positive).

  
  

  If your data are coded differently, you can use the Define status tool to recode your data.

• Filter: (optionally) a filter in order to include only a selected subgroup of cases (e.g. AGE>21, SEX='Male').

Methodology:

• DeLong et al.: use the method of DeLong et al. (1988) for the calculation of the Standard Error of the Area Under the Curve (recommended).

• Hanley & McNeil: use the method of Hanley & McNeil (1982) for the calculation of the Standard Error of the Area Under the Curve.

• Binomial exact Confidence Interval for the AUC: calculate an exact Binomial Confidence Interval for the Area Under the Curve (recommended). If this option is not selected, the Confidence Interval is calculated as AUC ± 1.96 its Standard Error.

Disease prevalence

Whereas sensitivity and specificity, and therefore the ROC curve, and positive and negative likelihood ratio are independent of disease prevalence, positive andnegative predictive values are highly dependent on disease prevalence or prior probability of disease. Therefore when disease prevalence is unknown, the program cannot calculate positive and negative predictive values.

Clinically, the disease prevalence is the same as the probability of disease being present before the test is performed (prior probability of disease).

• Unknown: select this option when the disease prevalence is unknown, or irrelevant for the current statistical analysis.

• The ratio of cases in the positive and negative groups reflects the prevalence of the disease: if the sample sizes in the positive and the negative group reflect the real prevalence of the disease in the population, this can be indicated by selecting this option.

• Other value (%): alternatively you can enter a value for the disease prevalence, expressed as a percentage.

  
  

Options

• List criterion values with test characteristics: option to create a list of criterion values corresponding with the coordinates of the ROC curve, with associated sensitivity, specificity, likelihood ratios and predictive values (if disease prevalence is known).

• Include all observed criterion values: When you select this option, the program will list sensitivity and specificity for all possible threshold values. If this option is not selected, then the program will only list the more important points of the ROC curve: for equal sensitivity/specificity it will give the threshold values (criterion values) with the highest specificity/sensitivity.

• 95% Confidence Interval for sensitivity/specificity, likelihood ratio and predictive values: select the Confidence Intervals you require.

• Calculate optimal criterion value taking into account costs: option to calculate the optimal criterion value taking into account the disease prevalence and cost of false and true positive and negative decisions (Zweig & Campbell, 1993). This option is only available if disease prevalence is known (see above).These data are used to calculate a parameter S as follows:

  
  

  

  where P denotes the prevalence in the target population (Greiner et al., 2000). The point on the ROC curve where a line with this slope S touches the curve is the optimal operating point, taking into account prevalence and the costs of the different decisions.

  
  

  Costs can be financial costs or health costs, but all 4 cost factors need to be expressed on a common scale. Benefits can be expressed as negative costs. Suppose a false negative (FN) decision is judged to be twice as costly as a false positive (FP) decision, and no assumptions are made about the costs for true positive and true negative decisions. Then for FNc you enter 2, for FPc enter 1 and enter 0 for both TPc and TNc.

  
  

  Because the slope S must be a positive number:

  
  

  The parameter S is 'cost-neutral' when (FPc-TNc)/(FNc-TPc) evaluates to 1, that is when FPc-TNc equals FNc-TPc. In this case S, and the 'optimal criterion value' depends only on the disease prevalence.

  
  

• FPc cannot be equal to TNc

• FNc cannot be equal to TPc

• When TNc is larger than FPc then TPc must be larger than FNc

• When TNc is smaller than FPc then TPc must be smaller than FNc

• FPc: the cost of a false positive decision.

• FNc: the cost of a false negative decision.

• TPc: the cost of a true positive decision.

• TNc: the cost of a true negative decision.

ROC graph

• Select Display ROC curve window to obtain the graph in a separate window.

  
  

  Options:

  
  

• mark points corresponding to criterion values.

• display 95% Confidence Bounds for the ROC curve (Hilgers, 1991).

Results

Sample size

First the program displays the number of observations in the two groups. Concerning sample size, it has been suggested that meaningful qualitative conclusions can be drawn from ROC experiments performed with a total of about 100 observations (Metz, 1978).

Area under the ROC curve, with standard error and 95% Confidence Interval

This value can be interpreted as follows (Zhou, Obuchowski & McClish, 2002):

• the average value of sensitivity for all possible values of specificity;

• the average value of specificity for all possible values of sensitivity;

• the probability that a randomly selected individual from the positive group has a test result indicating greater suspicion than that for a randomly chosen individual from the negative group.

When the variable under study cannot distinguish between the two groups, i.e. where there is no difference between the two distributions, the area will be equal to 0.5 (the ROC curve will coincide with the diagonal). When there is a perfect separation of the values of the two groups, i.e. there no overlapping of the distributions, the area under the ROC curve equals 1 (the ROC curve will reach the upper left corner of the plot).

The 95% Confidence Interval is the interval in which the true (population) Area under the ROC curve lies with 95% confidence.

The Significance level or P-value is the probability that the observed sample Area under the ROC curve is found when in fact, the true (population) Area under the ROC curve is 0.5 (null hypothesis: Area = 0.5). If P is small (P<0.05) then="" it="" can="" be="" concluded="" that="" the="" area="" under="" the="" roc="" curve="" is="" significantly="" different="" from="" 0.5="" and="" that="" therefore="" there="" is="" evidence="" that="" the="" laboratory="" test="" does="" have="" an="" ability="" to="" distinguish="" between="" the="" two="">

Youden index

The Youden index J (Youden, 1950) is defined as:

J = max { sensitivity_c + specificity_c - 1 }

where c ranges over all possible criterion values.

Graphically, J is the maximum vertical distance between the ROC curve and the diagonal line.

The criterion value corresponding with the Youden index J is the optimal criterion value only when disease prevalence is 50%, equal weight is given to sensitivity and specificity, and costs of various decisions are ignored.

When the corresponding Advanced option has been selected, MedCalc will calculate BC_a bootstrapped 95% confidence intervals (Efron, 1987; Efron & Tibshirani, 1993) for both the Youden index and it's corresponding criterion value.

Optimal criterion

This panel is only displayed when disease prevalence and cost parameters are known.

The optimal criterion value takes into account not only sensitivity and specificity, but also disease prevalence, and costs of various decisions. When these data are known, MedCalc will calculate the optimal criterion and associated sensitivity and specificity. And when the corresponding Advanced option has been selected, MedCalc will calculate BC_a bootstrapped 95% confidence intervals (Efron, 1987; Efron & Tibshirani, 1993) for these parameters.

Summary table

This panel is only displayed when the corresponding Advanced option has been selected.

The summary table displays the estimated specificity for a range of fixed and pre-specified sensitivities of 80, 90, 95 and 97.5% as well as estimated sensitivity for a range of fixed and pre-specified specificities (Zhou et al., 2002), with the corresponding criterion values.

Confidence intervals are BC_a bootstrapped 95% confidence intervals (Efron, 1987; Efron & Tibshirani, 1993).

Criterion values and coordinates of the ROC curve

This section of the results window lists the different filters or cut-off values with their corresponding sensitivity and specificity of the test, and the positive (+LR) and negative likelihood ratio (-LR). When the disease prevalence is known, the program will also report the positive predictive value (+PV) and the negative predictive value (-PV).

When you did not select the option Include all observed criterion values, the program only lists the more important points of the ROC curve: for equal sensitivity (resp. specificity) it gives the threshold value (criterion value) with the highest specificity (resp. sensitivity). When you do select the option Include all observed criterion values, the program will list sensitivity and specificity for all possible threshold values.

• Sensitivity (with optional 95% Confidence Interval): Probability that a test result will be positive when the disease is present (true positive rate).

• Specificity (with optional 95% Confidence Interval): Probability that a test result will be negative when the disease is not present (true negative rate).

• Positive likelihood ratio (with optional 95% Confidence Interval): Ratio between the probability of a positive test result given the presence of the disease and the probability of a positive test result given the absence of the disease.

• Negative likelihood ratio (with optional 95% Confidence Interval): Ratio between the probability of a negative test result given the presence of the disease and the probability of a negative test result given the absence of the disease.

• Positive predictive value (with optional 95% Confidence Interval): Probability that the disease is present when the test is positive.

• Negative predictive value (with optional 95% Confidence Interval): Probability that the disease is not present when the test is negative.

• Cost^*: The average cost resulting from the use of the diagnostic test at that decision level. Note that the cost reported here excludes the 'overhead cost', i.e. the cost of doing the test, which is constant at all decision levels.

  
  

ROC curve

The ROC curve will be displayed in a second window when you have selected the corresponding option in the dialog box.

In a ROC curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (100-Specificity) for different cut-off points. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. A test with perfect discrimination (no overlap in the two distributions) has a ROC curve that passes through the upper left corner (100% sensitivity, 100% specificity). Therefore the closer the ROC curve is to the upper left corner, the higher the overall accuracy of the test (Zweig & Campbell, 1993).

When you click on a specific point of the ROC curve, the corresponding cut-off point with sensitivity and specificity will be displayed.

Presentation of results

The prevalence of a disease may be different in different clinical settings. For instance the pre-test probability for a positive test will be higher when a patient consults a specialist than when he consults a general practitioner. Since positive and negative predictive values are sensitive to the prevalence of the disease, it would be misleading to compare these values from different studies where the prevalence of the disease differs, or apply them in different settings.

The data from the results window can be summarized in a table. The sample size in the two groups should be clearly stated. The table can contain a column for the different criterion values, the corresponding sensitivity (with 95% CI), specificity (with 95% CI), and possibly the positive and negative predictive value. The table should not only contain the test's characteristics for one single cut-off value, but preferably there should be a row for the values corresponding with a sensitivity of 90%, 95% and 97.5%, specificity of 90%, 95% and 97.5%, and the value corresponding with the Youden index or highest accuracy.

With these data, any reader can calculate the negative and positive predictive value applicable in his own clinical setting when the knows the prior probability of disease (pre-test probability or prevalence of disease) in this setting, by the following formulas based on Bayes' theorem:

and

The negative and positive likelihood ratio must be handled with care because they are easily and commonly misinterpreted.