Statistics: Results

Experiment 1

Agreements Disagreements Kappa
Detector Projects Synthetics Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 29 25 64 40 0 0 15 20 35 3 2 5 0.875 0.191 0.845 15 23.4%
GrouMiner 29 25 64 46 0 0 28 12 40 5 1 6 0.87 0.161 0.845 31 48.4%
Jadet 29 25 64 19 0 0 12 3 15 2 2 4 0.789 0.111 0.763 15 23.4%
MuDetect 29 25 64 48 0 0 36 8 44 2 2 4 0.917 0.127 0.905 40 62.5%
Tikanga 29 25 64 23 0 0 11 9 20 1 2 3 0.87 0.191 0.839 13 20.3%
Total 29 25 64 176 0 0 102 52 154 13 9 22 0.875 0.175 0.848 114 35.6%

Experiment 2

:
Agreements Disagreements Kappa
Detector Projects Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Precision
DMMC 5 81 0 0 6 68 74 2 5 7 0.914 0.064 0.908 8 9.9%
GrouMiner 5 66 0 0 0 64 64 1 1 2 0.97 0 0.97 0 0%
Jadet 5 39 0 0 3 35 38 1 0 1 0.974 0.069 0.972 4 10.3%
MuDetect 5 60 0 0 11 40 51 4 5 9 0.85 0.128 0.828 13 21.7%
Tikanga 5 44 0 0 4 37 41 1 2 3 0.932 0.077 0.926 5 11.4%
Total 5 290 0 0 24 244 268 9 13 22 0.924 0.071 0.918 30 10.3%

Experiment 3

Agreements Disagreements Kappa
Detector Projects Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 29 85 32 0 0 13 17 30 1 1 2 0.938 0.217 0.92 13 15.3%
GrouMiner 29 85 7 0 0 0 7 7 0 0 0 1 0 1 0 0%
Jadet 29 85 4 0 0 3 1 4 0 0 0 1 0.188 1 3 3.5%
MuDetect 29 85 34 0 0 21 11 32 2 0 2 0.941 0.2 0.926 23 27.1%
MuDetect_170922_1_ExtractingModel 23 75 30 60 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_CrossProject 29 85 47 0 0 27 16 43 1 3 4 0.915 0.197 0.894 29 34.1%
MuDetect_RankByDistance 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByPatternSupport 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByPatternUniqueness 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByPatternViolations 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByTop2Factors 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByTop3Factors 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByTop4Factors 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByTop5Factors_All 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankByViolationSupport 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_Confidence 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_DefectIndicator 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_PV_PS 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_PV_U 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_PV_U_PS 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_PV_VS_PS 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_PV_VS_PS_D 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_RankBy_Rareness 5 26 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
Tikanga 29 85 12 0 0 10 2 12 0 0 0 1 0.139 1 10 11.8%
Total 29 85 421 315 0 74 54 128 4 4 8 0.304 0.023 0.288 78 3.8%