Statistics: Results

Experiment 1

Agreements Disagreements Kappa
Detector Projects Synthetics Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 26 3 122 62 0 0 13 49 62 0 0 0 1 0.166 1 13 10.7%
GrouMiner 26 3 122 88 0 0 56 29 85 1 2 3 0.966 0.21 0.957 57 46.7%
Jadet 26 3 122 23 0 0 14 9 23 0 0 0 1 0.238 1 14 11.5%
MuDetect 55 4 171 149 0 0 119 25 144 1 4 5 0.966 0.134 0.961 123 71.9%
Tikanga 26 3 122 15 0 0 3 10 13 0 2 2 0.867 0.133 0.846 3 2.5%
Total 55 4 171 337 0 0 205 122 327 2 8 10 0.97 0.22 0.962 210 24.6%

Experiment 2

:
Agreements Disagreements Kappa
Detector Projects Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Precision
DMMC 5 80 0 0 4 76 80 0 0 0 1 0.048 1 4 5%
GrouMiner 5 90 0 1 1 86 87 2 0 2 0.967 0.011 0.966 3 3.3%
Jadet 5 52 0 0 2 47 49 2 1 3 0.942 0.036 0.94 4 7.7%
MuDetect 10 146 0 0 29 112 141 2 3 5 0.966 0.153 0.96 32 21.9%
MuDetectXP 10 91 0 3 30 53 83 0 5 5 0.912 0.192 0.891 30 33%
Tikanga 5 41 0 0 2 33 35 0 6 6 0.854 0.039 0.848 2 4.9%
Total 10 500 0 4 68 407 475 6 15 21 0.95 0.111 0.944 75 15%

Experiment 3

Agreements Disagreements Kappa
Detector Projects Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 30 177 42 0 0 13 27 40 0 2 2 0.952 0.199 0.941 13 7.3%
GrouMiner 29 173 14 0 0 7 7 14 0 0 0 1 0.25 1 7 4%
Jadet 29 173 12 0 0 11 0 11 1 0 1 0.917 0 0.917 12 6.9%
MuDetect 56 224 72 0 0 46 26 72 0 0 0 1 0.231 1 46 20.5%
MuDetectXP 51 215 151 0 0 92 54 146 2 3 5 0.967 0.218 0.958 95 44.2%
Tikanga 30 177 11 0 0 8 1 9 2 0 2 0.818 0.066 0.805 10 5.6%
Total 56 224 302 0 0 177 115 292 5 5 10 0.967 0.223 0.957 183 13.6%