Statistics: Results

Experiment 1

Agreements Disagreements Kappa
Detector Projects Synthetics Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 54 26 75 47 14 0 15 20 35 3 2 5 0.745 0.139 0.704 15 20%
GrouMiner 47 25 64 46 0 0 28 12 40 5 1 6 0.87 0.161 0.845 30 46.9%
Jadet 40 26 76 25 12 0 12 3 15 2 2 4 0.6 0.064 0.573 15 19.7%
MuDetect_170425_1_AllDataNodesEqual 29 25 63 40 40 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170601_2_OnlyDirectEdgesForDensity 28 25 62 41 41 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170820_NoStaticReceiver 29 25 64 50 50 0 0 0 0 0 0 0 0 0 0 0 0%
Tikanga 32 25 64 23 0 0 11 9 20 1 2 3 0.87 0.191 0.839 13 20.3%
Total 55 27 76 272 157 0 66 44 110 11 7 18 0.404 0.04 0.379 73 13.7%

Experiment 2

:
Agreements Disagreements Kappa
Detector Projects Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Precision
DMMC 5 81 0 0 6 68 74 2 5 7 0.914 0.064 0.908 8 9.9%
GrouMiner 5 66 0 0 0 64 64 1 1 2 0.97 0 0.97 0 0%
Jadet 5 39 0 0 3 35 38 1 0 1 0.974 0.069 0.972 4 10.3%
MuDetect_161207_NoEdgesToMissingNodeWeight 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_161216_ReviseEncodingOfConditions 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170320_Baseline 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170321_FilterAlternatives 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170328_FilterAlternatives_CallsOnly 5 82 52 6 0 24 24 0 0 0 0.293 0 0.293 0 0%
MuDetect_170405_MatchSubtypesJRE 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170407_MatchSubtypes 5 82 82 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170410_1_OnlyCalls 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170410_2_NoORFilter 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170411_ExpandTarget 5 82 82 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170412_Ranking 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_NumberOfMappingsHeuristic 5 82 164 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170414_Determinism 5 82 82 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170419_1_Fixes 5 82 144 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170421_4_AtLeast2Calls 5 60 100 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170421_5_ProductWeightFunction 5 80 160 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170424_3_SingleDataNodeLabel 5 80 160 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170425_1_AllDataNodesEqual 5 80 80 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170427_2_FilterDenseForMining 5 81 138 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170501_1_FixExceptionHandling 5 80 160 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170502_1_GettersTransitiveEdges 5 80 131 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170508_1_GetterHandling 5 80 149 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170531_3_MapCorrespondingEdges 5 60 120 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170601_2_OnlyDirectEdgesForDensity 5 60 60 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170820_NoStaticReceiver 5 60 97 0 0 0 0 0 0 0 0 0 0 0 0%
Tikanga 5 44 0 0 4 37 41 1 2 3 0.932 0.077 0.926 5 11.4%
Total 5 2179 3273 6 13 228 241 5 8 13 0.111 0.001 0.11 17 0.8%

Experiment 3

Agreements Disagreements Kappa
Detector Projects Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 49 112 47 44 0 11 13 24 0 1 1 0.511 0.065 0.477 11 9.8%
GrouMiner 46 101 4 0 0 0 4 4 0 0 0 1 0 1 0 0%
Jadet 30 62 4 0 0 3 1 4 0 0 0 1 0.188 1 3 4.8%
MuDetectXPOn 24 68 34 34 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170608_1_CountExamples 1 4 2 4 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170626_XSupport 24 64 24 48 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170627_XLimitAndAllTypes 25 64 32 32 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170702_MinSupport5 26 77 16 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170702_MinSupport8 27 78 12 12 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170702_XCrossPatternsConsecutive 27 78 6 12 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170719_1_FixFilteringAndConsiderConstructorCalls 22 59 26 26 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170720_1_KeepAltViolationsAndMinSup10 23 62 20 20 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170721_1_ImproveExampleFilter 25 77 29 29 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetectXProject_170725_1_RandomSamples 26 80 39 41 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_161207_NoEdgesToMissingNodeWeight 5 25 16 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_161216_ReviseEncodingOfConditions 5 25 13 26 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170320_Baseline 5 25 13 13 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170321_FilterAlternatives 5 25 13 13 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170328_FilterAlternatives_CallsOnly 5 25 13 13 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170405_MatchSubtypesJRE 5 25 13 26 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170407_MatchSubtypes 5 25 13 13 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170410_1_OnlyCalls 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170410_2_NoORFilter 5 25 14 19 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170411_ExpandTarget 5 25 14 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170412_Ranking 5 25 14 19 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_1_RemoveThisFix 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_2_UnusedDataNodeFix 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_3_RemoveNonDeterminism 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_4_Concat_XOR 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_4_RepeatedCalls 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_5_AllOfHoansFixes 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_5_AllOfHoansFixes_2 5 25 13 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170413_NumberOfMappingsHeuristic 5 25 14 19 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170414_Determinism 5 25 13 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170419_1_Fixes 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170421_1_Qualifiers 5 25 13 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170421_2_OverlapWithoutEdgesToMissingNodes 5 25 13 17 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170421_3_FixTypeResolution 5 25 12 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170421_4_AtLeast2Calls 5 25 12 14 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170421_5_ProductWeightFunction 5 25 12 13 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170424_1_HoansFixes 5 25 12 24 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170424_2_MoreFixes 5 25 12 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170424_3_SingleDataNodeLabel 5 25 12 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170425_1_AllDataNodesEqual 5 25 12 12 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170427_2_FilterDenseForMining 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170428_1_ConsiderThrowAnAction 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170428_2_ConsiderSyncAnAction 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170501_1_FixExceptionHandling 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170502_1_GettersTransitiveEdges 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170508_1_GetterHandling 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170508_2_RelaxInclusionCondition 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170508_3_ForeachToIterator 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170508_4_NoTransitiveDefEdges 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170509_OnlyCallAsSingleNode 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170518_ConstantsLikeLiterals 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170531_1_MissingDefPrefixNoViolation 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170531_2_PrioritizeEdgesByType 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170531_3_MapCorrespondingEdges 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170601_1_ExplicitNullCheck 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170601_2_OnlyDirectEdgesForDensity 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_DefPrefixWithConditionEdges 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_FixOrderPrio 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_MixedUpDefPrefix 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_NoTransitiveEdges 5 25 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_WithTransitiveEdges 5 25 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_debug1_OnlyDirectEdgesDensity 5 25 15 15 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_debug2_FixDirectEdgesOfIncDataNodes 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_debug3_FixDefPrefix 5 25 15 16 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_debug4_ExtractAltViolationsFilter 5 25 15 29 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_debug5_NoTransitiveEdges 5 25 15 29 0 0 0 0 0 0 0 0 0 0 0 0%
MuDetect_170809_debug6_NoReceiverStaticMethods 5 25 15 30 0 0 0 0 0 0 0 0 0 0 0 0%
Tikanga 30 62 9 0 0 7 2 9 0 0 0 1 0.173 1 7 11.3%
Total 53 153 1099 1287 0 21 20 41 0 1 1 0.037 0 0.037 21 0.2%