[PDF] Whats Wrong With My Benchmark Results? Studying Bad Practices

View - Download Whats Wrong With My Benchmark Results? Studying Bad Practices

Previous PDF

Next PDF

1

What'sWrongWithMyBenchmar kResults?

StudyingBadPr acticesinJMH Benchmarks

DiegoCosta,Cor-P aulBezemer ,PhilippLeitner,Artur Andrzejak

Abstract - Microbenchmarkingframewor ks,suchasJava'sMicrobenchmarkHarness(JMH),allowde velopersto writefine-g rained

performancetestsuitesatthemethodor statementlev el.How ev er,due tothecomple xitiesof theJa vaVirtualMachine,dev elopers

oftenstruggle withwritingexpressiv eJMHbenchmar kswhichaccuratelyrepresentthe performanceofsuchmethodsorstatements .In

thispaper, weempiricallystudybad practicesofJMHbenchmarks .Wepresenta toolthat lever agesstaticanalysistoidentify5 bad

JMHpractices .Ourempiricalstudyof123opensource Jav a-basedsystemsshowsthat eachofth ese5bad practicesare prevalent in

opensourcesoftw are.Fur ther,weconductseveral experimentstoquantifytheimpactof eachbadpracticeinm ultiplecasestudies ,

andfindthat badpractices oftensignificantlyimpact thebenchmark results.Tov alidateoure xperimentalresults ,we constructedse ven

patchesthatfix theidentifiedbad practicesf orsixof thestudied opensourceproject s,ofwhichsixw eremergedintothemainbr anch

oftheproject. Inthispaper ,we showthat developers strugglewithaccurate Javamicrobenchmar king,andprovidesev eral

recommendationstode velopersof microbenchmarkingframeworksonho wto improvefutureversion softheirframework.

IndexTerms - Performancetesting,microbenchmarking,JMH,badpractices, staticanalysis

1INTRODUCTION

Performancecharacteristics,such asresponse time,la- tencyorr esourceusage, aresomeofthemostimportant non-functionalproperties ofcriticalsoftwaresystems. In manysystemsit isessential thatperformanceissues are avoided,as theycanhave adevastatingimpact onthe perceivedqualityofthe software.Forexample,aone-second slowdowninthe checkoutprocesses wouldcostAmazon an estimated$1.6billion peryear[ 22].

Toavoidsuchproblems, theperformance ofasoftwar e

systemmustbe thoroughlytested. Similartofunctional test- ing(e.g.,thr oughintegrationand unittests),performance testingshouldbe doneatdif ferentgranularities. Theulti- mategoalof performancetestingis toachieve agoodend- to-endperformance.Even thoughtheimpact oftheperfor - manceofeach componentonthe end-to-endperformance ofasystem isnotnecessarily linear,obvious performance issuesatthe component-levelare likelytoripp lethrough tothesystem-level. Hence,wemust firstensur ethatthe performanceof eachcomponentof thesystemis adequate. Microbenchmarking[1],[ 2]is anapproach forprecise per- formanceevaluation ofani solatedsegmentof codeatthe method,loop,or evenstatementlevel. Severalframeworks havebeen proposedtofacilitatespecifyingand executing microbenchmarks.Forexample,theJava Microbenchmark

Harness(JMH)[ 1]isa popularframeworkfor languages

thattar gettheJavaVirtualMachine (JVM),suchas Javaand Scala.JMHmakes itpossibleto obtainastatisti calestimate •DiegoCostaand ArturAndrzejak arewith theInstituteof ComputerScience,Heidelber gUniversity, Germany.E-mail: •Cor-PaulBezemerheads theAnalyticsof Software, GamesandRepository Data(ASGAARD)lab attheUniversity ofAlberta,Canada. E-mail: bezemer@ualberta.ca •PhilippLeitneris withthe SoftwareEngineerin gDivisionat Chalmersandthe UniversityofGothenbur g,Sweden.E-mail: philipp.leitner@chalmers.se ofthe executiontimebyprovidinga scaffoldthat runs thetargeted segmentofcodealar genumberof times.In addition,JMHof fersmechanismsto preventasubsetofJVM optimizations,whichmay leadtomisleading performance results. Unfortunately,despitetheexistenceof microbenchmark- ingframeworks,w ritingmicrobenchmarks thatyieldcorrect insightsonthe performanceofthe benchmarkedtaskturns outtobe challenging.Priorwork establishedthatther e existseveralpitfalls toavoid[ 20]andthat manydevelopers strugglewithwritingappropriate microbenchmarks[ 37].In addition,the executionresults ofreal-life microbenchmarks oflarge opensourceprojectscan varysignificantlyacr oss repeatedexecutionsinidenticalenvir onments[ 30]. Inthispaper ,weempirically studybadpracticesofwrit- ingmicrobenchmarks thatcanleadtomisleading bench- markresults. Inparticular,westudy theoccurrence of5bad practices,whichwer eextractedfr omtheJMHdocumenta- tion,inthe JMHmicrobenchmark suitesof123 Java-based opensource projects.Inaddition,wequantify theimpact oftheidentified badpracticeson thebenchmark resultsof sixopen sourcepr ojects.Ourstudyaddressesthe following researchquestions:

RQ1:Howfr equentlydobadJMHpracticesoccur inreal-

lifeopensour cesoftware? Toassistdevelopers withavoidingbad practices,weimplemented a plugin(SpotJMHBugs[16])forthe SpotBugs staticanalysis tool.SpotJMHBugsisableto automaticallyidentifythe badJMHpractices thatare discussedinourstudy. Usingthetool, wefoundthat 35of123 studiedprojects had atleastone instanceof abadJMH practicein theirbenchmarksuites. In12of theseprojects, thereweremore than10instancesofbad JMH practices.Themost frequentlyoccurring bad 2

JMHpracticeis usingaccumulationto consume

computationina loop,whichcould yieldunre- liablebenchmarkr esults.

RQ2:Whatisthe impactof theidentifiedb adJMHprac-

ticesonthe benchmarkresults?

Wemanuallyfixed105benchmarks across6

projectsandmeasuredthe impactofthe bad practicesbycomparing theresults beforeand afterthefix. In73of the105fixed benchmarks, thebadJMH practicesignificantlyimpacted the benchmarkresults.

Toevaluatewhetherdevelopersin practicecare about

thesebadpractices, wesubmitted7 pullrequest scontaining fixesfor57 benchmarksfrom 6projects. Sixof thesepull requestswereacceptedand mergedintothemainpr oject repository,whichdemonstratesthesignificanceofour work. Inparticular, ourstudyshowsthatdespite theJMHdocu- mentationwarningexplicitly aboutallfive discussedbad JMHpractices,they arestill prevalentin opensourcesoft- ware.Evenworse,weshow thatthese badpracticeshave asigni ficantimpactonthemicrobenchmarks, whichoften leadsto unreliablebenchmark results.

2BACKGROUND

Inthis section,wepresenttheterminology thatisused

throughoutthispaper, andintr oducethesetofstudiedbad

JMHpractices.

2.1JMHMicr obenchmarks

Performancetesting[ 43]isused toexperimentally assess oneormor enon-functional qualityattributesofsoftware systems.Inthis paper, wefocuson onespecificperformance testingapproach, namelysoftwaremicrobenchmarking.In contrasttostr essorload tests,whichtesttheend-to-end performanceofa system,microbenchmarks are relatively short-runningandaimatmeasuring thefine-gr ainedper- formanceof specificunits ofprogram code.Forinstance, amicrobenchmark maymeasuremethod-levelexecution timesofa class,theperformance ofaspecific datastruc- ture,ortheimplementationof analgorithm.Consequently , microbenchmarksaretypically notusedtoevaluatesystem- levelservicelevel agreements, butratherto ensurethe performanceofcritical low-levelcodecomponents orto comparedifferentimplementation alternatives.

ForJava, theJavaMicr obenchmarkHarness(JMH) [1]is

commonlyused formicrobenchmarking [39].JMHis atool developedunder theOpenJDKumbr ellathatallows usersto specifybenchmarksthr oughJava annotations,usingasyn- taxthatis similartothe well-knownJUnitframework [40].

Everypublicmethod thatisannotated with@Benchmark

isexecutedas partofthe microbenchmark suite.Listing1 showsan examplebenchmarkfr omtheRxJavaproject, wheretheexecutiontimeand throughputof alatched observerare measured.

InFigure 1weillustratethe executionflow thatJMH

usestoevaluate microbenchmarksin fourmajorsteps. (1)Initially, anoptionalbenchmarkfixture isinvoked.The publicclassComputationSchedulerPerf{ @State(Scope.Thread) publicstaticclassInput extendsInputWithIncrementingInteger{ @Param({"100","1000"}) publicintsize; @Benchmark publicvoidobserveOn(Inputinput){

LatchedObservero =

input.newLatchedObserver(); input.observable.observeOn(

Schedulers.computation()

).subscribe(o); o.latch.await();

Listing1:JMH examplefrom theRxJavaproject.

Fork (JVM A)

Invocations

Fork (JVM B)

observeOn() observeOn() observeOn()

Benchmark

Fixture

setup() 1 Time Trial

Benchmark

Iterations

3 4

Warmup

Iterations

2 Trial

Benchmark

Iterations

Warmup

Iterations

Fig.1:JMH executionflowfor amicrobenchmark config- uredwith2forks,6 benchmarkwarmupsand 8benchmark iterations.Inthis example,the fixture methodsareconfig- uredtobeexecuted beforeand aftereachtrial. benchmarkfixture isthecodewhichinitializesthe bench- markenvironment (e.g.,byfillingadatastr ucturewith test data).(2)Afterwar ds,a definednumberofwarmupiterations areexecuted.Theseare identicalto benchmarkiterations (seebelow), buttheirr esultsare discarded.W armupitera- tionsare intendedtobringtheJVMinto asteadystate (e.g., executeallapplicable just-in-timecompilations). (3)Inthe actualbenchmark phase,adefined numberofbenchmarkit- erationsisexecuted.Each iterationtakesa definedamountof time(typically1s),duringwhich theframework repeatedly callsthemethod annotatedwith@Benchmark(asingleinvo- cationinJMHparlor) andrecor dsallconfigur edperformance counters(e.g.,throughput, executiontime,latency).(4)One entirerunofsteps 1-3(executingthebenchmarkfixtur e, zerotomanywarmupiterations, onetoman ybenchmark iterations)iscalled atrial.Bydefau lt,JMHexecutes each trialina separateVM, anddevelopersmay specifyhow manyforkseachbenchmarkwill sequentiallyexecute(e.g., wepresent anexamplewithtwoforksin Figure1).

Afterthelast iteration,JMH reportsa summaryofthe

resultstotheuser and/orsaves theresults intoanXML orJSONfile. Parameters(e.g.,intsizeinListing1)can beusedto easilydefinedif ferent benchmarkinstances ofthe samemethod.In theexamplein Listing1,JMH willeffec- tivelyexecutethe observeOn(Input)benchmarktwice; oncewithsize=100 andoncewith size=1000 . 3 TABLE1:BadJMHpractices collectedfrom theJMH documentation.

IDBadJMH PracticeDescription UndesiredEffect

RETUNotusing areturned computationDead codeelimination LOOPUsing accumulationtoconsume valuesinsidea loopLoopoptimization FINALUsinga finalprimitiveforbenchmark inputConstantfolding INVORunning fixturemethods foreachbenchmarkmethodinvocationJMH overhead FORKConfiguringbenchmarks withzer oforksPr ofile-guidedoptimization

2.2BadJMH Practicesin BenchmarkCreation

WhileJMHof fersaninfrastr ucturethatgeneratesthe boiler- platecodefor amicrobenchmark fromuser -specifiedcode andannotations,the responsibilityof creatinga reliableand correctbenchmarkremainswith thedeveloper .

Unfortunately,creatingmicrobenchmarks whichaccu-

ratelyrepr esenttheperformanceofthebenchmarkedcode isdiffi cult.Thereexistseveralcodingpitfallsand badJMH practicesthatcan affectthe reliabilityand correctnessof amicrobenchmark, asillustratedbythe 38samplebench- marks[35]inthe JMHdocumentation [1].Inthe remainder ofthissection, wediscuss themostimportant badJMH practicesdescribedin theJMHdocumentation (seeTable 1 foranoverview). Allcodeexamples inthissection were takenfrom theJMHsamples.

BadJMHpractice 1:Notusing aresult thatisreturned

bya methodinthe benchmark(RETU)

Description:Abenchmarktypically callsoneor more

methodsfrom themainapplicationcode.Ifsuch amethod returnsaresultthat isnotused inthebenchmark,theJVM mayconsiderpart ofthe calledmethodas "deadcode"and eliminatethatpart.

Symptoms:Becausethecall tothebenchmarked method

waseliminated, thecodewill appearfasterthan inactual usage.Listing2showsanexample oftheRETU badJMH practiceand twopossiblesolutions. InmeasureWrong(),

Math.log(x1)isredundant andmaybeeliminatedby the

JVM. Solution:Everyobjectthat isreturned byamethod called directlyfromthebenchmark shouldbeusedinthebench- markmethod.In measureRight1(),Math.log(x1)is usedasa return ofthebenchmark method,andthere- fore,noteliminated.Alternatively, theJMHinfrastr ucture offersaBlackholeobjectwhichcan beusedto pre- ventdead-codeelimination byconsuming theresult. In measureRight2(),Math.log(x1)isconsumedby a

Blackholeobject.

BadJMHpractice 2:Using accumulationtoconsume

computationinsidea loop(LOOP) Description:Developersoftenhave todesignbenchm arks thatmeasure amethodcallwithina loop.Ifthe method callre turnsanumericvariable,itisintuitiveto accumu- latether eturnedobjectsas awayofavoidingdead-code elimination.However ,thisleadstoanotherset ofJVM optimizationsthatoptimizes thecodebeyond whatwould beexpectedin realusage.

Symptoms:Thecodeappears fasterthanin actualusage,

astheloop canbeextensively optimizedbythe JVM.List- ing3showsanexample oftheLOOP badJMHpractice. The

1.https://bit.ly/2uyb79b

privatedoublex1; privatedoublex2; @Benchmark publicdoublemeasureWrong(){

Math.log(x1);

returnMath.log(x2); @Benchmark publicdoublemeasureRight1(){ returnMath.log(x1)+ Math.log(x2); @Benchmark publicdoublemeasureRight2(Blackholebh){ bh.consume(Math.log(x1)); bh.consume(Math.log(x2));

Listing2: Exampleofthe RETUbadJMH practiceand two

possiblesolutions.quotesdbs_dbs21.pdfusesText_27

[PDF] java questions asked in interview

[PDF] java ring ppt presentation download

[PDF] java se 11 books

[PDF] java se 11 pdf download

[PDF] java send http get request example

[PDF] java series program examples

[PDF] java sort(comparator)

[PDF] java spring @componentscan example

[PDF] java spring boot componentscan

[PDF] java spring componentscan annotation

[PDF] java standalone application with database example

[PDF] java statements pdf

[PDF] java structure

[PDF] java style guide

[PDF] java subtype vs subclass