[PDF] Mitigating the Halo Effect: Managing the Wow Factor in Music

Journal of Research in Music Performance Spring 2016 MitigatingtheHaloEffect:ManagingtheWowFactorinMusicPerformanceAssessmentNathan Zoanetti Helen Champion Victorian Curriculum and Assessment Authority, Victoria, Australia ABSTRACT Thisarticledescribesaprojectundertakento assistassess ors withrecognizingandamelioratingtheinfluenceofthehaloeffectwhenmultiplecriteriaareappliedinarubrictoscoremusicperformanceassessments.Thehaloeffectoccurswhenimpressionsofthequalityofaperformanceeitheroverall,oragainstoneorasubsetofthecriteria,undulyinfluencethescoresassignedtosubsequentcriteria.TheassessmentcontextdescribedinthisarticleissetwithintheVictorianCertificateofEducation(VCE),andspecificallytheVCEMusicInvestigationstudy.ThisVCEstudyprovidesaflexiblepathwayforstudentstoperform,composeandresearchdiversetraditional,contemporary,classicalandpopularmusicstylesusinginstrumentsrangingfromvoice,tabla,recorderandudtoorchestralandwind-bandinstruments,instrumentstypicallyfoundinrockbandsanddigitalinstrumentsinsoloandensembleperformances.Thisdiversitymustbeaddressedinpracticethroughvalidassessmenttasksandthereliableapplicationofassessmentcriteria.Thesetasksandcriteriamustbecoupledwithappropriateassessortraining,guidanceanddocumentation.Tosuppor tthisrequirement,experiencedassessorsreviewe drecordedstudentperformanceswhilstannotatingtheperformancefeaturestheyreferencedwhenallocatingscoresagainsteachoftenrubriccriteria.Alsorecordedwerethephrasesthatassessorsusedtodescribewell-documentedsourcesofcognitivebiasinperformanceassessmentcontexts,includingthehaloeffect.Thesesourcesofinformationweredistilledinanefforttoprovideenhancedtrainingandscoringguidanceandmaterialstosupportassessmentvalidityandreliability.Keywords:Haloeffect,ra tereffects,performance assessment,rater reliability,ratercognition,ratertraining

_________________________________________________________________________________________________________Zoanetti&ChampionMitigatingtheHaloEffect:ManagingtheWowFactor37INTRODUCTIONItiswelldocumentedthathumanjudgmentsofperformancequalitycanbesusceptibletovarioussourcesofsubjectivity,biasandconstruct-irrelevantvariance(Thorndike,1920;Rudner,1992).Intheassessmentandeducationalmeasurementliterature,thesemattersareoftendescribedundertheheadingsofratereffects(Wolfe,2004)orrateraccuracy(Engelhard,1996).AsBairdetal.(2013)summarized,ratereffectscanbegroupedintointer-ratereffectsandintra-ratereffects.Inter-ratereffectsper taintosystematicdifferencesbetweentheleniencyorseverityofraters,orsystematicdifferencesinthewaysthatraterstendtousetheavailablescorerange.Intra-ratereffectsoccurwherearaterassignsoneormorescores,whicharetosomeextentinfluencedbyfactorsunrelatedtotheassessmentconstruct.Suchfactorscouldincludethepersonalpreferencesofanassessorortheonsetof assessorfatigue followin galargenumberofassessme nts.Underthesecircumstances,theassignedscoresaregenerallyinconsistentwithinitialimpressionsoftherater'soverallleniencyorseverityandcanadverselyimpactassessmentvalidityandreliability.Thisarticlefocusesonaddressingonedistinctinter-ratereffect:thehaloeffect.Thehaloeffectoc curswhenimpressions ofthequalityofa performanceeitheroverall, oragainstoneorasubsetofthecriteria,undulyinfluencethescoresassignedtosubsequentcriteria(Eckes,2009).Thiseffecttypicallymanifestsitselfintheformofoverlysimilarscoresacrossconceptuallydistinctcriteria;thelevelofsimilarityoruniformityacrossthecriterionscoreswouldbegre aterthanthatarisin gfromanunbiased apprais aloftheperformance.Anothercommonlyciteddefinitionofhaloeffectis"...arater'stendencytoassignrateessimilarratingsonconceptuallydistincttraits"(Myford&Wolfe,2004,p.209).Inother words,judgmentsaboutthequali tyofdiffe rentaspe ctsordimensi onsofa performanceorproductarelessindependentofeachotherthanwouldbethecaseintheabsenceofthehaloeffect.Ifthecriteriaeachreflectdistinctandimportantskills,thenthehaloeffectcouldresultincriterionscores,whichdonotbestreflectthestrengthsandweaknessesofagivenperformance .Thisisnotwith standing thatsomecriteriawillbeintrinsicallymoreinter-relatedthanothersa ndthereforecouldbeexpect edtoatt ractrelativelymoresimilarscoresevenintheabsenceofthehaloeffect(Latimer,Bergee&Cohen,2010).Inaseniorsecondaryexaminationcontext,whereaspecificcurriculumanditsembeddedoutcomesunderpintheassessmentconstructandtherespectivecriteriaintherubric,beingabletorecoverthisprofileofstrengthsandweaknessesisarguablymorevalidandmoredia gnostically usefulthan allowingittobemaskedbythe haloeffect.Whethertheseconcernsareasimportantinotherperformancecontexts,particularlythosewherepresetscoringcriteriaarenotapplied(seeSadler,2009),isleftasanopenquestion.Studiesintothereliabilityofjudgmentsofmusicperformancequalityhavebeenreportedonpreviously(e.g.Garman,Boyle&DeCarbo,1991;Bergee,2003;Latimeretal.,2010).Somesuchstudieshavefocusedoninter-raterreliabilitymeasureswhenrubricshavebeen

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 38used(Ciorba&Smith,2009).Othersuchstudieshavefocusedondifferencesbetweenthelevelsofreliabilityofdifferentratingorscoringforms,forexamplecomparingrubricstopreviouslymorecommonplaceadjudicationforms(Norris&Borst,2007).Yetothersuchstudieshaveappl iedparticularmeasurement frameworksandcorresponding statisticalmodelstoanalyzeaspectsofraterreliability(Wesolowski,WindandEngelhard,2015).Thecitedstudieswerepredominantlyevaluativeinnature,withafocusonthedetectionandquantificationofvariousreliabilityand modelfitmeasures,leadingtodiscu ssionsofimplicationsforimprovingsubsequentassessmentadministrations.Inthecurrentstudy,theinitiativetoexploreandpreemptivelyaddressthehaloeffectwasinitiallymotivatedbytwomainsourcesofevidence.Thefirstwasdatarevealingthattherubriccriteria,manyofwhichcouldberegardedasconceptuallydistinct(refertoAppendixA),attractedsimilarscores.ThePearsoncorrelationvaluesbetweenthescoresawardedfordifferentpairsofcriteria(oftenreferredt oasinter-itemcorrelat ions)allexceeded0.8andsocouldbeconsideredhigh.Whilemoreformalmeasurementframeworksandstatisticalmodelscouldhavebeenappliedtodetectthepossiblepresenceofthehaloeffectandotherratereffects(e.g.Myford&Wolfe,2004),thiswasnotwithinthescopeofthepresentstudy.Beyondquantitativeevidence,thesecondsourceofevidencearoundthepossibleimpactofthehaloeffectarosethroughdiscussionswithassessors.Thesediscussionsrevealedacollectiveawarenessthatawardingfull marksforallcr iteriawhentheoverallimp ressionofaperformance,oraspectsofit,wereoverwhelminglypositivewasdifficulttoavoid.Thesesourcesofevidence,wh ilstbyn omeansdefinitive,whentakenincomb inationweredeemedsufficienttohypothesizethatthehaloeffectcouldbeimpactingscoringprocessesinthisassessmentcontext.Tominimizethehaloeffectinoperationalassessmentprograms,MyfordandWolfe(2004)notedtheimportantroleofassessortrainingasapreventativestrategy.Theyrecommendeffortstoraiseawarenessofthehaloeffectanditsinfluencesothatassessors canworktoconsciouslyamelio rateit.In effecttheyar eadvoca tingthedevelopmentofeffectivemetacognition,acapacitydefinedbyKuhnandDean(2004)as"awarenessandmanagementofone'sownthought"(p.270).Itisinthisspiritthatthepresentprojectisbeingundertaken.RESEARCHQUESTIONSTheworkshopsoughttoprovideinsightsaboutthefollowingresearchquestions:1. Howdomusicperformanceassessorsdescribethephenomenonofthehaloeffect?2. Howdomusicperformanceassessorsdescribepotentialstrategiesformitigatingthehaloeffect?3. Whataretheevidentiaryfeaturesofmusicperformancesthatassessorsrefertowhenassigningscorestoeachofthecriteriaintheprescribedassessmentrubric,andhowdotheydescribethese?

_________________________________________________________________________________________________________Zoanetti&ChampionMitigatingtheHaloEffect:ManagingtheWowFactor394. Whatarethesalientdifferencesbetweenperformancequalitiesthatdistinguishonecriterionscorefromtheadjace ntcriterionscoresinthe prescribedassessmentrubric,andhowdoassessorsdescribethese?Thisarticlefo cusesprimarilyonthef irsttwoquestionsbut presentssomefindingsrelevanttootherquestionsw heretheyrelat etotheb roaderissueofrat ereffects.Ultimately,bylearninghowbesttoraiseawarenessofhaloeffectswithassessorssothattheycouldconsciouslyavoidit,thisresearchwasintendedtoleadtoanincreaseinthevalidityofstudents'scor es,andw ouldensurethatthemostvalidrel ativerankingofstudentachievementcouldalsobeobtained.METHODAgroupofeightexperiencedassessorswasinvitedtoaworkshoptoreviewaudiorecordingsofstudentperformanceswhi lstannotatingtheperformanceevid encetheyreferencedwhenallocatingscoresagainsteachcriterion.Theseassessorswerespecialistsinarangeofinstrumentsandmusicstylesandeachhadbetweenthreeand20yearsofexperienceassessingmusicperformanceexaminations.Thetaskforthedaywastolistento,scoreanddiscussrecordingsofasampleofthepreviousyear'sexamination performances.Theworksho pemployedtwoqualitative researchmethodologiesinsequence.ThefirstmethodwassimilartoretrospectiveverbalprotocolanalysisasdescribedbyEricssonandSimon(1993).Thismethodprovidedanopportunitytogatherinformationaboutratercognition,byaskingindividualassessorstorecounttheirthinkinga stheylistenedt oandscoredeachrecorded performa nce.Inadvanceoflisteningtoeachrecordedperformance,assessorswereaskedtoannotatetheirthinkingwhilstapplyingthescoringrubrictosupportthisapproach.Thiswasintendedtoaidtheirrecall,notingthatthevalidityofretrospectiveverbalprotocolsdiminishesastheelapsedtimesincethecognitivetaskofinterestincreases.Thesecondmethodinvolvedafocusgroupmethodo logy(Kruegerand Casey,2009).Thismethodextend edtheretrospectiveverbalreportsbyinvitinggroupdiscussionaboutthequalitativefeaturesofeachperformance,andhowthesefeaturesinfluencedoverallimpressionsofperformancequalityandhowtheycoul db elinkedto differentlevelsof eachcriterionandto theexpectationsoftheseniorsecondarycontextmorebroadly.Thework shopwasledbyt hechiefassessor,aswouldbe thecasef orasse ssortraining.VCAArepresentatives,includingtheauthors,observedtheworkshopandtooknotes.Anaudiorecordingwasmadetoenablefollowupanalysisoftheverbalreportsandgroupdiscussion.Thisprocessaimedtoidentifyandrecordphrasesandstatementsthatdescribedcognitionandmetacognitionrelevanttothehaloeffect,andrelevanttootheraspectsofdifferentiatingbetweenadjacentscorecategoriesforspecificcriteriawithintherubric.

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 40Thegroupdis cussedthisevidence andnotedsome commonlanguagetheyconsideredsuitableforadoptionbyotherassessors,withtheintentionofprovidingabasisforcriterion -referencedratherthansolelyimp ressionisticdiscuss ionsofperforma ncequality.Assessorswereaskedtoreflectnotonlyontheevidencetheyreferenced,butalsoonthejudgmentprocessestheyapplied.Throughthisprocessanumberofphraseswerederivedfordescrib ingthe occurrenceof,orconscious avoidanceof,t hehaloeffect. Importantly,these phraseswereint heeverydaylanguageof musicians,not language couchedintheoreticalt erminolog yfromtheassessmentoreducational measurementlexicon.Thewordsandphraseswerethereforeconsideredsuitableforbeingrecastasuser-friendlyguidanceforthebroaderassessorgroup.Asap relimin arygoal,thisapproachaimstoi ncreasethequali tyofdiscussi onamongstassessorsthroughafocusonevidenceaboutsalientperformancefeaturesandperformancedifferencesagainsteachprescribedcriterion,ratherthanafocusonagreeingingeneraltermsabouttheoverallqualityofaperformance.Evenifassessorsfirstcometoashar edconclusionabout theoverallqualityofaperforma nce,thelanguage beingdevelopedhereisintendedtoassistassessorsinunpackingthefeaturesoftheperformance,referencedagainsttherelevantcriteria.Tofacilitatethis,theapproachbeingundertakenisconcernedwithco-constructinganinventoryofcommonlanguagetosupportdiscussionsofperformancequality,focusingoneachcriterioninturn.Theapproachspecificallytargetsthehaloeffectbyaddingalayerofmetacognitiveguidancethatassessorswillbeaskedtorefertowhenrecordingscoresforeachperformance.Thisguidanceisfield-testedduringeachexaminationperiodanditisenvisagedthatitwillbeintegratedintoassessortrainingandsupportmaterialsforotherlearningareasifthereisevidenceofitseffectiveness.Tosupportimplementationof thisadditional,writtenguidance,a briefpresentationandexplicationofthehaloeffectis nowdel iveredaspart oftheregularassessort rainingmeetings.CURRICULUMANDASSESSMENTCONTEXTTheVCEisthemainseniorsecondaryqualificationinVictoria,Australia.Eachstudyinthecertificateisassessedbyamixofcourseworkandexternalexaminations.VCEMusicInvestigationUnits3-4wereintroducedin2011toprovideaflexibleoptionforstudentsinterestedinbothperformanceandcomposition.Overthefouryearsthatthestudyhasbeenoffered,studentshavepresentedawide-rangeofperformanceprogramsacrosstraditional,contemporary,classicalandpopularmusicstylesandusingtraditional,digital,orchestral,rockandotherinstruments.Thisdiversityposesissuesofequityandrigorthatmustbeaddressedinpractice;thereisaneedforassessmenttasksandcriteriawhichsupportrel iablescoringwhilstca teringfortherangeofin struments,stylesandabilities.Further,thesetasks andcriteriamustbecouple dwithappropriat eassessortraininganddocumentation. Recentassessorwork shopsandtrai ningmeetingsare

_________________________________________________________________________________________________________Zoanetti&ChampionMitigatingtheHaloEffect:ManagingtheWowFactor41providinginsightsintohowimprovementsmightbemadetofurtherstrengthentheequityandrigoroftheassessmentprogram.VCEMusicIn vestigationisa performance/portfoliostylestudy,typical lyundertakeninthefinalyearofsecondaryschooling.Thestudychallengesstudentstoexploreanareaofpers onalmusic alinterestan ddevelop theirabilitiesasperfor mersthroughresearchandco mposition,improvisationand /orarrangeme nt.Studentsmaychooseanyinstrumentorinstrumentsfortheirperformanceandelecttoworkasasoloistorasamemberofgroup.ThecurriculumisprovidedintheVCEMusicstudydesign,aframeworkfromwhichschoolsdesignlearningprograms.Detailsofthecurriculumandassessmentspecificationsareavailableatwww.vcaa.vic.edu.auThedeviceofaFocusAreaisusedinthecurriculumtoorganizethestudent'sworkacrosstheschoolyea r.TheFocu sAreamightbeabout thepracticeofa performeror performers,amusicstyleorgenre,ortheuseofaninstrumentinaparticularcontext.Predictably,MusicInvestigationstudentschoosediverseareasofmusicalinterestasthebasisoftheirstudy.Forqualityassurance,VCAAreviewsandapprovesFocusAreasearlyintheschoolyear.Table1showsacross-sectionoftheFocusAreaschosenbystudents.TABLE1:samplefocusareasCLARINETThemanipulationofvariousclarinettechniquesinromanticandneo-classicalclarinetpieces.RECORDERHowperformersandcomposerscreateinterestinthemelodiclineinarangeofstylesthroughtheuseofperformancetechniquesandconventions,suchasdiminution,ornamentation,freeembellishment,andbothneo-classicalandavant-gardevariationwriting.VOICEAnimatedHollywoodmusicals:artistryandinterpretation.VIOLINHowchangesintheconstructionoftheviolinsincethebaroqueperiodhaveallowedfortheemergenceofnewtechniques.ELECTRICBASSThedevelopmentofpitch-basedrhythmicsupportseenthroughbassincludingitsevolutionintoamelodicinstrument.TABLADefiningNorthIndianTablarepertoireinamodernWesterncontext.PerformanceexaminationstructureTheperformanceexaminationforMusicInvestigationcontributes50%ofthetotalstudyscore.TheexaminationrequiresstudentstopresentaprogramofatleastfourworksthatdemonstratestheirunderstandingoftheFocusAreatheyhavestudiedthroughouttheyear.Themaximumtimeallowedforasoloperformanceis25minutes.Forstudentswhoareassessedasamemberofagroupthetimevariesfrom25minuteswhereonlyonestudentisassessedinthegroup,to40minuteswherefivetosixstudentsareassessed.Agroupmayhavebetweentwoandeightperformers.Soloperformancesareassessedbytwoassessor s,oneofwhomisaninstrumenta nd/orgenrespecialistandtheother ,a coordinatingassessorwhoworksacrossanumberofinstrumentsand/orgenres.Group

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 42performancesareassessedbytwoormoreassessorsdependingonthenumberofstudentstobeassessedsothattheperformanceofeachassessedperformercanbeconsideredbytwoassessors.Aswiththepanelforsoloperformanceassessments,eachpanelincludesaninstrumentspecialist/sandacoordinatingassessor.PerformanceexaminationassessmentcriteriaTheassessmen tcriteriafortheMusicInvestigatione nd-of-yearperformance examinationarederivedfromthekeyknowledgeandkeyskillsdescribedinthestudydesign.ThecriteriaarepublishedontheVCAAwebsitealongwithexplanatoryannotations.Thisstra tegysupportstransparencyandconnectio nacrossthelearningpr ocess.Onlycriteria2through11are consid eredhere.Thefirstas sessmentc riterionis relatedtocomplianceasopposedtoperfo rmancequa lityandwastherefore notrelev antforthiswork.ThesecriteriaarelistedinAppendixA.Eachcriterionisscoredoutofamaximumof10byeachassessor.Assessorsareencouragedtodiscussthefeaturesofperformanceswithrelationto thepublishedcriter ia,butthe yareexpecte dtoassignscoresindependentlyofeachother.FINDINGSLanguageusedbyassessorstodescribethehaloeffectMostofthediscussionpertainingtothehaloeffectwasdirectedtowardsthetopendofthepe rformancedi stribution.Thiswasconsistent withthesentimentthatassessorsbelievedthehaloeffectcouldemergemostreadilywhentheyweredrawninbyahighqualityperformance.One assessornotedthatwhentheirini tialimpress ionofaperformancewasthatitwas"amazi ng",ittook amoreconscio useffort torefocus onunpackingtheevidenceagainstthecriteria.Intheirwords:"That'smychallenge,tomoveawayfromthat[initialimpression]andsay,hey,butbythiscriterion..."Thisrealizationisexactlythesortofmetacognitiveprocessthatthisworkshopandongoingtrainingactivitiesaimedtofoster.Ava rietyofphraseswerei nformally utteredormoreformally proposedbyas sessors,basedontheirre flections afterlistening toanumberofhighscoringrecordin gsofexaminationperformances.Theseph rasesarerecountedinthissectiona ndhavebeenseparatedthematically.Itwasagreedthatcentraltomitigatingthehaloeffectwastheideathateachcriterionmustconsciouslybejudgedorreferenced aspart ofunpackingevidenc efroman overallperformance.Consequently,thefirststatementtobenotedfortheinventorywas:JudgeeachcriteriononitsmeritsThisstatementencapsulatestheintentoftheworkshopbutalsoprovidesalinktotheannualtrainingthatassessorsundertake.

_________________________________________________________________________________________________________Zoanetti&ChampionMitigatingtheHaloEffect:ManagingtheWowFactor43Thenextstatementcapturedherereferencestheroleofanassessorjuxtaposedwiththeusualr olesthatassesso rsasmusiciansand teachersunde rtake,thatcontributesignificantexperiencestotheexpertisetheybringtobearintheirjudgments:Asmusician sandteachersyousometimes [experience]the emotiveresponse...[butinthiscontextyouneedto]bringtheassessorintoplay...Again,thisstatementaddsvaluetotheinventoryasbothatruismandareminderoftheassessmentprinciplesthatunderpintheannualassessortrainingworkshops.Athirdsetofstatements,thistimetargetingexceptionalperformances,particularlythosethatwereexceptionalinatleastonedistinctrespect,includedI'llneverhearanythinglikethisagain,butthen,iftheyhaven'tmetthatcriterion,...ThisisthebestI'veseenin10years.Imayneverseesomethinglikethisinanother10years,butisitactuallyfullmarksforthis[criterion]?Itiseverybody'schallenge...tostandback...becausethatsortofresponse...IhearthatinmyselfsooftenwhenI'veheardamazingperformances.AspectsofwhatI'veheardtodayaretheverybestI'veheardbut...Thesestatementsdemonstrateassessorawarenessthatwhilstthecontextfortheperformancesisanexaminationtoawardscoresthatcontributetostudentachievementinaseniorsecondarycertificateofeducation,aproportionofthestudentcohortdemonstratelevelsofachievementthatarecommensuratewiththoseexpectedintheearlyyearsoftertiarymusicstudy.Thes tatementsalsod emonstrateawareness thattheass essmentprocessiscrit erionreferenced,a ndthat theva riouscriteriaset expectationsforconceptuallydistinctaspectsofmusicalperformance.Afourthsetofstatementsfocusedontheideathattheawardingofamaximumscoreforacriterionshouldtriggeranexplanationintermsoftheevidenceapparentintheperformance.Challengeeachofyour10sJustifyyour10sThesestatementsaresimplisticyettheydescribeineverydaylanguagethesortsofstop-checksormetacognitiveguidancethatmightmitigatethehaloeffectwhereitseemedmostlikelytoemerge;thatis,ininstanceswhereaperformancehasthewowfactorinthiscontext.Afterbeingconsid eredcollectiv ely,anumberofs tatementsofmetacognitiveguidanceweredistilledtocoverthekeythemesandtoleveragethelanguagecapturedthroughthegroupdiscussions.MetacognitiveguidancetomitigatethehaloeffectFollowingareviewofthewordsandphrasesusedbyassessorsinrelationtothehaloeffect,aninitialsetofshortandstraightforwardguidancestatementswasconstructed.Again,thefocuswasonthehigherscoresforeachcriterion.Atthetimeofwriting,thesewere:TABLE2:Assessoradvicecard

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 44AspectsofthisperformancewereamazingbutIstillneedto:1. AssumetheroleofAssessor2. Judgeeachcriteriononitsmerits3. Justifyeachofmy10sThesestatementsweredraftedasabasisfordevelopinganextraresource,possiblyintheformofacuecard,forassessorstobringwiththemtotheperformanceexaminations.Inthewordsofoneassessor,theseweredescribedaschecking/red-lightsentences,whichcouldbereferredtoandcouldbeusedtopromptrobustdiscussionbetweenassessorsobservingthesameperformance.Giventheneedtocontaintheadministrativeburdenforassessors,onegoalwastoensurethatthisguidanceremainedsuccinct.UnpackingperformanceevidenceagainstthecriteriaDuringtheworkshop, raisingaw arenessaboutthehaloeffect andprovidingmetacognitiveguidancetoavoiditwere seenasproductiveinitia lstrateg iesforsafeguardingreliabilityandvalidity.Inparallelwiththedevelopmentofthesestrategies,detaileddiscussionsofpe rformanceevidencewerefacilitated afterea chrecordedperformancehadbeenlistenedto. Thepurpose ofthesedisc ussionswastoreinforcecommonexpectationsaroundeachcriterionscorepoint.Thiswasviewedasimportantforsupportingassessorsinmakingco nsistentandevidence-baseddeterminatio nsofthecriterionscoresthateachpe rformanceshouldrece ive.Ofparticu larimportancewas discussionaboutthenatureofsalientdifferencesinperformancequalityevidencethatwouldstronglyi ndicatetheappropriatene ssofaparticularscore categoryov ertheadjacentscorecategories. Asampleofthela nguageusedbyassessorstodescribethequalityofevidenceagainstthevariouscriteriaistabledinthethirdcolumnofAppendixA.Elicitinganddocumentingthisextralayerofdetailaboutperformanceevidenceispartofalargerprogramofworkfocusingonratereffects,butwithinwhichthisexplorationofthehaloeffecthastakenplace.Allofthesestreamsofworkarecomplementaryastheyeachpromotethecentralityofthecriteriaandaimtoforgevalidandreliablelinksbetweenthequalityofevidenceapparentinobservedperformancesandthepatternsofscoresallocated.Astheassessorsreviewedtherangeofperformances,discussionsfocusedontwoissuesoftenevidentinperformanceswithawowfactor.Theseissueswereagreedtohaverelevanceirrespectiveofdifferencesininstruments,styles,genresandrepertoireselectedbythestudents.Firstwasthematterofglitchesandsecondwasthematterofrepertoireselection.Discussionaboutglitchesfocuse donevidencein theperformancesuchasoccasionalwrongnotesorslightinconsistenciesinintonation,forexample:RangejumpsimpressivegiventheirdifficultyAcoupleofstumblesdon'tconstituteareductioninmarks.It'sstilla10because...Prettymuchflawless...inexecutionandcleanness

_________________________________________________________________________________________________________Zoanetti&ChampionMitigatingtheHaloEffect:ManagingtheWowFactor45They're10sandthey'veshownthatsomanytimesthatwehavetogivethema10,ordowepullitbackbecausetheydidinfactfalldownhereorthereveryslightly?Canastudentmakeupforerrors?Thepresenceofthisphenomenoninlivemusicperformancewasacknowledgedandthegroupagreedonguidelinestoassistindeterminingthelevelatwhichglitchesshouldimpactonscoring.Therationaleunderpinningtheseguidelinesfocusedonthewoweffectandtheextenttowhichtheglitchesorinaccuraciesinterruptedtheflowandintensityoftheperformancefortheaudience.Forexample,inrelationtoaccuratepitchingsuchaspitchingofhighnotesbyatrumpetplayer,itwasagreedthatanoccasionallackofclaritydidnothaveadisruptiveeffectonthelistenerandthataperformancewherethestudentdidmakesome smallerrors couldcommuni cateawoweff ect.However,repea tedexamplesofmissedhighnotescouldcausethelistenertofocusonthehighnotesandnotonthemusic,infactcreatinganeffectofwow,but....Thesecondi ssueofrepertoires electionsforthe examinati onprogramincludedconsiderationoftherangeofperformanceandexpressivetechniquesavailableinthemusicworksselectedbythestudenttodemonstratetheirskillagainstthecriteria.Forexample:CertainsamenesstoprogramintermsofdynamicandtechniqueDidn'tallow[him]tofullyexploitthecriteriaStudentdidn'tchoosetopush[himself]Theskillinthepart[he]demonstratedwasveryhighbut[he]couldhaveallowedasectiontoshowahigherleveloftechnique.Thesecommentsacknowledgethatthequalitiesaperformercandemonstrateinperformanceareinpartdictatedbytherepertoireselectedandthatwhilststudentsandtheirteachersneedtoconsiderhowrepertoireoffersopportunitiesorimposeslimitationstodemonstrateskillagainsteachoftheassessmentcriteria,theassessors'taskistoscoretheperformancetheyhear.Itwasagreedthatinsomecasesaninitialresponsemightbe"wow"butconsiderationoftherangeofqualitiesevidentintheperformancecouldmitigatethisresponseinrelationtooneormorecriteria.CONCLUSIONSThispaperdescribesinitialstepsundertakentostrengthensupportforassessorsaroundknownchallengesinjudgingthequalityofcomplex,extendedperformancesagainstprescribedcriteria.Morespecifically,strategiesforaddressingtheratereffectknownasthehaloeff ecthavebeen outlinedinsomedetai l.Whilsttheimp lementation ofthesestrategiesisinitsearlystages,participatingassessorsanecdotallyreportedthat:assessorsupportfortheprojectanddirectionisstrong;assessorsfeelcomfortablewiththeprocessbecausetheadditional materialisb eingderived fromauthenticconversat ionandnot imposed;and,thediscus sionsareackn owledgingth ediversityofperformancesthestudentspresentinresponsetothecurriculum.

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 46Thisprogramof workcontinuestobu ildonan drefinethea ssessortrainingprograms.Guidancematerialsneedtobeasrigorousaspossibleandresponsivetowhatisunderstoodanddocumentedaboutthreatstoreliabilityandvalidityinthejudgmentofperformancequality.Otherstrategiestomitigateratererrorswillalsobeimplementedconcurrentlytostrengthentheoverallapproach.Theprimarymethod,whichisalreadyastapleofassessortraining,reliesontheuseofexemplarrecordingsspecificallychosentocoverdifferentregionsoftheavailablescorerange.Somerecordingmightbechosenwithspecificallytargetedperformanceidiosyncrasies,ormu sicalfea turesthatarerelat ivelynew,forexample arisingth roughtheuseofnew technologiesinmusicp erformanceexaminations.Theseexemplarscanbeusedtoidentifyassessorswhodonotconformtothejudgingstandard,andcanbeusedtoidentifyarangeofraterserrorssuchasleniencyorseverity,haloeffect,restrictionofrange,andcentraltendencyatthepointofassessortraining.Considerationwillalsobegiventohowtheeffectivenessofthisevolvingprogrammightbeevalua tedusing longitudinal,quantitatived atafromtheasses smentprogram,shouldsuchanevaluationbetractable.REFERENCESBaird,J.,Hayes,M.,Johnson,R.,Johnson,S.,Lamprianou,I.(2013).Markereffectsandexaminationreliability:Acomparativeexplorationfromtheperspectivesofgeneralizabilitytheory,Raschmodelingandmultilevelmodeling.OxfordUniversityCentreforEducationalAssessment.RetrievedOctober1,2013fromhttp://www.ofqual.gov.uk/files/2013-01-21-marker-effects-and-examination-reliability.pdfBergee,M.J.(2003).Facultyinterjudgereliabilityofmusicperformanceevaluation.JournalofResearchinMusicEducation,51,137-149Ciorba,C.R.,&Smith,N.Y.(2009).Measurementofinstrumentalandvocalundergraduateperformancejuriesusingamultidimensionalassessmentrubric.JournalofResearchinMusicEducation,57,5-15.Eckes,T.(2009).Many-facetRaschmeasurement.RetrievedOctober1,2013,fromhttp://www.coe.int/t/dg4/linguistic/Source/CEF-refSupp-SectionH.pdf.Engelhard,G.,Jr.(1996).Evaluatingrateraccuracyinperformanceassessments.JournalofEducationalMeasurement,33(1),56-70.Ericsson,K.A.&Simon,H.A.(1993).ProtocolAnalysis:VerbalReportsasData,MITPress,Cambridge,MA.

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 48fromhttp://www.vcaa.vic.edu.au/Documents/exams/music/mus-invest-specs-crit-w.pdfWesolowski,B.,Wind,S.A.,&Engelhard,G.(2015).RaterFairnessinMusicPerformanceAssessment:EvaluatingModel-DataFitandDifferentialRaterFunctioning.MusicaeScientiae,19(2),147-170.Wolfe,E.W.(2004).Identifyingratereffectsusinglatenttraitmodels.PsychologyScience,46,35-51.

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 50CriterionAnnotationsSampleinventoryoflanguageusedbyassessors(indevelopment)theselectedrepertoire•theabilitytovarytoneinawaythatisappropriatewithintheFocusAreaVeryexpressive,reflectingemotionalresponsetothemusic5.SkillinarticulatingandphrasingWithinthecontextoftheFocusArea:• creationofpurposefulshapethroughartisticvariationofexpressiveelementsincludingtonalquality,tempo,phrasing,articulation,dynamicsandtexture•communicationbeyondthesourcematerial(notationand/orreferencerecordingofaworkbyusingappropriatenuancesandperformancetechniquesSophisticated,well-defined,refined,elegant,informedExceptionalclarityDexterouschangingbetween[articulation-tonguing,bowing]techniquesPhrasing-widevarietyofshapingVerycleanandappropriatelyweightedSensitivephrasingWiderangeofdynamicswithtransitionsthoughtfullymanagedDifferentdynamicpalettesevidentacrosstheprogram6.SkillindifferentiatingthestructuresandtextureswithineachworkoftheFocusArea•theperformancecommunicatesunderstandingofthevarietyofstructuresandtexturescontainedintheprogram•effectivedifferentiationofthestructuresandtextureswithineachworkoftheprogram•thevarietyofstructuresandtexturesintheprogramisrepresentativeofstructuresandtexturesthatarecharacteristicoftheFocusAreaHighlyinformedAwarenessofissuesevidentDepthofunderstandingdemonstratedExcellentsenseofdirectionusingcontraststodifferentiate7.SkillindifferentiatingthemusicallinesintheselectedworksasAsappropriatetotheinstrumentand/orinstrumentalcontext:ConsistentFluentSynchronization

____________________________________________________________________ Journal of Research in Music Performance Spring 2016 52CriterionAnnotationsSampleinventoryoflanguageusedbyassessors(indevelopment)programarerepresentativeoftheFocusArea•thedegreetowhichthestudentsucceedsthroughtheirperformanceinmakingconnectionsbetweenworksintheprogramandtheFocusStatementsuchasfactorsthatunifytheworksorrealizingcharacteristicsrelevanttotheFocusArea•theextenttowhichtheprogramdemonstratesdiversityappropriatetotheFocusAreahighlyevidentinsightful,informedinterpretation11.SkillsinthepresentationofacohesiveprogramrelevanttotheFocusArea•useofrelevantmusicalelementsofperformancepresentation•theflowoftheprogram(whichmayincludebriefverbalcommentary)•effectivemusicalcommunicationwithaudienceCommandingpresenceFluent,highlyorganizedpresentationPolishedExcellentpoiseThewholepackageofawell-structuredprogrampresentedinaconvincingmannerIssuesforconsideration:- endurance- performancecontouring- maximizingdegreeofdifficultyconsiderations