An Efficacy Trial of Research-Based Curriculum Materials With

130375_7ED556336.pdf

An EfficacyTr ialofResearch-Based

CurriculumMaterials WithCurr iculum-Based

ProfessionalDev elopment

JosephA. Ta ylor

Abt AssociatesInc.

Stephen R.Getty

Colorado College

Susan M.K owalski

ChristopherD .Wilson

Biological SciencesCurriculum Study

Janet Carlson

StanfordUniversity

PamelaV anScotter

Biological SciencesCurriculum Study

This studyexamined theefficacy ofa curriculum-basedintervention for high schoolscience students.Specifically, theintervention wastwo yearsof research-based,multidisciplinarycurriculum materialsfor sciencesup- ported bycompr ehensiveprofessionaldevelopment forteachersthatfocused on thosematerials. Amodest positiveeffect wasdetected whencomparing outcomes fromthisintervention tothose ofbusiness-as-usual materials and professionaldevelopment.However ,this effectwastypicalfor interven- tions atthis gradespan thatar etested usinga stateachievementtest.Tests of mediation suggesta large treatmenteffecton teachersandintur na strong effect ofteacher practiceon studentachievement - r einforcing thehypothe- sized keyr oleofteacherpractice. Testsof moderationindicate nosignificant treatmentby demographicinteractions. K EYWORDS: curriculummaterials, curriculum-basedprofessional develop- ment, efficacytrial, highschool science,inquiry, BSCS5E Instructional Model

The CurrentStateof Curr iculumEf ficacyResearch

Science educationis ata criticaljuncture whereevidence ofthe effects of curriculummaterials isgreatly needed.It isnot enoughto developcurric- ulum materialsand professionaldevelopment (PD)programs whose

American EducationalResearc hJournal

Month XXXX,V ol.XX,No.X, pp.1-34

DOI: 10.3102/0002831215585962

2015 AERA.http://aerj.aera.net by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from component featuresare basedon extantresearch andhope theyare effec- tive. Manystakeholders ineducation, practitionersin particular, needevi- dence fromrigorous trialsabout whichcomprehensive programs(i.e., year-long programsusing multiple,integrated features)have thegreatest effects onstudent outcomes.Although numerousfederal agencieshave funded thedevelopment ofcurriculum materialsover thepast 30years, the fieldof scienceeducation stilllacks evidenceregarding whatprograms (or typesof programs)have noteworthyeffects. Asa consequence,school and districtdecision makershave hadlittle guidancetoward implementing potentially moreefficacious programsthat mightdisplace thosethat have smaller orno effects(Slavin, 2008)and noway toappropriately respond to theneed forimproved STEMeducation. The Institutefor EducationSciences (IES)established theWhat Works Clearinghouse (WWC)to providestakeholders ineducation withinfor ma- tion onprograms thathave undergonerigorous efficacytrials. TheWWC provides stakeholderssuch asschool districtsinfor mationwith whichto make evidence-baseddecisions relatedto instructionalinterventions. At JOSEPHA. TAYLORis aprincipal associate/scientistat AbtAssociates Inc.,55 Wheeler Street, Cambridge,MA 02138;e-mail: Joseph_Taylor@abtassoc.com. Hewas director of researchand developmentat BiologicalSciences CurriculumStudy (BSCS) when thisresearch wasconducted. Hisresearch focuseson studyingthe impactof curriculum materialsand professionaldevelopment onstudent outcomesand the policy implicationsof suchstudies. S TEPHENR. GETTYis thedirector ofthe QuantitativeReasoning Centerat Colorado College inColorado Springs,Colorado. Hisresearch ineducation focuseson meth- ods tomeasure studentmotivation inSTEM disciplinesand faculty/teacherinterven- tions toincrease studentmotivation. S USANM. KOWALSKIis asenior scienceeducator atBSCS. Herresearch focuseson developing andstudying sciencecurriculum andprofessional developmentpro- grams inonline contexts. C HRISTOPHERD. WILSONis seniorscience educatorat BSCS.His researchfocuses onthe measurement ofteacher andstudent learning inscience education,measuring teacher pedagogicalcontent knowledgewith humanand automatedscoring, and the impactof lessonanalysis-based professionaldevelopment. J ANETCARLSONis anassociate professor(research) andthe directorof theCenter to Support Excellencein Teacher (CSET)intheGraduate Schoolof Educationat Stanford University.She wasthe executivedirector atBSCS whenthis research was conducted.Her researchfocuses onthe interplaybetween transformative profes- sional development,the enactmentof curriculum,teacher practice,and student learning. P AMELAVANSCOTTERis currentlysenior associatedirector atBSCS, whereshe oversees the programmaticwork incurriculum development,professional development,and research andevaluation. Sheserved asprincipal investigatorfor theproject that developed theinstructional materialsthat werethe subjectof thisstudy. Shehas a backgroundin anthropologyand linguistics.

Taylor etal.

2 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from the timethis articlewas written,just eightstudies ofscience interventions had metthe evidencestandards ofthe WWC(IES, 2014),and onlyone is related tocurriculum materialsor PDfor useat thehigh schoollevel. This lackof evidenceis particularlytroubling aswe findourselves at a timeof rapidrefor m.As thenewFramework forK-12 Science Education(National ResearchCouncil [NRC],2012) andthe Next Generation ScienceStandar ds(NGSS; NGSSLead States,2013) havebeen released, teachersacross thecountry willbe expectedto teachnew disci- plinary coreideas, crosscuttingconcepts, andscientific practices.While national documentsclearly describethe threedimensions ofNGSSwell, min- imal empiricalevidence accompaniesthese documentsabout therole of research-based curriculummaterials insupporting studentattainment of standards aswell asthe natureof thePD programsmost likelyto helpteach- ers implementeffective instruction.It istherefore imperativethat curriculum and PDprograms thatattempt to,or claimto, supportsuch goalsare sub- jected torigorous trialsthat canmake confidentcausal claimsabout their impacts.

Study Overview

This studysought totest thecausal linkbetween acurriculum-based sci- ence educationintervention andincreased studentachievement. Thepri- mary goalsof theresearch wereto (a)test theoverall efficacyof research- based curriculummaterials withassociated PDfor improvinghigh school science achievement,(b) explorethe roleof teacherpractice inthe relation- ship betweenuse ofthe curriculummaterials andimproved studentachieve- ment outcomes,and (c)explore theextent towhich treatmenteffects were equitable acrossdemographic groups.The curriculummaterials understudy are titledBSCS Science:An InquiryAppr oach(hereafter referredto asAn Inquiry Approach). Thematerials weredeveloped withfunding fromthe National ScienceFoundation (ESI9911614 andESI 0242596)

Theoretical Underpinningsof Research -Based

CurriculumMaterials

In thisarticle, wedefine curriculummaterials toinclude boththe student text aswell asteacher supportmaterials. Thissection addressesthe theory behind boththe studentand teachercurriculum materials. The Roleof Curr iculumMaterialsinScienceClassrooms Curriculum materialscan bea meansto improvestudent interestand achievement inscience (NRC,2007). Thenotion thatcurriculum materials truly matterand directlyinfluence thelear ningprocess hasbeen supported

in theliterature fordecades (e.g.,Forbes &Davis, 2010;Schmidt, McKnight,Efficacy ofCurriculum withPD

3 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from & Raizen,1997; Usiskin,1985). Curriculummaterials playa definingrole in classrooms, affectingboth whatand howteachers teach(NRC, 2002).Ball and Cohen(1996) explainthis powerfulinfluence: Unlike frameworks,objectives, assessments,and othermechanisms that seekto guidecurriculum, instructionalmaterials areconcrete and daily.They arethe stuffof lessonsand units,of whatteachers and studentsdo. .. .Not onlyare curriculummaterials well- positioned toinfluence individualteachers' workbut, unlikem any other innovations,textbooks arealready ''scaledup'' andpart of the routineof schools.They have''reach'' inthe system.(p. 6) Further,Schmidt, Houang,and Cogan(2002) cautionagainst effortsto improve instructionthat areisolated fromefforts toimprove curriculum materials availableto teachersand students.''If wepretend thatthe textbook doesn't exist - andconduct PDin waysthat assumeteachers willimplement an entirelydifferent approachto contentthan thetexts take - believeme, the textbook willwin'' (p.18).

Constructivismand StudentLear ning

Constructivism isa keyfoundation thatframes research-basedcurricu- lum materialsdesigned toemphasize opportunitiesfor studentsto develop conceptual understandingsof science.Our workis basedon twocommon theoretical basesfor constructivistresearch: Ausubeliantheory (Ausubel, Novak, &Hanesian, 1978)and thework ofL. S.V ygotsky(1978). Ausubelian theorystates thata learner's priorknowledge isanimportantfac- tor indeter miningwhatislear nedin agiven situation.Vygotsky'swork emphasizes therelationship betweenthe teacher'sprior knowledgeand the students'prior knowledgeas wellas theimportance ofthe socialcon- struction ofknowledge. Studentsand teachersmay usesimilar wordsto describe conceptsyet havevery differentpersonal interpretationsof those concepts. Vygotsky'sworkimplies thatscience curriculumand instruction should takeinto accountthe differencesbetween teacherand studentcon- ceptions andshould providetime forstudent-to-student interactionso that learnerscan developconcepts fromthose whoseunderstandings andinter- pretations arecloser totheir own. Discussions ofconstructivist teachingand learning havebeen hampered by inconsistencyin howit isenvisioned bydifferent scholarsand research- ers, includingbeing equatedwith completelyunguided or''discovery'' learn- ing. Hmelo-Silver,Duncan,and Chinn(2007) discussthe rangeof perspec- tives andoffer thisdefinition ofa constructivistlear ningenvironment: an environment inwhich ''studentsare cognitivelyengaged insensemaking, developing evidence-basedexplanations, andcommunicating theirideas. The teacherplays akey rolein facilitatingthe learning processand maypro- vide contentknowledge ona just-in-timebasis'' (p.100). Itis thisinterplay Taylor etal. 4 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from between thestudent assensemaker andthe teacheras facilitatorthat defines our viewof constructivistlear ningenvironments. Todaywe seethe rootsof constructivismreflected incomprehensive reviews ofthe literatureon learning, suchas How PeopleLear n(NRC,

2000). Theauthors summarizethree keyideas aboutlear ning,suggesting

that studentscome tothe classroomwith preconceptionsthat shapetheir learning,student competencerequires adeep foundationof knowledgeas well asan understandingof howthis knowledgerelates toa framework, and studentsbenefit fromexplicitly monitoringand takingcontrol oftheir own learning.TheInquiry Approachcurriculum materialsincorporated into thestudy interventionwere stronglyinfluenced bythese findings.

Coherence, Focus,andRigor

Though morethan adecade old,the findingsfrom theT rendsin Mathematics andScience Study(TIMSS) analysis(Schmidt etal., 2001)and the researchsynthesis How PeopleLear n(NRC, 2000)provide clearand com- pelling guidancefor thedevelopment ofeffective curriculummaterials. In general, thesereports indicatethat curriculummaterials inthe United States needto bemore focusedby havinga storylineor ganizedaround key concepts,more coherentby havingexplicit connectionsbetween ideas, and morerigorous bysetting highstandards forlear nerswith respectto both their cognitiveand metacognitivedevelopment. Curriculummaterials devel- oped withina frameworkthat iscoherent bothwithin yearsand acrossyears facilitates adeeper studentconceptual understanding(American Association for theAdvancement ofScience [AAAS],2001; Carlson,Davis, &Buxton,

2014; NRC,1999, 2007).Y et,persistent evidenceindicatesthatcurriculum

materials inscience arefragmented, lackingcoherence, andnot wellarticu- lated througha sequenceof gradelevels (AAAS,2001; Kesidou& Roseman,

2002; Schmidtet al.,1997, 2001;Schmidt, Wang, &McNight, 2005).As

a result,curriculum materialsin theUnited Statesgenerally covermany con- cepts, oftenrepeating conceptsannually withoutdepth (Schmidtet al.,1997,

2001). Mostmaterials focuson detailsthat aretangential tothe keyideas and

fail tomake connectionsacross unitswhen thesame keyidea ispresented in different ways(Kesidou &Roseman, 2002). Educative MaterialsandT eacherSuppor tintheClassroom Research-based curriculummaterials forstudents willnever eliminate the importantrole ofthe teacherin theclassroom. Teachers ultimatelyshape how curriculummaterials areenacted inthe classroom(Beyer &Davis, 2012; Forbes &Davis, 2010).T eachersselect elementsoftextto includefor instruc- tion, andthey emphasizeor deemphasizeaspects ofa curriculumbased on their ownunderstanding andbeliefs aboutwhat isbest forstudents. Remillard (2005)described thecomplex teacher-curriculumrelationship asEfficacy ofCurriculum withPD 5 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from contextually based,dependent onboth theteacher andthe curriculum,and tightly interconnectedwith otherteaching practices.If ateacher's under- standings andbeliefs aboutinstruction alignwith thephilosophy ofthe cur- riculum, thenit islikely thatthere willbe asynerg isticrelationship between use ofthe materialsand practice(Powell &Anderson, 2002).On theother hand, ateacher mayunderstand instructionand holdbeliefs aboutpractices that divergefrom thephilosophy ofthe materials,creating agap between what curriculumdevelopers intendedand whatthe teacheractually enacts in theclassroom (Ball& Cohen,1996). Becauseofthe ubiq uitous placementofcurriculumma terialsintheschool setting,thereis uniquepotent ial forcurricu lummaterialstosu pportte achers asle arners.Teachersma yusetheircurricul ummater ialstode epentheirc on- tentkno wledge,gainideasforho wtopre sentcomplex informa tionto stu- dents,ordet ermin ehowtheymightass essstud entlea rning.Somere searchers describecurric ulummaterialsthat explicitlyaddres stheteach eraslearneras ''educative''(Beyer,Delg ado,Davis,&Kraj cik,2009;Davis& Krajcik,20 05). Davisand Krajcik (2005)identif iednineheuristi cstodescribeeducative sci- encema terialsandhowscie ncecur riculum materialscansu pportteachers' enactmentofreform -ba sedinstructionwiththeir students.Theheuristic sfocus on teachersubjec tmatterknowledgea swellasp edagogicalcontent knowl- edge(P CK).Inaddition ,t heyarticulatethe importance ofincludingarationale forcu rriculardesigndecisi onsandpro vidingsupportsforte achers toadapt materials.Specifica lly,embeddededucativeteach erresourcescan include informationandration aleonth einstructionalmode l,a dditiona lscientific background,alternati veunderstandingsstu dentsmayhaveassociatedwith theco ntent,aswellas suggest ionsfor enhancin gstudents'abilitiesto functio n asa grou p.Assuch,ed uca tivescience curriculabecomeare sourceforteach - ers,su pportingthemasthe yusemater ials intheirown instruc tionalsettings. Schneider andKrajcik (2002)observed thatteachers whouse educative materials enhancetheir contentlear ningand betterimplementspecificstrat- egies andrepresentations suggestedin thematerials. Similarly,Davis and Krajcik (2005)noted alink betweenteachers' useof educativecurriculum materials forscience andtheir PCKfor correspondingscience topics. Theoretical Underpinningsof Curr iculum-BasedProfessionalDevelopment Although educativecurriculum materialshave clearadvantages, theyare often complex,and teachersbenefit fromadditional supportto fullyunder- stand thecurricula theyare tryingto implement(Davis &Krajcik, 2005).For example, teachersoften donot fullyavail themselvesof theinfor mationpro- vided inthe teacher'smaterials. Inaddition, evenwhen teachersdo study the educativematerials, theywill likelyinterpret theinfor mationusing their own experientiallenses (McNeill,2009). Thus,the evidencesuggests that integrating educativematerials withface-to-face PDcould bethe mostTaylor etal. 6 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from effective approachto enhancingteachers' understandingof thephilosophy and keyfeatures ofcurriculum materials.This shouldbe particularlytrue when thePD incorporateselements ofeffective PDpractice. Loucks- Horsley, Hewson,Love, andStiles (2003)reviewed extensiveresearch on the characteristicsof effectivePD andidentified severalthat areparticularly germaneto PDaimed atsupporting theuse ofcurriculum materials.These effective practicesinclude providingcoherent, ongoing(i.e., multi-event) programs thatmirror bestpractice; keepinga focusdirectly onlear ning and teaching;and providingteachers opportunitiesto developdeep under- standing ofconcepts andparticipate incommunities ofreflective practice. Last, whenteachers adoptresearch-based curriculummaterials, itis essential that theylear naboutkeyfeatures ofthe materialsas wellas therationale for why developersincorporated thosekey features(Lin &Fishman, 2006). These featuresinclude theinstructional modelfor thematerials (Penuel, Gallagher,& Moorthy,2011) aswell askey elementssuch asbuilding an inquiry culture,using sensemakingstrategies, andunderstanding thestory- lines foreach unitof instruction.The conceptof educativematerials for teachers supportedby complementaryPD merges twoareas ofresearch: (a) therole ofeducative materialsand (b)the numerousreports indicating that PDfocused onthe implementationof well-designedmaterials can have asignificant impacton teachingand learning (Briars& Resnick,

2001; Darling-Hammond,1997; Heller, Daehler,Shinohara,& Kaskowitz,

2004; Ladewski,1994; Powell& Anderson,2002; Schneider& Krajcik,2002).

The ResultingExper ienceforStudents UsingAn Inquiry

Approach: TheT reatmentCondition

CurriculumMaterials

These threemajor theoreticalunderpinnings forcurriculum materials (constructivism, coherence,and educativeness)for medthe foundationfor the designof An InquiryAppr oach. Asa result,this highschool program aims tosupport notonly thelear ningof scienceconcepts butalsothedevel- opment ofa cultureof learning thatempowers studentsandteachersto learn science andconduct scientificinquiry.

Constructivism

One ofthe primaryways thatthe materialsattend tothe researchon constructivism isby structuringlear ningaround theBSCS5EInstructional Model (Bybee,1997; Bybee& Landes,1990). Inparticular ,the BSCS5E (engage, explore,explain, elaborate,and evaluate)Instructional Modelsup- ports theteacher inscaffolding thelear ningexperiences forstudents and provides aresearch-based, socialconstructivist storylinethroughout each chapter.The BSCS5E InstructionalModel organizes theinstructional Efficacy ofCurriculum withPD 7 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from sequence sothat studentshave multipleopportunities todevelop adeep understanding ofconcepts throughpractice, feedback,revision, andreflec- tion. SeeT able1fordetails. In theinit iallessonofeac hchapter, the reareoppor tunitiesforthestu- dentstoco nsid er,express,represe nt,andsharet heircurrentunderstand ing aboutaco ncep t.Thisisacritica loppor tuni tyf orstudentsthats etsthestage fora ndpromot eslearning(NRC, 2000).Thisint urnhelpsthe teacherframe thesub sequentlessonsandl abactivitiesw herestu dents havetheopportunit y toex plorequestions insmallteams.Thes eex periencesre sultinasetofco m- monex periencesonwhichteams will continueto buildtheir understa nding. Exploratory labactivities leadto otherlessons andinteractive readings that helpstudents formulate andarticulatetheirfoundational understanding. Activities designedto reinforceand expandstudents' understandingfollow. These labactivities oftenask thestudents totest theirunderstanding ina dif- ferent settingor byadding anew variable.Many ofthe lessonsalso provide opportunities forcollaborative learning thatmodelthescientific enterprise. Such workin heterogeneousgroups promotesthe back-and-forthprocess essential toknowledge construction(V ygotsky,1962). The roleof forma tiveassessmentinthesecurriculummaterialsisimportant forboth th estudentsan dtheteachers(Atkin,2 002;Black&W iliam,1998; NRC,

2001).D uringeachlesson, thestudents completetasksand respon dtoques -

tions thats erveasbench marksfor students andt eacherstoa ssesstheirlear ning

Table 1

How CurriculumMaterials Embodythe Principlesof Learning

Principles ofLear ningFromKnowing

What StudentsKnow

a

Related Featuresof theMaterials

Instruction isorganized around

meaningful problemsand goals.Activities centeron relevantproblems and/or currentresearch withan inquiry focus.

Instruction mustprovide scaffoldsfor

solving meaningfulproblems and supporting learningfor understanding.Instruction followsthe 5EInstructional

Model; scaffoldingis particularly

strong inthe explore/explaincycles.

Instruction mustprovide opportunities

for practicewith feedback,revision, and reflection.The materialsinclude bothfor mative and summativeassessments aswell as metacognitive strategiesaligned with the activities.

The socialarrangements ofinstruction

must promotecollaboration and distributed expertiseas wellas independent learning.The materialsinclude anappropriate mix ofsmall teamactivities, partner projects, jigsaws,presentations, class discussions, andindividual work. a

Pellegrino, Chudowsky,and Glaser(2001).

Taylor etal.

8 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from experiences.Inadditio n, studentshavemul tipleopportunitiesto develop explanationsfromevi denc eusingappropriatescaffo lds(McNeill& Krajcik,

2007).Atthe end of achapter, studentsc omplet eacomprehensivelab activity

or othertyp eoflessonde signe dtodem onstratetheirunderstandin gfort hem- selvesandf orthei rteacherwi threspect toco reco nceptspresented inthec hap- ter.This experience servesasasummative assessmentf ort he chapter . The instructionalmodel inthese materialssupports boththe teacherand the studentin creatinga cultureof inquiryin theclassroom andprovides opportunities forstudents todevelop anunderstanding ofscience byprac- ticing science(NGSS LeadStates, 2013)and reflectingon itsnature, their experiences, andtheir findings.Through acoherent sequenceof theselab activities andquestions aimedat improvingcritical thinkingskills, thestu- dents, overtime, haveopportunities tolear nexplicitly aboutthe natureof science inthe contextof learning rigorouscontent.

Coherence

Thef eatureofcoherence incu rriculummaterialsis crit icaltostudent learning(Ruth erford,2000;Schmidtetal., 1997)andf oundationalto the cur- riculummateri alsusedduringthein tervent ion.Because weknow thatstu- dentsare moreli kelytolearnbe stwhentheir learn ingexp eriences are groundedwithi nacoherentconc eptu alframework (NRC,2000),thedevelo p- mentof such aframew ork wasacriticalfir ststep int hedesignofthecurric - ulumma terialsusedinthisst udy.The use ofafra meworkfacil itatedth edevel- opmentofa focu sed ,conceptualstor ylinewithineachchapter andacross chapters,resultin ginacoherentlearnin gex per ienceforstuden ts.Eachof theco reunitsc omprisesfour chapters,withea chunitex posingstudentsto fundamentalconcepts inoneofthescience disc iplines (i. e.,phys icalscience, lifesc ience,earthscie nce,science andsociety), andthismultidis ciplina ry cyclerepe atsinGrades10 and 11(seeframewo rkin Table2) .Th elastchapt er in eachcor eunitallow sstuden tstoapplywhat theyhavelearned thus farin an integratedcontext.The resultofthisart iculatio namongunit sis thatth e numberofma jor conceptsst udentslearnisfew erandtheexperienc eswith thosecon ceptsaremoresophi sticate d,with thegoalthatstudents'under- standingofthem isde eperandmore comp lex(Schmidt etal., 2001). Tofurther promotecoherence inAn InquiryAppr oach, thedevelop- ment teamused theUnderstanding byDesign process (Wiggins& McTighe, 2005).This processis dividedinto threemain stages.In StageI, Desired Results,theteam identif iedtheendu ringunderstandingsfor student s basedonth ena tional standards(theNationalScien ceEducationSta ndardsat thattime )andscient ificexpe rtise.InStageII ,AssessmentEvidence,th ete am developedtheass essment tasksthatwouldserve asevidenceth atthestud ents hadga inedthetar getedun derstandings.InSta geIII,LearningPlan,theteam developedthesequ enceofle arningexperienc esth atwerehypoth esizedtoEfficacy ofCurriculum withPD 9 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from

Table 2

Curricular Frameworkfor An InquiryApproach WithNext GenerationScience StandardsAlignment

Major ConceptsAddressed atEach GradeLevel

Units 910 11

Science asInquiry (also

integrated throughout

each contentarea) ?Questions andconcepts that guidescientific investigations?Design ofscientific investigations

?Communicatingscientific results?Evidence asthe basisfor explanations andmodels ?Alternative explanationsand models Physical Science?Structure andproperties of matter ?Structure ofatoms ?Integrating chapter (DCI PS1.A,PS1.C) ?Motions andforces ?Chemical reactions ?Integrating chapter (DCI PS1.B,PS2.A, PS2.C,

PS3.A, PS3.B)?Interactions ofener gyandmatter

?Conservation ofener gyandincrease indisorder (DCI PS3.A)

Life Science?Cell structureand function

?Behavior ofor ganisms ?Integrating chapter (DCI LS1.A,LS1.B) ?Biological evolution ?Molecular basisof heredity ?Integrating chapter (DCI PS3.D,LS1.C-D, LS3.A-B, LS4.A-C, LS4.E,ETS1.A) ?Matter,ener gy,andorganization inliving systems ?Interdependence ofor ganisms ?Integrating chapter (DCI LS1.C) Earth-Space Science?Origin andevolution of theuniverse ?Origin andevolution of the Earthsystem ?Integrating chapter (DCI PS1.B,PS4.A-C,

ESS1.A-C, ESS4.B)?Geochemical cycles

?Integrating chapter (DCI ESS2.A-B,ESS3.D) ?Energyin theEarth system (DCI PS3.C,ETS1.B)

Science ina Personal

and SocialPerspective;

Science &T echnology?Personal andcommunity health

?Natural andhuman-induced hazards ?Abilities oftechnological design (DCI PS4.C,LS2.A-B, LS2.C)?Population growth ?Natural resources ?

Environmental quality

(DCI

LS4.D, ESS3.C,ESS4.A,

ESS4.C, ETS2.B) ?Science andtechnology inlocal, national, andglobal challenges ?Understandings aboutscience and technology (DCI ESS3.C,ETS2.A)

Note.Codes beneathmain conceptsshow thealignment ofthe programwith theDisciplinary CoreIdeas (DCI)in theNext GenerationScience

Standardsfor physicalsciences (PS),life sciences(LS), earthand spacesciences (ESS),and engineering,technology, andapplications ofscience

(ETS). 10 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from enablestud entstoconstr uctthe targetedunderstandin gsand besuccessfulon thea ssessments.Also,inthethi rdstage, deve lopersm adeextensiveus eof conceptualflowgra phics(CFGs ).TheseCFGsarevi sualdiagramsthatil lus- tratethe flowofid easthr ough thechapter andtherelative strength ofconcep- tualconn ectionsamongthem(see Figure1).U seofCFGs throughoutth e developmentprocessre finedandstren gthenedthefocu softhe materials,in turnst rengtheningbothrigorandco herence. These processesfor developingthe materialspositioned theprogram well withthe NGSS,helping toensure coherencewithin aunit andacross grade bands.For example,the Inquiry Approachprogram supportsengag- ing studentsin thepractices ofscience inmany ways.The BSCS5E Instructional Modelthat organizes andsequencesinstructionprovides opportunities forstudents toask questionsabout phenomena,design simple experiments, useevidence collectedfrom theirclassroom experiencesto develop explanations,and communicatescientific ideasto theirpeers. Thus, engagingstudents inthe practicesof scienceis integralto thestructure of theprogram. Further, theconceptsinT able2 clearlyalign withthe Figure 1.Example ofa conceptualflow graphic(CFG). Thedark arrowsrepresent connections tothe centralconcept ofthe chapter.The lighterarrows represent connections amongideas throughthe sequenceof activities.Dashed arrows indicate weakerconnections.

Efficacy ofCurriculum withPD

11 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from Disciplinary CoreIdeas (DCI)of theNext GenerationScience Standards (NGSS) forhigh schoolscience. InT able2, codesbeneath mainconcepts show thealignment ofthe programwith theDCI inthe NGSSfor physical sciences (PS),life sciences(LS), earthand spacesciences (ESS),and engi- neering, technology,and applicationsof science(ETS). Finally,T able3 illus- trates anexample ofhow onecrosscutting conceptis wovenacross multiple units andyears ofthe program.All sevencrosscutting conceptsare woven throughout allthree yearsof theprogram. Tomake thematerials educative,we integrateda varietyof teachersup- ports thatalign withthe heuristicsof educativematerials suggestedby Davis and Krajcik(2005). Includedin thesupport materialsof An Inquiry Approachare backgroundon thephilosophy behindthe instructional model; practicalstrategies forimplementing theinstructional modelas intended; strategiesto supportmeaningful, collaborativelear ning;and strat- egies forempowering studentsto monitorand supporttheir ownlear ning (e.g., throughthe effectiveuse ofstudent notebooks).The teachermaterials also includeadditional contentbackground forteachers whofind them- selves teachingoutside theirarea ofexpertise. Thisincludes information on commonconceptions studentshave aboutspecific conceptsand how to bestaddress them(see positiverelationship betweenteacher knowledge in thisarea andstudent outcomesin Sadler, Sonnert,Coyle, Cook-Smith,& Miller,2013). Italso includesspecific ideasfor bothfor mativeand summa- tive assessmentof studentlear ning. In sum,An InquiryA pproachis ac omprehensivesetofcurricu lum materialsi ntendedtopromotethe typesof learningjust descri bed.By comprehensive, wemean thateach levelofthe programsupports teachers ands tudentsforafu llyear of highschoolscie nceco ntent witho utthe needfo rsupplem entationandisdesigned to beuse deverydayforthe entires choolyear.An InquiryAppr oachis at hree-yearprogram,typical ly usedin Gr ades9through1 1. However,becau sethede siredoutcomemea- surewas ad ministeredinthespringofstud ents'10th -gradey ear, the effi- cacy studyw aslimitedto estimatingthe program's effectsa fterjusttwo yearsof use. Thetreatmentcondi tionalsoin cludedas even-dayPDpro- gram forteach ers,providedea chyearofth ecurriculu mprogram tosu p- portimplemen tation.TheP Dprogramis describedin thefollowing section.

The ProfessionalDevel opmentProgram

In thecontext ofthe curriculum-basedPD providedin thisstudy, we translated recommendedpractices intoa PDprogram thatengaged teachers in ayear-long experiencewith aclear focuson studentlear ningand the effective implementationof theprogram. Theproviders ofthe PDprogram were alsodevelopers ofthe curriculummaterials. Inaddition, theyhad Taylor etal. 12 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from

Table 3

Examples ofHow theInquiry ApproachProgram AddressesOne ofthe SevenNext Generation Science Standards(NGSS) CrosscuttingScientific Concepts NGSSCrosscutting ConceptDisciplinary UnitsGrade 9Grade 10

Systems andsystem models

a

‘‘Defining thesystem under

study—specifying its boundaries andmaking explicit amodel ofthat system—provides toolsfor understanding andtesting ideas thatare applicable throughout scienceand engineering."" (NRC,2012) Physical ScienceStudents considerthe movementof matter intoand outof systemsduring chemical reactions.Students considerthe transferof energy within asystem duringcollisions in both microscopicand macroscopic models.

Life ScienceStudents examinecells assystems with

important matterand energy inputs and outputsand cellsas components of largerstructuresin plantand body. systems.Students simulatepopulation systems using modelsof predator-prey interactions.

Earth-Space

ScienceStudents simulatethe transferof

electromagnetic radiationinto andout of asystem usingsprings andropes to model starlightwith different frequencies andamplitudes. Students usephysical modelsand chemical reactionsto understandthat systems onEarth havereservoirs of matter,inputs andoutputs, andfluxes of matterbetween thosereservoirs.

Science and

SocietyIn severalinvestigations, students

analyze inputsand outputsof and processes withinecosystems andhow fire suppressionpolicies affectforest ecosystems.Students modelhow humanactivities modify naturalprocesses andalter the distribution ofspecies andresources in ecosystems.a

The othersix crosscuttingconcepts are(a) patterns; (b)cause andeffect; (c)scale,proportion,and quantity;(d) energy andmatter; (e)structure

and function;and (f)stability andchange. 13 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from extensive experienceproviding PDon thiscurriculum program.The research teamalso attendedeach PDsession andmonitored thePD imple- mentation, providinginput asnecessary. The goalsofPD weret omaximize impl ementationfidelityb yhelpin g teachersdeepenth eirun derstandingofthe natureofthematerialsb ymod- elinglessons,encouragi ngcol laborationaroundco mmonexperienceswith the materials,improvin gteachers'contentk nowledge,aswellas enhanc- ingth eirabilitytoimple mentth einstructional modelthatorganizesand sequencesallinstructio ninAn InquiryApproach. Theseven-day PDpro- gram eachyear wasc omposed ofathree-day summer ins tituteandfo ur one-daysessio nsthroughouttheschool year.Theextendedd ur ationof theP Dena bledustoworkwiththet eachersthrough outthe year ,particu- larlyat thebe ginningofe achnewunit,whi chfocusedon adifferent sci- encedisci pline.Theextendeddurati onal soallowedustoin troducenew featuresofthe program astea chers' und erstandingoftheprogram expandedand theircomfort levelwithpre viously introdu ce dfeatures increased.Thus, inthis study, theface-to-facePDsessions complemented the educativeaspects ofthe teacher s upport materialsa ndaimedto pro- vide teachersw iththeexperiencesneeded totake fulladvantageof research-based curriculummate rials. TeachersintheP Dp ro gramwere engagedas collaborative learnersofconte ntwiththePDf aci litatorsfrom BSCS.The PDprov id ersused(orapproximated)t hepedagogicalmetho ds suggested inthe prog ramforstudents.As teachersengagedin activities as sciencel earners,theactivitiesbecamethe commonexperie ncethat anchoredsubs equentconversationsof pedagogy.

The ComparisonCondition

During thetwo yearsof thestudy, thecomparison groupcontinued to use theirown extantcurriculum materialsand receivedthe usualPD planned bytheir schoolsand districts(i.e., businessas usual[BaU]). In Grade 9,teachers inthe comparisonschools usedone ofeight differenttext- books providedby theirschool districts.Most ofthe eighttextbooks were each usedby onlyone ortwo comparisonteachers. However, the PrenticeHall PhysicalScience andEarth Sciencetextbooks wereused by over halfof the28 comparisonteachers (16teachers inone ofthe largerpar- ticipating schooldistricts). InGrade 10,most BaUschools progressedto their standard 10th-gradebiology curriculumand textbooks.That said,a one-unit sample ofartifacts collectedby external researchersindicated thatBaU teachers usedtheir textbooksonly 24%of thetime, indicatingthat theysup- plemented thedistrict-supplied textbookwith manyother curriculum materials.Taylor etal. 14 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from

Theoryof Change

Ourthe oryofchangeis thatt hecombinati onof educativ ecurriculu mmate- rials forteachers,r esearch-based materials fors tudents,andcurriculum-based PDwill produc eapositiveeffecton bo th studentsand teachers andthatthe effectonstud entsis inpartmedi atedby positive ef fectsonteacher s'practice. More specifically,research-based studentmaterialsprov idescaffoldingfor exemplary teacherpracticewhile the educativet eachermaterialsand face-to- face PDprov idethen ecessarys upportsforteachersto enactt hatcurriculum within theiro wncontexts.

Study Description

Setting andP articipants

The studyreported heretook placein 18high schools(9 treatment,9 comparison) inthe stateof Washington. These18 schoolsinitiallyenrolled in thestudy forthe 2009-2010school year. Approximatelyhalf ofthe 18 schools werein suburbanareas. Theremaining schoolswere inrural areas. Table4 providesan overviewof studentand schoolcharacteristics forthe treatment andcomparison groups.Both groupswere somewhatdiverse in termsof studentdemographics, andeach grouphad asimilar blendof sub- urban andrural schools.Differences instudent demographicsacross groups were accountedfor inthe analyses.

Table 4

Demographics byTreatment Group

Demographics byGroup Treatment (n= 1,509)Comparison (n= 1,543)

Female747722

Special education166143

Free/reduced lunch796592

English languagelear ners9470

American Indian3818

Asian82169

Black12187

Hispanic454379

Hawaiian/Pacific Islander815

Mixed race/ethnicity1981

White787794

Number ofsuburban schools45

Number ofrural schools54

Efficacy ofCurriculum withPD

15 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from

StudyEligibility

All traditionalhigh schoolsthat hadnot usedAn InquiryAppr oachin the pastand thatparticipated inthe stateachievement testingprogram were eligiblefor thestudy. Further, fora schooltobeeligible, theprincipal agreed toencourage teacherattendance atPD sessionsand toallow access to classroomsfor datacollection.

ResearchQuestions

This studywas guidedby aprimary researchquestion relatedto theeffi- cacy ofAn InquiryAppr oachplus curriculum-basedPD aswell asa setof exploratory questionsrelated tomediation andmoderation oftreatment effects. ResearchQuestion 1:Primary analysesof treatmenteffects onstudent achieve- ment: Controllingfor covariates,what isthe maineffect oftreatment onstudent achievement? ResearchQuestion 2:Exploratory analysesof mediation:T owhat extentdoes teacher practicemediate theeffect oftreatment onstudent achievement? ResearchQuestion 3:Exploratory analysesof moderation(interactions): To what extent dostudent demographiccharacteristics moderatethe effectof treatment on students(i.e., whatare theinteraction effectsof treatmentwith student characteristics)?

Design

This studyuses apretest/posttest controlgroup design(Shadish, Cook, & Campbell,2002) wherethe meansof posttreatmentoutcome measuresare compared acrossthe treatmentand comparisongroups afterbeing con- trolled forpretreatment differencesin outcomes.The unitof randomassign- ment togroups wasthe school(or ''cluster''of students),and assuch, the design isalso oftenreferred toas acluster-randomized trial(Raudenbush,

1997). Neithermatching norblocking wasused priorto randomassignment

as thelate timingof schoolsjoining thestudy madeit impossiblefor reliable stratification levelsto beestablished.

Group Allocationand Attrition

As twoof thedevelopers ofAn InquiryAppr oachwere alsoon the research team,a numberof safeguardswere employedto limitexperimenter bias. Thefirst suchsafeguard wasusing anexter nalresearcher tomake ran- dom assignmentsto groupsusing arandom numbergenerator .The researcher didnot makethe assignmentsuntil schoolshad consentedto par- ticipate inthe study,and noschools leftthe studyafter learning oftheir group assignment.W eusedinformation inT able5todeter minehow Taylor etal. 16 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from equivalent thegroups werein scienceachievement (pretreatment). Specifically, wecompared treatmentand comparisongroup meanson the

8th-grade sciencebaseline covariatefor thebaseline andanalytic samples,

respectively. Thebaseline samplewas theset ofstudents inschools who were randomlyassigned togroups atthe onsetof thestudy. Theanalytic sample wasthe setof studentsin schoolsrandomly assignedat theonset of thestudy forwhich aposttreatment (Grade10) scienceoutcome measure was available.By comparingthe baselineand analyticsample sizeswithin Table5, thedegree ofoutcome measureattrition isapparent. Theoverall attrition rateof individualstudents inthe study,based onavailability of the 10th-gradescience outcomescore, is18%, withan attritionrate of18% in thetreatment groupand 17%in thecomparison group.Thus, thediffer- ential attritionrate acrossgroups is1%. ConsultingT able5, itis clearthat the randomassignment processwas notcompletely successfulin distributing baseline achievementlevels evenlyacross groups.The baselinedifference in the 8th-gradestate scienceachievement scoresacross groupswas notewor- thy (Hedges'g= .23)and infavor ofthe comparisongroup. Thisbaseline difference wasaccounted forin thetreatment effectmodels describedlater in thisarticle. Finally,we notehere thatthe primaryanalysis forthe main effect oftreatment isan intent-to-treatanalysis. Thus,all studentsretained their originaltreatment groupassignment inthe analyticsample (i.e.,stu- dents crossinggroups duringthe interventionwere treatedin theanalysis as ifthey remainedin theiroriginal treatmentgroup forthe fulltwo years).

Measures

In thissection, wedescribe thetwo measuresused inthe analysis.The first isthe outcomemeasure usedto estimatethe maineffect oftreatment on student achievement.The secondis ameasure ofclassroom practiceand cul- ture thatwas usedin themediation analysis.

The OutcomeMeasur e

In thisstud y,wesought ano utcomemeasure withfo urkeyfeatures:(a) importanceto alls takeho lders,includingteachers,students,parents,an dadmin- istrators;(b)al ignm entwiththestudentabilitie sandun derstandi ngsthatthe InquiryApproac hcurriculummaterial sseektoimprove;(c) afairoutco memea- sure fort hecomparisongroup; and (d)strong psychometricproperties. Forthis study,theo utcomeme asurec hosenwastheWashi ngtonstatesciencea ssess- ment(the Hi ghSchoolPro ficiency Exam;HSPE),whichclearlymeetsthef irst criterion.Asfort hesecon dcriterion,th eHSPE hasreasonable alignmentwith theInquiryApproac hprogram dueto itsb roadcov erageofscien ceconten t (earth/space, physical,a ndlife sciences)asw ellasi tsfocuso nscienceprac tices (e.g.,develo pingquestionsandde signinginvestigations,ev idenceasth ebasis fore xplanationsandmodels,an dcomm unicatingscientif icresults).WeEfficacy ofCurriculum withPD 17 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from

Table 5

Pretreatment SampleSizes andCharacteristics forthe Baselineand AnalyticSamples ina Two-LevelCluster-Randomized Trial

TreatmentGroup ComparisonGroup

Sample SizeSample CharacteristicsSample SizeSample Characteristics

Students/

SchoolsIndividual/School

MeanStandard

DeviationStudents/

SchoolsIndividual/School

MeanStandard

Deviation

Baseline sample

Individual baselinemeasure

Eighth-grade scienceachievement test1,845 385.9627.816 1,868391.85 27.544

School baselinemeasure

Eighth-grade scienceachievement test9 386.159.27 9388.55 8.73

Analytic sample

Individual baselinemeasure

Eighth-grade scienceachievement test1,509 388.8427.379 1,543394.92 26.549

School baselinemeasure

Eighth-grade scienceachievement test9 388.218.76 9390.58 10.21 18 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from concludedthat thethirdc riterionwoul db emetbytheH SPEasthis test isbase d on Washingtonstatestandardst owhich allschoo ls(treatment andcomparison) are heldaccountabl e.ThepsychometricpropertiesoftheH SPE are strong.For the 10th-gradescienceoutcome measure,therepo rte dC ronb ach'salphais.87. Fort hebaselin eachievementcovariates - the stateachievementtestsco resfor

7th-gradewriting,8 th-grademath,and8t h-gradescience - theal phaval ues

are .77,.90, and.89 ,res pectively(Edu cationTestingService,2012).

The TeacherPractice Measure

Weuse dtheR eformed TeachingObserv ationProtocol(RTOP;Pibu rn et al.,200 0;Sawadaet al., 2002)asthe primarymeasu reof te ach erpractice. Fromth ispointforward, allrefere ncesto''teach erpracti ce''shouldberead as ''teacherpracticeasindicatedbytheRTOP.''TheRTOPinstrumentmeasures thee xtenttowhichsc iencean dmathematic steachinga lignswi ththerecom- mendationsforresearch-bas ed instructionalreformdescri bedinnationalscience andm athematicsstandardsd ocumentsofthelate 1990s.Thein strume ntisma de up of25 Likert-typei tems,d ividedintofiv esubscale s:(a)Le ssonD esign& Implementation,(b)C onte nt - PropositionalKnowledge,(c)Content - ProceduralKn owledge,(d)ClassroomCu lture - CommunicativeI nteractions, and (e)Cl assroomCulture - Student-Teacher Relationships.Atotalscoreacross all itemsisalso calculated.E achs calevaries fromascore of0,behavior never occurred,toascoreof4,pervasive orextr emelydescriptiveofthelesson .As a whole,theprot ocoladd ressesteacheratte ntiontostudents' priorknowledg e, studenteng agementinalearning community,a ndthe extenttowhichteachers supportana tmosphere ofproblemsolvingand student-gene ratedideas. Validationstudieso ftheRTOPsuggestthat itcanh avestrongpsycho metric properties.Thereliabi lityes timate(R 2 ) forthee nt ireinstrumentis.95 4 (Piburne tal.,200 0). Toguard agai nstexperimenterbias,twoexter nalresearche rs conductedt he classroomobserva tions.Tocalibratethe observer sbefor ethesitevisi ts,thetwo observers watchedc lassroominstruction fromvideo recordings.A ftereach video,th eobservers thenindependen tlyprovid edratingsacrossthe25R TOP items. Afterthi s''pre-discussion''rating,th etwoobserversdi scussed theirratin gs and thebasisf orthe irratin gs. Classroom observationswere madethroughout theyear .Nearly all teachers inthis studywere observedeight times(approximately onceeach month). Wechosethis comprehensiveapproach toincrease thelikelihood that theaverage RTOP scoreforeachteacher wasrepresentative ofhis or her typicalpractice. Inthe analyses,the outcomemeasure forteachers is their meanR TOPscoreacrosstheir observations.The external observers of teacherpractice werenever toldof teachers'treatment groupassignments. However,it likelybecame discernible asthe studentsusedtheirInquiry Approachtextbooks thatare designedto beused mostdays, ifnot everyEfficacy ofCurriculum withPD 19 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from day, inclass. Therefore,we acknowledgethat wecannot ruleout thepossi- bility ofobserver biasin theratings ofteacher practice(R TOPscores). Becausetwoex ternalre searcherseach scoredonehalfofthetreatmen t andco mparisonclassrooms, interraterreliabilitywas calculatedtotestf orcon- sistencyinscorin gbe tweenraters.A sampleof7.4%of theobservatio nsw as scoredbyboth rat ers(29 outofatotal of394), andinte rra ter reliabilitywas calculatedusingthe intraclassco rrelati oncoefficient(ICC) .Analysisofthe commonlyscored observationsyiel dedanintraclasscorr elationcoeffici ent of .966(to talRTOPscores, two-way mixedeffectsmo del,absolutea greement, averagemeasur es).Interpretationof theintraclasscorrelationco eff icientis similartothat ofCo hen'sk app a,whereacomm onlyusedruleofthum bis that.40 to.59 represe ntsmode rateinterraterreliab ility,.60to.79is substantial, andgr eaterthan.80 isoutstandi ng(Landi s& Koch,1977). The sameexter nalresearchersalsoused aFidelity ofImplementation Observation Protocol(BSCS, 2009)to assesstreatment teachers'use ofthe instructional materialswith students.In particular, thetool examinedthe quality ofteachers' useof theBSCS 5EInstructional Model.The average score onthis protocolacross 183independent observationsof treatment teachers was2.13 ona 3-pointscale (71%),indicating overallprogram use consistent withthe developers'intent.

Analysisand Findings

ConfirmatoryAnalyses:Main Eff ectof TreatmentonStudents The primarypurposeofthe st udywasto addressResearchQ uestion1: Controllingforcovariates,w hatist hemaineffect oftreatmento nstude nt achievement?Becau seassignme nttotreatmentor comparison (BaU)conditio ns occurredat thescho olleve lwhiletheoutcomeof interestoccursat thestudent level,wechosem ultil evelmod elingtoestimatetheeffects ofthetreatmento n studenta chievement.Preliminaryanalysesco nfirmedthi smod elingchoiceas theeffe ctofclus teri ngissizeable(uncondi tion alICC=.13),a ndthedata meetthe mu ltilevelmodelingassumptions ofhomogeneityofLe vel1variance (x 2 = 22.87,p= .153)andn ormalityof residuals(Q-Qplotsof residu alsat both levelshave pattern sthataregenerally linear). Tore finetheestim ateofth etreatmenteffect, wein cluded inthemodel a setofco vari atesthatwehypothe sizedt obecorr elatedwiththeoutcome measure:students 'scoresonthe10th-g radestatesc ienc easses sment (SCI10).Thissetof covariat esinclud edbo thdemographicandac hievement variables.Somestuden t-level (Level1)covariateswer ealsousedina ggre gate atth eschool level(Level 2).TheLeve l1covaria tesincl udedach ievement scoressuch asthosefr omth e8th-gradestat escienc eassess ment(SCI8),the

8th-gradestatema thassessmen t(MAT8 ),andthe7th-gradesta tewriting

assessment(WRIT7). TheLevel1demogra phiccov ariatesin clud edfreeandTaylor etal. 20 by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from reduced-pricelunchstat us(FRL)a saproxyforso cioeconomicsta tus,gend er (GEND),Engli shlanguagelearner status(ELL), specialeduc ationstatus (SPED),grade level(GRAD E),andas etofracecontrastcode sthat incl ude AmericanIndian (AMIND),Asian (ASIAN),Black(BLK ),Hispanic/ Latino (HISP),Hawaii /PacificIslander(HPI),and thosewhoindica tedtwoormore ethnicities(MIX),ea chofwhichallow sachie veme ntcomparis onsbetween these lectedgroupofst udentsand the referencegroupof Whitest udents. The Level2 modelincluded thetreatment variable(TREA T)as wellas a schoolmean aggregatefor theeighth-grade scienceassessment score (MNSCI8), aschool meanaggregate forthe eighth-grademath assessment score (MNMAT8),anda schoolmean aggregatefor theFRL status (MNFRL). Becausewe hadlarge numbersof Level1 units(students) inthe sample buta relativelysmall numberof Level2 units(just 18schools), we were muchmore judiciousin includingLevel 2covariates, includingonly the mosttheoretically influentialon theoutcome, aseach ofthese consumes a degreeof freedomin theLevel 2statistical significancetests. We grand- mean centeredall independentvariables tofacilitate thedesired covariate adjustment. Themain effectmodel wasspecified andrun asdescribed next usingST ATA12statisticalsoftware.

Level 1:

SCI10 ij 5p 0j 1p 1j

SCI8ðÞ

ij 1p 2j

MAT8ðÞ

ij 1p 3j

WRIT7ðÞ

ij 1p 4j

FRLðÞ

ij 1p 5j

GENDðÞ

ij 1p 6j

ELLðÞ

ij 1p 7j

SPEDðÞ

ij 1p 8j

GRADEðÞ

ij 1p 9j

AMINDðÞ

ij 1p 10j

ASIANðÞ

ij 1p 11j

BLKðÞ

ij 1p 12j

HISPðÞ

ij 1p 13jk

HPIðÞ

ij 1p 14j

MIXðÞ

ij 1e ij :

Level 2:

p 0j 5b 00 1b 01

TREATðÞ

j 1b 02

MNSCI8ðÞ

j 1b 03

MNMAT8ðÞ

j 1b 04

MNFRLðÞ

j 1r 0j : Descriptive statisticsfor theoutcome variable(SCI10) areprovided in Table6. Themain effectestimates fromthe multilevelmodel areprovided in Table7. This outputsuggests thatthe treatmentgroup studentswould have scored anestimated 3.68scale scorepoints higher, onaverage, thanstudents in thecomparison grouphad thegroups beenfully equivalentprior totreat- ment. Thisdifference ( b 01 ) isstatistically significantat thea= .05signifi- cance level( p= .035). Like manyefficacy trials,this studywas subjectto attritionand aresulting loss ofdata. Asa result,the researchteam replicatedthis treatmenteffect analysis afterimputing themissing datausing amultiple imputationalgo- rithm withinST ATA12.Specifically,withinSTA TA, weused theEM Algorithm withmultiple imputationsto addressmissing data.The treatmentEfficacy ofCurriculum withPD 21
by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from effect estimatefrom theidentical modelapplied tothe imputeddata sets yielded verysimilar results( b 01 = 3.37,SE= 1.65,p= .041),suggesting that themissing datadid notintroduce asystematic biasin thetreatment effect estimate. As anadditional wayto interpretthe treatmenteffect, theresearch team also computedthe effectsize. TheHedge's geffect size(with smallsample size adjustmentv),ameasure ofpractical significance,was computedby using thetreatment effectcoefficient ( b 01 ) fromthe multilevelmodel as

Table 6

Posttreatment Outcomesfor theAnalytic Sampleand Estimated

Effects ina Two-LevelCluster-Randomized Trial

TreatmentGroup ComparisonGroup Estimated Effects

Outcome

MeasureCovariate-

Adjusted

MeanUnadjusted

Standard

DeviationCovariate-

Adjusted

MeanUnadjusted

Standard

DeviationCovariate-

Adjusted

Mean DifferencepValue

10th grade

science384.52 43.305380.84 39.1193.68 .035

Table 7

Estimates ofFixed Effectson 10th-GradeScience AchievementScore

Independent

VariableLevel CoefficientStandard

ErrorzValuepValue95% Confidence

Interval

SCI8 Student0.71 0.0326.14 \.001 0.650.76

MAT8Student 0.320.02 16.13\.001 0.280.35

WRIT7 Student0.80 0.322.50 .0120.17 1.43

FRL Student-2.61 1.01-2.59 .010-4.58 -0.63

GEND Student-5.41 0.90-6.02 \.001 -7.17-3.65

ELL Student-8.86 2.20-4.02 \.001 -13.17-4.54

SPED Student-4.70 1.62-2.90 .004-7.88 -1.52

GRADE Student-11.96 2.54-4.72 \.001 -16.93-6.99

RACE-AMIND Student-6.92 3.28-2.11 .035-13.36 -0.48

RACE-ASIAN Student-1.70 1.70-1.00 .317-5.05 1.64

RACE-BLK Student-5.31 1.82-2.92 .004-8.87 -1.74

RACE-HISP Student-5.38 1.29-4.18 \.001 -7.90-2.86

RACE-HPI Student-10.25 5.36-1.91 .056-20.76 0.26

RACE-MIX Student-3.92 2.47-1.59 .113-8.75 0.92

TREATSchool 3.681.75 2.11.035 0.257.10

MNSCI8 School0.34 0.341.00 .318-0.32 1.00

MNMAT8School 0.080.23 0.34.734 -0.380.54

MNFRL School-2.89 7.89-0.37 .715-18.34 12.57

Taylor etal.

22
by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from the covariate-adjustedmean differenceacross groups(the numerator).The denominator wasthe pooledstandard deviationweighted forsample size differences acrossgroups: g5 vb 01   ni1ðÞs 2i

1nc1ðÞs

2c ni1nc2 q: The Hedge'sgvalue forthe treatmenteffect was.09 standarddeviations. The

95% confidenceinterval forthe effectsize is[.01, .17].The initialpower anal-

ysis wasconducted usingOptimal DesignPlus EmpiricalEvidence (v.3)soft- ware (Raudenbushet al.,2011). Inthis analysis,we assumedan uncondi- tional ICCof .15,a Level2 covariatecorrelation ( R 2 value) of.50, 18 schools, and150 studentsper school.These valuescorresponded toa mini- mum detectableeffect sizeof d= .40.In actuality,the powerwas signifi- cantly betterthan weanticipated, allowingus todetect amuch smallereffect than expected.First, theactual ICCwas lowerthan ourinitial estimate (unconditional ICC= .13),and theaverage numberof studentsper school was higher( n= 170).In addition,the analysisincluded covariatesat both Level 1and Level2 ofthe model,and theLevel 2covariates providedus with muchbetter precisionthan weanticipated ( R 2L2

50:91). Asa result,we

had sufficientpower inour studyto detecta smallereffect sizethan weini- tially expected. The WWCwould characterizean effectsize of.09 asa ''statisticallysig- nificant positiveeffect.'' Althoughthe WWCreserves thecharacterization of ''substantively important''for effectslarger than.25, thiseffect ismeaningful in thecontext ofthe smalleffect sizesoften observedin highschool inter- ventions. Forexample, Hill,Bloom, Black,and Lipsey(2007) conducted a synthesisof effectsizes forrandomized controltrials andfound thatfor ele- mentary schoolstudies, theaverage effectsize was.33, andfor highschool studies, theaverage effectsize was.27. However, thishigh schoolaverage included effectsizes computedon outcomemeasures thatwere proximal or targetedtothe intervention.Based onwhat Hilland colleaguesobserved in theelementary schoolstudies, inclusionof proximaleffect sizeslikely inflated theaverage effectsize forhigh schoolinterventions. Specifically, they foundthat theaverage effectsize variedby thebreadth offocus for the outcomemeasure, reportingthat ''withinstudies ofelementary schools, mean effectsizes arehighest forspecialized tests(0.44), next-highestfor nar- rowly focusedstandardized tests(0.23), andlowest forbroadly focusedstan- dardized tests(0.07)'' (p.8). Giventhat theeffect sizereported forthis study (.09) wascomputed usingscores froma broadlyfocused standardizedtest (HSPE 10Science) andis associatedwith ahigh schoolintervention, we find theeffect sizeto bewithin expectation.Efficacy ofCurriculum withPD 23
by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from Another wayto interpretthis effectsize isto compareit tonor mative expectations forachievement growth(i.e., averagepre-post yeareffect sizes for 9th-and 10th-gradescience students).This effectsize expressesstudents' expected gainin scienceachievement overthe courseof oneyear .Looking across aset ofnationally normed tests,Bloom andcolleagues(2008)esti- mated thatthe averagepre-post yeareffect sizefor sciencein 9thgrade is .19 and.22 for10th grade.Thus, thetwo-year expectedgain inachievement can beestimated as.41 standarddeviations. Theeffect sizeof .09detected in this two-yearintervention studyis noteworthyas itcorresponds to.09/.41 or

22% ofthe two-yearexpected gain.Multiplying .22by 18school monthsfor

a two-yearintervention, weestimate thattreatment groupstudents emerge from thestudy (i.e.,start 11thgrade) nearlyfour monthsahead ofcompar- ison groupstudents inscience achievement. As af inalwayto express thepracticalimportance ofthetreatment effect, we convertedtheef fectsizeinto animprovementindex using thepro pertie s of thenorm aldistribution.I nanormaldistribution ,a1.0 SDeffect sizeis equiv- alentt o34p ercentilepo ints.Therefore,an effectsi zeof.09equatesto an improvementind exof3.06(3 43.09) percentilepoints. So, ifthecomparison studentsw ereattheme anofthe norm edsample, the50thp ercentile ,thetreat- ment groupstud entswouldthe nbepla cedatjustove rthe5 3rdpercentile. There aretwo keyreasons whythese estimatesof thetrue treatment effect arelikely conservative.First, wejoin otherresearchers whohave noted thatusing newinterventions thatrequire unfamiliarpractices can often leadto an''implementation dip,''where useof theprogram features is mechanisticand canresult ina negativeeffect onoutcomes forsome time priorto ultimatelyimproving outcomes(e.g., Fullan,2001; Hall& Hord, 2001).An InquiryAppr oachencourages teachersto useinstructional practices thataren otcomm onplac einhighschools(Banilow eret al.,2013 ),and as such,t hesepracticeswerel ikelyu nfamiliart oamajorityo ftreatmentteacher s. Thisim plementationdip,ifafacto rint hisstu dy, wouldreduce thesize ofthe treatment effect.Incontras t,manycomparis onteacherswer eus ingprograms or setsof activities thattheyusedroutin elyprio rtotheresearch.Se cond ,obser- vationsindicate thatthe learningexperien cesof studentsinthecomparis on groupi ncludedsomeresearch-basedp ractices similarto thosepromotedin thetreatmentgroup.Thiswouldalsotendtoreducethetreatmenteffect. Other maineffects inT able7 areinterestingaswell. Theachievement covariates wereall highlypredictive ofthe SCI10outcome measure.In gen- eral, therace covariateshave statisticallysignificant ( a= .05)main effectson achievement withthe exceptionof therace dummycodes thatcompare the achievement ofAsian, Hawaiian/PacificIslander ,and mixedethnicitystu- dents toWhite students,respectively. Inthis sample,males hadhigher mean scoresthan females,economically advantagedstudents hadhigher Taylor etal. 24
by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from mean scoresthan economicallydisadvantaged students,native language English speakershad highermean scoresthan Englishlanguage learners, and studentswithout aspecial educationdesignation hadhigher mean scores thanthose withsuch adesignation. Ina latersection, wereport whether therelationships betweenkey demographicvariables andachieve- ment describedhere differby treatmentgroup. ExploratoryAnalys es:MediationandIndirect Treatment Effects The purposeof themediation analysiswas toaddress Research Question 2:T owhatextentdoes teacherpractice mediatethe effectof treat- ment onstudent achievement?The researchteam hypothesizedthat the nature ofteachers' practiceis criticalto theefficacy ofthe curriculummate- rials andthat improvingteacher practiceis partof themechanism bywhich the causaleffect ofthe treatmentis realized.This issupported bysyntheses of interventionstudies suchas thatconducted byNye, Konstantopoulos,and Hedges (2004),who observedthat theproportion ofvariance instudent out- comes attributablesolely tobetween teachervariance canbe asmuch as

20%. Asa result,this efficacystudy soughtto usethe RTOP tocollect com-

prehensive dataabout teacherpractices totest whetherthe treatment(curric- ulum materialsplus PD)has anindirecteffect onstudents' scienceachieve- ment viateacher practiceas amediating variable(see Figure2). In Figure2, patharepresents theeffect ofthe treatmenton teacherprac- tice (RTOP);pathbrepresents theeffect ofteacher practice(the mediator) on thescience achievementoutcome (SCI10),controlling forthe treatment; and pathc#represents theeffect ofthe treatmenton thescience achieve- ment outcome(SCI10), controllingfor teacherpractice. Thatis, c#is the direct (unmediated)effect oftreatment. Theproduct ofpaths aandbis often used torepresent themediating (indirecteffect) ofthe treatmenton theout- come (MacKinnon,2008). The mediationdesign forthis studyis oftenreferred toas a3 ?!2?!1 design becausethe treatmentis atthe thirdlevel (school),the mediatoris measured atthe secondlevel (teacher),and theoutcome ismeasured at the firstlevel (student).W etested mediationinthis3 ?!2?!1 design using amodeling approachadvocated byleading methodologists (MacKinnon, 2008;Pituch, Murphy,& Tate, 2010).In thisapproach,separate equations forthe mediatorand outcomecan beused toestimate theindirect effect. Thefirst setof equationsin thefollowing estimatespath a. The teacher-level equationfor themediator is RTOP ij 5p 0j 1r 0ij ;

Efficacy ofCurriculum withPD

25
by Joseph Taylor on May 6, 2015http://aerj.aera.netDownloaded from whereRTOP ij is theteacher-level mediator, p 0j is theR TOPmeanforschool j, andr 0ij is theteacher-level randomeffect. Theschool-level equationis p 0j 5b 00 1b 01 TREAT j 1u 0j ; whereb 01 is theeffect ofthe treatmenton theR TOPscores (pathaof Figure

2), andu

0j is theschool-level randomeffect. Themain effectof treatmenton teacher practiceestimated fromthese modelsis b 01 = 16.74(

SE= 3.11,p\

.001), correspondingto rawgroup meansand standarddeviations of71.4 (10.1) and55.0 (7.6)for treatmentand BaU,respectively. Thisis alarge effect with correspondingHedge's gvalue =1.85. Estimatingthe bandc#paths requires athree-level model.The student-levelequation forthe outcomeis SCI10 ijk 5p 0jk 1e ijk ; wherep 0jk represents theoutcome meanfor teacherjof schoolk,ande ijk is the student-levelrandom effect.The teacher-levelequation addsthe media- tor (RTOP)asa predictor: p 0jk 5b 00k 1b 01k

RTOPðÞ

jk 1r 0jk ; whereb 01k represents thewithin-school impactof RTOP onthe meanSCI10 score. Theschool-level equationsare b 00k 5g 000 1g 001

TREATðÞ

k 1u 00k andb 01k 5g 010 ; whereg 001 is pathc#of Figure2 andg 010 is thefixed effectof RTOP on SCI10 (controllingfor treatment),or pathb. Thefixed effectsfrom this three-level modelare c#(g 001 ) =1.56 (SE= 2.21,p= .49)and b(g 010 )=

0.13 (

SE= 0.07,p= .07).The presenceof astrong treatmenteffect onthe teacher-practice mediator,anearly significantassociation betweenteacher practice andstudent achievementand asmall, remainingdirect treatment effect ( c#), isconsistent withour mediationalhypothesis. Amore formal test isdescribed inthe following. The indirecteffect ofteacher practicecan beestimated asthe productof theaandbpaths orthe abproduct. Thisproduct is( g 01 )( g 010 )= (16.74)(0.13) =2.18. The95% confidenceinterval forthe indirecteffect was computedusing thePRODCLIN Program(MacKinnon, Fairchild,& Fritz, 2007),yielding [-0.12,4.84] anda correspondingprobability oftype

1 error(

p= .064).Although mediationis sometimesconsidered tobe presentTaylor eta