[PDF] Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion

View - Download Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion

Previous PDF

Next PDF

forAdaptiveMotion Deblurring

MaitreyaSuin

?KuldeepPurohit?A.N.Rajagopalan

IndianInstituteof TechnologyMadras, India

Abstract

Thispapertac klesthepr oblemofmotiondeblurringof

dynamicscenes.Although end-to-endfullycon volutional designshaver ecentlyadvancedthe state-of-the-artinnon- uniformmotiondeblurri ng, theirperformance-complexity trade-offisstillsub-optimal.Existingapproac hesachie ve alarg ereceptivefieldbyincreasingthenumber ofgeneric convolutionlayersandk ernel-size,butthiscomes atthee x- penseofof theincrease inmodelsize andinference speed. Inthiswork, wepr oposeanef ficientpixeladaptive and featureattentivedesignforhandling larg eblurvariations acrossdifferentspatial locationsandprocesseachtestim- ageadaptively.Wealsopr oposeaneffectivecontent-aware global-localfiltering modulethatsignificantlyimproves performancebyconsidering notonlyglobal dependencies butalsobydynamicallye xploitingneighboringpixel infor- mation.We useapatch-hierarc hicalattentivear chitecture composedofthe abovemodule thatimplicitlydisco versthe spatialvariationsin theblurpr esentinthe inputimag eand inturn,performs localandglobal modulationofinterme- diatefeatures. Extensivequalitativeandquantitativecom- parisonswithprior artondeblurring benchmarksdemon- stratethatourdesignof ferssignificant improvements over thestate-of-the-artin accuracyas wellasspeed.

1.Introduction

Motion-blurredimagesform duetorelati vemotion dur- ingsensore xposureandare favoredbyphotographers and artistsinman ycasesfor aestheticpurpose,butseldomby computervisionresearchers, asmany standardvisiontools includingdetectors,track ers,andfeature extractorsstruggle todealwith blur.Blind motiondeblurringis anill-posed problemthataims torecov erasharp imagefroma given imagedegraded duetomotion-inducedsmearingofte x- tureandhigh-frequenc ydetails. Duetoitsdiverse applica- tionsinsurv eillance,remotesensing, andcamerasmounted ?Equalcontribution.

10-1100101

Runtime for an HD image (seconds)

28.52929.53030.53131.532

PSNR (dB)

Nah CVPR17

Kupyn CVPR18

Tao CVPR18

Ours(a)

Zhang CVPR18

Zhang CVPR19

Gao CVPR19

Kypyn ICCV2019

Ours(b)

Figure1.Comparison ofdifferent methodsinterms ofaccuracy andinferencetime. Ourapproachoutperforms allprevious meth- ods. onhand-heldand vehicle-mountedcameras, deblurringhas gatheredsubstantialattentionfromcomputer visionandim- ageprocessingcommunities inthepast twodecades. Majorityoftraditional deblurringapproachesare based onvariational model,whosekeycomponent isthere gular- izationterm.The restorationqualitydepends ontheselec- etersinv olvinghighlynon-convexoptimizationsetups[14]. achallengingcomputer visionproblemas blursarisefrom varioussourcesincludingmoving objects,camerashak e anddepthv ariations,causingdif ferentpixelstocapturedif- whilegeneralizingacross differenttypes ofreal-world ex- amples,whereblur isfar morecomplex thanmodeled[3].

Recentworks basedondeepconvolutional neuralnet-

works(CNN)have studiedthebenefits ofreplacingthe imageformationmodel withaparametric modelthatcan betrainedto emulatethenon-linear relationshipbetween blurred-sharpimagepairs .Suchw orks[13]directlyregress todeblurredimage intensitiesando vercomethe limited representativecapabilityofvariationalmethodsin describ- ingdynamicscenes. Thesemethodscan handlecombined effectsofcameramotionand dynamicobjectmotion and 1 3606

achievestate-of-the-artresultsonsingleimagedeblurring task.They havereacheda respectablereductioninmodelsize,but stilllackinaccuracyand arenotreal-time.

ExistingCNN-basedmethods have twomajor limita-

tions:a)W eightsofthe CNNarefixedandspatially in- variantwhichmaynotbe optimalfordif ferentpixels ina dynamicallyblurredscene (e.g.,sky vs.moving carpix- els).Thisissue isgenerallytackled bylearninga highly non-linearmappingby stackingalar genumberof filters. Butthisdrastically increasesthecomputational costand memoryconsumption.b) Ageometricallyuniform recep- tivefieldissub-optimalforthetask ofdeblurring.Lar ge imageregions tendtobeusedtoincrease thereceptiv efield eventhoughtheblurissmall.This inevitablyleads toa networkwithalarge numberoflayers andahigh compu- tationfootprintwhich slowsdo wnthecon vergenceof the network. tivefieldandtheaccuracyof anetwork isanon-tri vialtask (seeFig.1). Ourwork focusesonthe designofef ficientand interpretablefilteringmodules thatoffer abetteraccurac y- speedtrade-off ascomparedtosimplecascadeof convolu- tionallayers.W einv estigatemotion-dependentadaptability withinaCNN todirectlyaddress thechallengesi nsingle imagedeblurring.Since motionbluris inherentlydirec- tionalanddif ferentforeach imageinstance,adeblurring networkcanbenefitfromadapting totheblur presentin eachinputtest image.We deploycontent-a waremodules whichadjustthe filtertobe appliedandthe receptive field ateachpix el.Ouranalysis showsthatthe benefitsofthese dynamicmodules forthedeblurringtaskaretw o-fold:i) Cascadeofsuch layersprovides alarge anddynamically adaptivereceptivefield.Directionalnature ofblurrequires adirectionalrecepti vefield, whichanormalCNNcannot achievewithinasmallnumberoflayers. ii)Itef ficiently enablesspatiallyv aryingrestoration,since changesinfilters andfeaturesoccur accordingtothe blurinthe localregion. Noprevious workhasinv estigatedincorporatingaw areness ofblur-v ariationwithinanend-to-endsingleimagedeblur- ringmodel.

Followingthestateoftheartin deblurring,weadopt a

multi-patchhierarchicaldesign todirectlyestimate there- storedsharpimage. Insteadofcascading alongthedepth, weintroducecontent-a warefeature andfiltertransforma- tioncapability throughaglobal-localattentive moduleand residualattentionacross layerstoimpro veperformance. Thesemoduleslearn toexploit thesimilarityin themotion betweendifferent pixelswithinanimageand arealsosen- sitivetoposition-specificlocalcontext.

Theefficienc yofourarchitectureisdemonstrated

throughacomprehensi vee valuationontwobenchmarks andcomparisonswith thestate-of-the-art deblurringap- proaches.Ourmodel achieves superiorperformancewhile Figure2.Ov erallarchitectureof ourproposednetwork.CAblock representscrossattention betweendiff erentlev elsofencoder - decoderanddif ferentlev els.Alltheresblockcontains onecon- tentaw areprocessingmodule.Symbol"+"denoteselementwise summation. beingcomputationallymore efficient.The majorcontribu- tionsofthis workare:

•Weproposean efficientdeblurringdesignbuiltonnew convolutionalmodulesthatlearnthetransformationof featuresusingglobal attentionandadapti velocal fil-ters.We showthatthesetwo branchescomplementeachotherand resultinsuperior deblurringperfor- mance.Moreov er,theefficientdesignofattention-moduleenablesus touseit throughoutthenetw orkwithouttheneed forexplicit downsampling.

•Wefurtherdemonstratetheef ficacyof learningcross-attentionbetweenencode-decoder aswellas differentlevelsinourdesign.

•Weprovideextensi veanalysisandev aluationsondy-namicscenedeblurring benchmarks,demonstratingthatourapproach yieldsstate-of-the-artresults while

being3×fasterthanthenearestcompetitor [26].

2.Proposed Architecture

Todate,thedriving forcebehindperformance improve-

ersandlar gerfilterswhich assistinincreasingthe"static" receptivefieldandthegeneralizationcapabilityof aCNN. networkperformancedoesnotal waysscale withnetwork depth,asthe effectiv ereceptiv efieldofdeepCNNsismuch smallerthanthe theoreticalvalue (investig atedin[12]). Weclaimthatasuperior alternative isadynamic frame- workwhereinthefilteringand thereceptiv efieldchange 3607

acrossspatiallocations andalsoacross differentinput im-ages.Oure xperimentsshow thatthisapproachisacon- siderablybetterchoice duetoits task-specificefficac yandutilityforcomputationally limitedenvironments. Itdeliv ersconsistentperformanceacross diverse magnitudesofblur .

Althoughprevious multi-scaleandscale-recurrentmeth- odshav eshowngoodperformanceinremovingnon- uniformblur, theysufferfrome xpensiveinferencetime andperformancebottleneck whilesimplyincreasing model depth.Instead,inspired by[26], weadoptmulti-patch hi- erarchicalstructureas ourbase-model,which comparedto multi-scaleapproachhas theaddedadv antageofresidual- likearchitecturethatleadsto efficientlearning andfaster processingspeed.The overall architectureofour proposed networkisshownin Fig.2.W edividethenetwork into3 levelsinsteadof4asdescribedin [26].We foundthatthe relativeperformancegainduetothe inclusionofle vel4is negligiblecomparedtotheincrease ininferencetime and numberofparameters. Atthebottom level inputslicedinto

4non-ov erlappingpatchesforprocessing,andaswegrad-

uallymov etowardshigherlevels,the numberofpatches decreaseandlo werlev elfeaturesareadaptively fusedusing attentionmoduleas shownin Fig.2.The outputoflevel1 isthefinal deblurredimage.Note thatunlike [26],wealso avoidcascadingofournetworkalong depth,asthat addsse- verecomputationalburden.Instead, weadvocate theuseof performanceimprov ementsovereventhedeepest stacked versionsoforiginalDMPHN[26]. Majorchangesincorpo- ratedinour designaredescribed next.

Eachlev elofournetworkconsistsofanencoder anda

decoder.Boththeencoderand thedecoderaremadeofstan- dardconv olutionallayerandresidualblockswhereeachof theseresidualblocks contains1con volutionlayer followed byacontent-a wareprocessing moduleandanotherconvo- lutionallayer. Thecontent-awareprocessingmodule com- prisestwo branchesforglobalandlocalle velfeature pro- cessingwhichare dynamicallyfused attheend. Theresid- ualblocksof decoderandencoder areidenticale xceptfor theuseof crossattentionin decode r.W ehav ealsodesigned cross-levelattentionforeffective propagationof lowerle vel featuresthroughoutthe network.W ebegin withdescrib- ingcontent-aw areprocessingmodule,thenproceedtowards thedetaileddescription ofthetw obranchesand finallyhow thesebranchesare adaptively fusedatthe end.

3.Content-Awar eProcessingModule

Incontrastto high-level problemssuchas classification anddetection[22], whichcanobtain largerecepti vefield bysuccessiv elydown-samplingthefeaturemapwithpool- ingorstrided convolution, restorationtaskslik edeblurring needfinerpix eldetailsthat cannotbeachieved fromhighly

downsampledfeatures.Mostofthe previousdeblurring ap-proachesusesstandard convolutional layersforlocal fil-teringandstack thoselayerst ogethertoincrease there-ceptivefield.[1]usesself-attentionandstandard convo- lutiononparallel branchandsho wsthatbest resultsareobtainedwhenboth featuresarecombined togethercom-paredtousing eachfeatureseparately .Inspiredby thisapproach,wedesign acontent-aw are“global-local"pro- cessingmodulewhich dependingonthe input,deploys twoparallelbranchesto fuseglobaland localfeatures.The “global"branchis madeofattention module.For decoder,thisincludesboth selfandcross-encoder -decoderattentionwhereasforencoder onlyself-attentionis used.For localbranchwedesign apixel-dependent filteringmodulewhich determinestheweight andthelocal neighbourhoodtoap- plythefilter adaptively .We describeindetailthesetwobranchesandtheir adaptive fusionstrategy inthefollowing

sections.3.1.Attention [21]innatural languageprocessingdomain, ithasbeen in- troducedinimage processingtasksas well[15,11]. The mainbuilding blockofthisarchitecturei sself-attention whichasthe namesuggestscalculates theresponseat apo- sitionina sequencebyattending toallpositions withinthe samesequence.Gi venan inputtensorofshape(C,H,W) itisflattened toamatrix z?RHW×Candprojectedto d aanddcdimensionalspacesusing embeddingmatrices W

A,B?RHW×daandC?RHW×dcareknown asquery,

keyandvalue,respectively .Theoutput oftheself-attention mechanismfora singleheadcan beexpressed as

O=softmax?ABT

⎷da? C(1)

Themaindra wbackofthis approachisveryhighmemory

requirementdueto thematrixmultiplication ABTwhich requiresstoring ahighdimensionalmatrixofdimension (HW,HW)forimagedomain. Thisrequiresa largedo wn- samplingoperationbefore applyingattention.[15] and[17] usealocal memoryblockinstead ofglobalall-to-all for makingitpractically usable.[1]uses attentiononlyfrom thelayerwith thesmallestspatial dimensionuntilit hits memoryconstraints.Also, theseworks typicallyresorttoquotesdbs_dbs21.pdfusesText_27

[PDF] hierarchical network design pdf

[PDF] hierarchical regression table apa

[PDF] hierarchical structure journal article

[PDF] hierarchy java example

[PDF] hierarchy of law reports

[PDF] hifly a321

[PDF] hifly a380 interior

[PDF] hifly a380 model

[PDF] high appellate court definition

[PDF] high court

[PDF] high efficiency boiler

[PDF] high level french adjectives

[PDF] high net worth individuals survey

[PDF] high paid jobs in demand uk

[PDF] high paying jobs in high demand uk