[PDF] Generating Artistic Portrait Drawings From Face - CVF Open Access PDF Yi_APDrawingGAN_Generating_Artistic_Portrait_Drawings_From_Face_Photos_With_Hierarchical_CVPR_2019

This allows dedicated drawing strategies to be learned for different facial features Since artists' drawings may not have lines perfectly aligned with image features,

When children draw, they carefully choose their materials, crayons, colours, patterns, plus the size and position of what they want to draw Children's drawings are

[PDF] Generating Artistic Portrait Drawings From Face - CVF Open Access

This allows dedicated drawing strategies to be learned for different facial features Since artists' drawings may not have lines perfectly aligned with image features,

[PDF] A System for Drawing Pictures with Eye Movements - University of

to drawing pictures, and (c) a user-tested implementation of the idea within a their eyes—to draw pictures with their eyes, and thus benefit from the same

[PDF] Using Drawings in Play Therapy: A Jungian Approach - ERIC

proaches as they pertain to drawing tech- niques within the counseling session Using Drawings in Play Therapy Play is how children explore the ex-

[PDF] An Analysis of Students Drawings for the Purpose of - CORE

drawings according to the number and variety of marine organisms The results of Keywords: Biology/art; students' drawings; interdisciplinarity; team teaching

APDrawingGAN:GeneratingArtistic PortraitDrawings

fromFacePhotoswith HierarchicalGANs

RanYi, Yong-JinLiu

CSDept,BNRist

TsinghuaUniv ersity,China

SchoolofComputer ScienceandInformatics

CardiffUniversity, UK

{LaiY4,RosinPL}@cardiff.ac.uk

Figure1:(a) Anartistdra wsaportrait drawingusing asparseset oflinesandveryfe wshadedre gionstocapture the

distinctiveappearanceofagiven facephoto. (b)OurAPDra wingGANlearnsthis artisticdrawingstyleandautomatically

transformsaf acephotointo ahigh-qualityartisticportraitdrawing. (c)Usingthe sameinputf acephoto,six state-of-the-art

styletransfermethods cannotgeneratedesired artisticdrawings: DeepImageAnalogy [

20],CNNMRF[ 18],Gatys[ 11]and

HeadshotPortrait[

32]changef acialfeaturesor failtocapturestyle,CycleGAN [40]andPix2Pix [15]producef alsedetails

aroundhair, eyesorcornersofthe mouth.

Abstract

Significantpro gresshasbeenmadewithimagestyliza-

tionusingdeep learning,especially withgener ativeadver- sarialnetworks(GANs). However, existingmethods failto producehighqualityartisticportr aitdrawings. Suchdr aw- ingshavea highlyabstract style,containing asparse setof continuousgraphical elementssuchaslines,and sosmall artifactsare moreexposedthanfor paintingstyles.More- over,artiststendtousediffer entstrate giestodr awdiffer ent facialfeatures andthelinesdrawnar eonlyloosely related toobviousima gefeatur es.Toaddressthesechallenges, wepropose APDrawingGAN,anovelGAN basedarchitec- turethatbuildsupon hierarc hicalgeneratorsand discrim- inatorscombiningbothaglobal network (forimag esasa

*Correspondingauthorwhole)andlocal networks(forindividual facialre gions).Thisallowsdedicated drawingstr ategiesto belearnedfordifferentfacialfeatures.Sinceartists"dr awingsmaynot havelinesperfectly alignedwithima gefeatur es,we developanovel losstomeasuresimilaritybetween generated andartists"drawings basedondistancetransforms,leading toimprovedstrokesinportrait drawing.TotrainAPDrawing- GAN,weconstruct anartisticdr awingdatasetcontaining high-resolutionportraitphotosand correspondingprofes-sionalartisticdr awings.Extensivee xperiments,andauserstudy,showthatAPDrawingGAN producessignificantly betterartisticdr awingsthanstate-of-the-art methods.1.Introduction

Portraitdrawings arealongstandinganddistinctart

form,whichtypically useasparse setofcontinuous graph-

icalelements(e.g., lines)tocapture thedistinctiv eappear- anceofa person.They aredrawn inthepresence oftheper-sonortheir photo,andrely onaholist icapproachof obser-vation,analysisandexperience. Anartisticportrait drawingshouldideallycapture thepersonalityand thefeelingsof theperson.Even foranartistwithprofessional training,itusu- allyrequiresse veralhours tofinishagoodportrait(Fig.

1a).

Trainingacomputerprogramwith artists"drawings and

automaticallytransformingan inputphotointo high-quality artisticdrawings ismuchdesired.Inparticular, withthe whichusesCNNs toperformimage styletransferw as proposed[

11].Lateron, generativeadversarialnetwork

(GAN)basedstyle transfermethods(e.g., [

15,40,2,5])

haveachievedespeciallygoodresults, byutilizingsetsof (pairedorunpaired) photosandstylized imagesforlearn- ing.Thesee xistingmethodsare mostlydemonstratedus- ingclutteredstyles, whichcontainman yfragmentedgraph- icalelementssuch asbrushstrok es,andha vea significantly lowerrequirementforthequality ofindividual elements (i.e.,imperfectionsare muchlessnoticeable).

Artisticportrait drawings(APDrawings)aresubstan-

tiallydifferent instylefromportraitpaintingstyles studied inprevious work,mainlyduetothe followingfive aspects. First,theAPDra wingstyleis highlyabstract,containinga smallnumberof sparsebut continuousgraphicalelements. Defects(suchas extra,missing orerroneouslines) inAP-

Drawingsaremuchmorevisible thanother stylessuchas

paintings(e.g.,impressionist andoilpainting) involving a densecollectionof thousandsofstrok esofv aryingsizes andshapes.Second, therearestronger semanticconstraints forAPDrawing styletransferthanforgeneralst yletransfer. Inparticular, facialfeaturesshouldnotbe missingordis- placed.Even smallartifacts(e.g.,around theeye)canbe clearlyvisible,distracting andunacceptable.Third, theren- deringinAPDra wingsisnot consistentbetweendifferent facialparts(e.g.,eyes vs.hair).F ourth,theelements (e.g. theoutlineof facialparts) inAPDrawings arenotprecisely locatedbyartists, posingachallenge formethodsbased on pixelcorrespondence(e.g.,Pix2Pix[

15]).Finally, artists

putlinesin APDrawingsthat arenotdirectly relatedtolow levelfeaturesintheviewor photographofthe person.Ex- amplesincludelines inthehair indicatingtheflo w,or lines indicatingthe presenceoffacialfeaturese venif theimage containsnodiscontinuities. Suchelementsof thedrawings arehardto learn.Therefore,e venstate-of-the-art image styletransferalgorithms (e.g.,[

11,15,18,20,32,40])of-

tenfail toproducegoodandexpressi veAPDra wings.See Fig.

1cforsome examples.

Toaddresstheabov echallenges,we proposeAPDraw-

ingGAN,ano velHierarchical GANarchitecturededicated toface structureandAPDrawingstylesfor transforming facephotostohigh-qualityAPDra wings(Fig.

1b).To effec-tivelylearndifferentdrawingstyles fordifferent facialre-gions,ourGAN architectureinvolvesseverallocalnetworksdedicatedtof acialfeaturere gions,alongwithaglobalnet- worktocaptureholisticcharacteristics. Tofurther copewithline-stroke-based styleandimpreciselylocatedele-mentsinartists" drawings,we proposeano veldistancetransform(DT)loss tolearnstrok elinesin APDrawings.

Themaincontrib utionsofour workarethree-fold:

WeproposeaHierarchicalGAN architectureforartis- ticportraitdra wingsynthesisfrom afacephoto,whichcangeneratehigh-quality andexpressi veartistic por-traitdrawings. Inparticular,ourmethodcan learncomplexhairstylewithdelicate whitelines.

Artistsusemultiple graphicalelementswhen creatingadrawi ng.Inordertobestemulateartists,ourmodel separatestheGAN" srenderedoutput intomultiplelay-ers,eachof whichiscontrolled byseparatedloss func-tions.We alsoproposealossfunctiondedicated toAPDrawingwithfourlosstermsinourarchitecture,in-cludingano velDT loss(topromoteline-strokebasedstyleinAPDra wings)anda localtransferloss(forlo-calnetworks topreservefacialfeatures).

Wepre-trainourmodelusing 6,655frontalf acephotoscollectedfromten facedatasets, andconstructan AP-Drawingdataset(containing140high-resolution facephotosandcorresponding portraitdrawings byapro- fessionalartist)suitable fortrainingand testing.TheAPDrawingdatasetandcodeis available.

2.RelatedW ork

Imagestylizationhas beenwidelystudied innon-

photorealisticrenderingand deeplearningresearch. Below wesummarizerelated workin threeaspects.

2.1.Styletransfer usingneuralnetw orks

Gatysetal. [

11]firstproposed anNSTmethod usinga

CNNtotransfer thestylisticcharacteristics ofastyle image toacontent image.For agiv enimage,its contentandstyle featuresarerepresented byhighlayer featuresandte xture informationcapturedby Grammatrices[

10]ina VGGnet-

work,respectively. Styletransferisachievedbyoptimizing animageto matchboththe contentofthe contentimageand thestyleof thestyleimage. Thismethodperforms wellon oilpaintingstyle transferofv ariousartists.Ho wever ,their styleismodeled astexture features,and thusnot suitable forourtar getstylewith littletexture.

LiandW and[

18]useda Markov RandomField(MRF)

lossinsteadof theGrammatrix toencodethe style,and proposedthecombined MRFandCNN model(CNNMRF).

Yongjin.htm

CNNMRFcanbe appliedinboth non-photorealistic(art-work)andphoto-realisticimagesynthesis, sincelocalpatch matchingisused inMRFloss andpromoteslocal plausibil-ity.However ,localpatchmatchingrestrictsthismethodtoonlywork wellwhenthestyleandcontent imagescontainelementsofsimilar localfeatures.

Liaoetal. [

20]proposedDeep ImageAnalogyfor visual

attributetransferbyfindingsemantically meaningfuldense correspondencesbetweentw oinputimages. Theycompute correspondencebetweenfeature mapsextracted byaCNN. DeepImageAnalogy wassuccessfully appliedtophoto-to- styletransfer, butwhentransferringAPDrawing style,im- agecontentis sometimesaffected, makingsubjectsin the resultingimagesless recognizable.

Johnsonetal. [

16]proposedthe conceptofperceptual-

loss-basedonhigh-le velfeatures andtrainedafeedfor- wardnetworkforimage styletransfer.Similarto[

11],their

texture-basedlossfunctionisnot suitableforour style. styletransfer, mostexistingmethodsrequirethe styleimage tobeclose tothecontent image.

2.2.Nonphotorealistic renderingofportraits

Inthefield ofNPR,man ymethodsha vebeen devel-

opedforgenerating portraits[

29].Rosinand Lai[28]pro-

posedamethod tostylizeportraits usinghighlyabstracted flatcolorre gions.Wang etal.[

38]proposeda learning-

basedmethodto stylizeimagesinto portraitswhichare composedofcurv edbrushstrok es.Bergeretal.[

3]pro-

posedadata-dri venapproach tolearntheportraitsketching style,byanalyzing strokesand geometricshapesin acol- lectionofartists" sketchdata. Lianget al.[

19]proposed

amethodfor portraitvideostylization bygeneratinga fa- cialfeaturemodel usingextended MaskR-CNNand ap- plyingtwo strokerenderingmethodsonsub-re gions.The abovemethodsgenerateresultsofas pecifictypeof art,e.g., curvedbrushstrokeportrait, portraitsketching. However, noneofthem studythestyle ofartisticportrait drawing.

Therearealso someexample-based stylizationmethods

designedforportraits. Selimetal. [

30]proposeda portrait

paintingtransfer methodbyaddingspatialconstraintsinto themethod[

11]toreduce facialdistortion. Fiseretal. [9]

proposedamethod forexample-based stylizationofportrait videosbydesigning several guidingchannelsand applying theguidedte xturesynthesismethod in[

8].Howe ver,all

thesemethodsuse similartexture synthesisapproachesthat makethemunsuitableforthe APDrawingstyle.

2.3.GANbasedimage synthesis

GenerativeAdversarialNetworks(GAN)[

12]hav e

achievedmuchprogressinsolvingmany imagesynthesis problems,inwhich closelyrelatedto ourwork arePix2Pix andCycleGAN.Pix2Pix[

15]isa generalframew orkforimage-to-image

translation,whiche xploresGANsin aconditionalset- ting[

22].Pix2Pixcan beappliedtoavarietyofimagetrans-

lationtasksand achieves impressive resultsonvarioustasks includingsemanticse gmentation,colorizationand sketchto phototranslation,etc.

CycleGAN[

40]isdesigned tolearntranslation between

twodomainswithoutpaireddata byintroducingc ycle- consistencyloss.Thismodelis particularlysuitablefor tasksinwhich pairedtrainingdata arenota vailable.When appliedtoa datasetwithpaired data,thismethod produces resultssimilarto thefullysupervised Pix2Pix,but with muchmoretraining time. ingstylesand oftengeneratesblurry ormessyresults dueto thefiv echallengessummarizedinSec.

1forAPDrawings.

3.Over viewofAPDrawingGAN

Wemodelthe processoflearning totransformfacepho-

tostoAPDra wingsasa functionΨwhichmapsthe face photodomainPintoablack-and-white line-stroke-based

APDrawingdomainA.Thefunction Ψislearnedfrom

pairedtrainingdata Sdata={(pi,ai)|pi?P,ai?A,i=

1,2,...,N},where Nisthenumber ofphoto-APDrawing

pairsinthe trainingset.

Ourmodelis basedonthe GANframe work,consist-

ingofa generatorGandadiscriminator D,bothof which areCNNsspecifically designedforAPDra wingswithline- stroke-basedartistdrawingstyle. ThegeneratorGlearns tooutputan APDrawingin Awhilethediscriminator D learnstodetermine whetheranimage isareal APDrawing orgenerated.

Sinceourmodel isbasedon GANs,thediscriminator D

istrainedto maximizetheprobability ofassigningthe cor- rectlabelto bothrealAPDra wingsai?Aandsynthesized drawingsG(pi),pi?P,andsimultaneously Gistrained tominimizethis probability.Denote thelossfunction as L(G,D),whichis speciallydesignedto includefourterms L Thenthefunction Ψcanbeformulated bysolvingthe fol- lowingmin-maxproblemwiththe functionL(G,D): min

GmaxDL(G,D)=Ladv(G,D)+λ1LL1(G,D)

+λ2LDT(G,D)+λ3Llocal(G,D)(1)

InSec.

4,weintroduce thearchitectureof APDrawing-

GAN.Thefour termsinL(G,D)arepresentedin Sec.

Finally,wepresentthetraining schemeinSec.

6.An 2.

4.APDrawingGANAr chitecture

UnlikethestandardGANarchitecture, herewepropose

ahierarchicalstructure forbothgenerat oranddiscrimina-

Figure2:The framework oftheproposed APDrawingGAN.ThehierarchicalgeneratorGtakesafacephoto pi?Pasinput

andcanbe decomposedintoa globalnetwork (forglobalf acialstructure),six localnetworks (forfourlocal facialregions,

thehairand thebackgroundre gion)anda fusionnetwork. Outputsofsix localnetsarecombinedintoIlocalandfusedwith

theoutputIglobaloftheglobal networkto generatethefinal outputG(pi).Theloss functionincludesfour terms,inwhich a

novelDTlossisintroducedto betterlearndelicate artisticlinestyles. Thehierarchicaldiscriminator Ddistinguisheswhether

theinputis arealAPDra wingornot basedonthe classificationresults bycombiningbothaglobaldiscriminator andsixlocal

discriminators. tor,eachofwhich includesaglobalnetworkand sixlocal networks.Thesixlocal networkscorrespond tothelocal facialregionsofthe lefteye,righteye, nose,mouth,hair andthebackground. Furthermore,thegenerator hasanad- ditionalfusionnetw orktosynthesize theartisticdrawings fromtheoutput ofglobaland localnetworks. Thereason behindthishierarchical structureis thatinportrait drawing, artistsadoptdif ferentdrawing techniquesfordifferentparts ofthef ace.For example,finedetailsareoften drawnfor eyes,andcurvesdra wnforhair usuallyfollowtheflow of hairbut donotpreciselycorrespondtoimage intensities. Sinceasingle CNNsharesfilters acrossalllocations inan imageandis verydif ficulttoencode/decode multipledraw-quotesdbs_dbs17.pdfusesText_23

[PDF] [PDF] Generating Artistic Portrait Drawings From Face - CVF Open Access

The Analysis of Childrens Drawings: Social - ScienceDirect