[PDF] Advanced Python for Scien0fic Compu0ng

View - Download Advanced Python for Scien0fic Compu0ng

Previous PDF

Next PDF

AdvancedPythonforScien2ficCompu2ng

MichaelMilligan

milligan@msi.umn.eduFollowalong!

Togetthemostoutofthis...

• BasicknowledgeofPython • WorkingPythoninstall(feelfreetouseours!) - EnthoughtPythonDistribuCon/Canopyprovides scienCficandmathlibrariespre-installed • MSIlogin+SSHorNX• Followalong!

WhyPythonforScienCficCompuCng?

• Rapiddevelopment - Easy,readablesyntax- VersaCletoolsforexperimentaCon/learning- Comprehensivelibraries • PowerfulFeatures - Processdataatnear"naCvecode"speeds- ExcellentvisualizaConpackages- Comprehensivelibraries

Whenyouleavetoday,youshouldbeableto...

• PrograminteracCvelywithipython

• understandthebasicsofnumpyandscipy• Efficientlycomputewithlargearraysofdata• Loadandsavedatato/fromfilesondisk• Usematplotlibtoplotdata• TakeadvantageofsupercompuCngresourceswith

parallelcompu2ng • Knowwheretoturnformorehelpwiththesetopics

Details

• WearedescribingEnthoughtPythonDistribuCon. - EssenCally:Pre-assembledcompilaConofPython2.7+ numpy,scipy,otherusefullibraries - Freeforacademicuse,abasicversionisfreefornon- commercialuse - Yourcomputers,departments,etcmayhaveadifferent • InMSI:moduleloadpython-epd

WorkshopConvenCons

• UNIXshellcommandsareindicatedwiththepercentsign. • IPythoninterpreter commandshaveIn/Outlabels • Neithersignindicates

IPython:InteracCvePython

• PowerfulenvironmentforinteracCvework • Runas"ipython"from anyterminal • --pylabopConauto- loadsnumpy,setsupgraphicsforploang • Inspectanyobjectwith "?"orhelp()

IPython:InteracCvePython

• BuildupaworkspaceofobjectsandfuncCons • Fullhistoryaccess throughOut[],%recall,up/downarrowkeys • %load,%edit,or%run externalfiles • Lotsmore,type%magic

NumPyandSciPy

• NumPyprovides:

• thebasicarrayandmatrixdatatypes• EfficientimplementaConsoflow-levelmathoperaCons• Alargelibraryofhigh-levelmathfuncConsbuiltfrom

efficientprimiCves • SciPyprovides: • Ahomeforawidevarietyofopen-sourcemathemaCcal andscienCficalgorithms • ModulesforopCmizaCon,signalprocessing,linear algebra,staCsCcs,interpolaCon,andmore

NumPyarrays

• ArraydatatypewithvectorizedoperaCons(similartoMatlaborIDL) • SupportssameoperaConsas

Pythonlisttype

• ...excepteveryelementisof samedatatype • ...sotheycanbestoredin memorypackedlikeCarrays

NumPyarraysarefast

Here we are comparing a "pure Python" loop to the equivalent in numpy

MulCdimensionalarrays

• NumPyarraysarerectanglesinarbitrarilymanydimensions • +-*/operate element-by-elementforsame-shapearrays

Arrayslicing

• IndexnotaCongivesaccesstoany"slice"ofanarray • Arrayslicescanbe assigned-thischangestheoriginalarray • X=M[1,:,:].copy() wouldavoidchangingM

Othercommonmethods

• Numpyarrayshavemanyusefulbuilt-inmethods

Othercommonmethods

• ...andthenumpymoduleprovidesmore

CondiConsandtests

• Vectorizedlogicaloperators+indexingfuncCons • Outputofindex funcConscanbeusedtoslicearrays

CondiConsandtests

• Vectorizedlogicaloperators+indexingfuncCons • Outputofindex funcConscanbeusedtoslicearrays

Moreusefulnumpymodules

• numpy.fft-FFTs,forward/inverse,1-DandN-D • numpy.random-generaterandomnumbers,many distribuConstochoosefrom • numpy.matrix-specialarraysthatobeymatrixmath• numpy.polynomial-moduleforrepresenCngand manipulaCngarbitrarypolynomials

Ploangmadeeasy

• Matplotlibprovideshigh-quality2-D(andsome3-D)ploang • DisplayinwindoworoutputtoPDF,SVG,PNG,etc• Implementedasmodularobject-orientedsystem • PylabprovidesaMatlab-ishinteracCve interfacetoMatplotlib • Accesswithipython--pylab• Defaultstopoppingupplotsinaseparatewindow

Somebasicexamples...

Someadvancedexamples...

• Theseexamplesarefromthematplotlib.orgexamplessecCon...

Someadvancedexamples...

Someadvancedexamples...

Someadvancedexamples...

Someadvancedexamples...

SciPyexpandsthemenu

• Clusteringalgorithms(scipy.cluster)

• IntegraConandODEs(scipy.integrate)• InterpolaCon(scipy.interpolate)• Inputandoutput(scipy.io)• Linearalgebra(scipy.linalg)• MulC-dimensionalimageprocessing(scipy.ndimage)• OpCmizaConandrootfinding(scipy.opCmize)• Signalprocessing(scipy.signal)• Sparsematrices(scipy.sparse)• SpaCalalgorithmsanddatastructures(scipy.spaCal)• SpecialfuncCons(scipy.special)• StaCsCcalfuncCons(scipy.stats)• Andthensome...

SciPyisalsofast

• MostSciPyrouCnesusefastNumPylow-levelmathoperaCons • SomeSciPyrouCnesusehighlyopCmized externallibraries - E.g.scipy.linalglinkstoBLAS,LAPACKorMKL behindthescenes

Dataondisk

• Chancesareyouwanttoloadandsavedata • numpyandscipy.ioofferavarietyoffaciliCes

Dataondisk:textfiles

• Verycommonforsmallerdatasets:simplecolumnsofnumbers • numpy.loadtxt()-simpleinterface,gooddefaults• numpy.genfromtxt()-morecomplex,handles

Dataondisk:textfiles

• Numpy.savetxt()-writetocolumnsofnumbers

Dataondisk:binaryformats

• Binarydataismuchmorescalable

• Smallerfilesondisk• Fastertoloadandsave• Maybenecessarytoexchangedatawithothersopware• SCcktoportable(machine-independent)formats

Dataondisk:binaryformats

• NumPyna2veformat(.npy)

• numpy.load()andnumpy.save()• Orusenumpy.savez()tostoremanyarraysincompressed.npz• Fast,portable,butmostlyonlysupportedbyPython

• scipy.io.matlab-supportforMatlab(.mat) • scipy.io.loadmat()andscipy.io.savemat() • scipy.io.idl-read(nosave)IDL.savfiles • scipy.io.readsav()

Dataondisk:binaryformats

• Manystandardformatssupported

• scipy.io.netcdf-NetCDF3interface• h5pyexposesHDF5API• PyTablesisanexcellenthigh-levelinterfacetoHDF5• pyfitsforFITSdatasets• Etc...

ScalingupwithparallelizaCon

• Forbigjobsyouwilleventuallywanttoparallelizeyourcode • ThePythoninterpreterhastroublewith mulCthreading-mulC-processisusuallybest • Approachdependsontheproblemyouneed tosolve

Parallelprocesses

• Manyjobsneedtoprocesslotsofdata,don'tneedtocommunicateamongstthemselves • SomeCmescalled"embarrassinglyparallel"• GNUParallel--asimplewaytolaunchjobs • Launchonejobforeveryfileinadir,lineinafile,etc• CanworkwithPBSonitascatousemanynodes

GNUParallelexample

GNUParallelexample

• -jshouldmatchppn(unlessyouknowwhatyou'redoing)-thisisprocessespernode • Willrunonejobperlineofinputonstdinorin argfile-maxofnodes*ppnrunningatonce • See"manparallel"formorefeatures

MPIforPython

• MPI"MessagePassingInterface"enablesparallelprocessestocommunicateefficiently • Commonlyoneprocesswillbe"controller"and manageworkerprocesses

• Inherentsupportforscaser-gatheroperaCons• MPIiswell-supportedonourclusters• mpi4pyinterfacestoMPIfrominsidePython• CaveatforMPIgurus:numpydoesnothave

Examplewithmpi4py

• Simple"Helloworld"script

Examplewithmpi4py

• Simple"Helloworld"script

Morewithmpi4py

• Possibletopassnumpyarrayslikebuffers

Morewithmpi4py

• Alsoworkswith(pickle-able)Pythonobjects • MuchslowerthanC-basedarrays,butveryconvenient

Toomuchtocover...

• Ipythonnotebook-connecttoipythonwithabrowserforaMathemaCca-likenotebookinterface

• PyCUDAandPyOpenCL-GPUcompuCng• SymPy-MathemaCca-stylesymbolicmath• DatabasesareeasytoconnecttoPython;or

IPythonnotebookexample

• Example:IPythonnotebookwithpylabandsympy • notebookcreates graphicalloginabrowser • sympy:symbolicCAS• Totrythis:

CommunityandDocumentaCon

• AcCvelydevelopedandsupported

• ExcellentdocumentaCon• www.python.org/doc• Scipy.org• wiki.scipy.org• Ipython.org• matplotlib.org• Mpi4py.scipy.org

NextStep

• Hands-on • Youcanalsoruntheexamplesonyour laptop'sPythondistribuCon • Enthoughtisinstalledinalllabsandon supercomputersatMSI • FullacademicversionofEnthoughtCanopy installed(notdefaultyet) • QuesCons!quotesdbs_dbs20.pdfusesText_26

[PDF] advanced python programming pdf

[PDF] advanced python programming pdf 2018

[PDF] advanced python projects pdf

[PDF] advanced python scripting for arcgis pro

[PDF] advanced r packages

[PDF] advanced real analysis book pdf

[PDF] advanced real analysis textbook pdf

[PDF] advanced regular expression in python

[PDF] advanced research methodology pdf

[PDF] advanced search app for android

[PDF] advanced segmentation adobe analytics

[PDF] advanced setting app for android

[PDF] advanced shell scripting pdf

[PDF] advanced spoken english lessons pdf

[PDF] advanced sql book reddit