Advanced ParaView Visualisation

Transcription

1 Advanced ParaView Visualisation Tips and tricks beyond GMV-like operations Graduate Class Numerical Simulation / IANUS Technische Universität Dortmund Sven Buijssen sven.buijssen@tu-dortmund.de November, 5th

2 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 2

3 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 3

4 Choosing a suitable module ParaView is a big package and can be configured and compiled against numerous third-party libraries, e.g.: without FFMPEG: create sequence of PNGs, TIFs or JPGs from GUI with FFmpeg: additionally create AVIs directly from GUI with Python: possible to write small filters in GUI, no need to code C++ without MPI: Only 1 ParaView data server possible. with MPI: Arbitrary number of ParaView data/render servers possible (clusters with/without GPUs) with OpenGL: Use GPU on compute server (if not careful: GPU on display host!) with Mesa: Use software rendering for hosts without GPUs 4

5 Why compiling ParaView yourself? Kitware Inc. provides ready-to-use binaries for Linux, Mac OS and Windows. Why bother compiling ParaView yourself? ParaView is a big package and as such contains numerous minor and major bugs. New releases are issued every 4 6 months. Find a bug, report it and a fix may be available in CVS/Git long before a new binary release is available. Both developing plugins and compile-time changes (e.g. Postscript export for rendered objects) require self-compilation. Provided binaries are without MPI support. Because MPI is implemented with different (internal) APIs by different vendors. Kitware Inc. would have to create a different binary for cross product of operating system and MPI implementation. 5

6 Choosing a suitable module at TU Dortmund (1) The workstations & servers at Institute of Applied Mathematics and on LiDOng clusters have multiple ParaView installations: Official ParaView releases (version and 3.6.1) Multiple CVS builds. Main difference there: with OpenGL or with Mesa. Why care? Because running an unsuitable version may result in slower performance than possible (Mesa means software rendering, Open GL implies hardware rendering) segfault on startup You re lucky segfault (because of invalid GL command) any time when building a pipeline in GUI You loose a pipeline and have wasted your time. (Segfault may happen for gmvgl, too, when run in context without OpenGL support.) 6

7 Choosing a suitable module at TU Dortmund (1) Choices at one of the workstations & servers at Institute of Applied Mathematics: Linux 32 bit workstations (most of them have GPUs): User sits in front of the workstation feasible and recommended to use OpenGL version, e.g. % module load \ paraview/3.7.0-gl-cvs postscriptexport-gmvimport User is connected to workstation via ssh (e.g. in a NX session) or uses one of the 32 bit compute servers without a GPU use Mesa version!! E.g. % module load \ paraview/3.7.0-mesa-cvs postscriptexport-gmvimport Linux 64 bit workstation and servers (only a few have GPUs, but those are reserved for GPU computing, not visualisation): use Mesa version, e.g. % module load \ paraview/3.7.0-mesa-cvs postscriptexport-gmvimport 7

8 Choosing a suitable module at TU Dortmund (4) Choices at one of the workstations & servers at Institute of Applied Mathematics: Solaris SPARC servers, i.e. SunRay users: Sun Solaris 10 s OpenGL has a bug that causes PV to segfault and typically crashes the Xserver, too ( immediate logout!) use Mesa version, e.g. % module load \ paraview/3.7.0-mesa-cvs postscriptexport-gmvimport 64 bit Itanium server: no ParaView installed Windows terminal server: no ParaView installed 8

9 Choosing a suitable module at TU Dortmund (5) Choices on LiDOng cluster: Linux 64 bit gateway servers and compute nodes (neither one has GPU(s)): use Mesa version, e.g. % module load \ paraview/3.7.0-mesa-cvs postscriptexport-gmvimport 9

10 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 10

11 Choosing a suitable module for client/server setup (1) ParaView supports separating the tasks of data display, reading of data and rendering of data. By default, ParaView uses builtin mode: reading, rendering and display on one machine Via Connect method ParaView client can connect to local or remote ParaView server processes In case of a remote server log in via ssh and create a tunnel for ParaView connection. Example: % ssh lidong1.hrz.tu-dortmund.de \ -L<client port>:localhost:<server port> Caution: ensure that ports are not already in use! Check this beforehand e.g. with % netstat -an grep <client-side port number> on host where ssh is invoked and % netstat -an grep <server-side port number> on host you are connecting to. In case of a local multi-core machine (that noone else uses) skip this step and use default ports in the following 11

12 Choosing a suitable module for client/server setup (2) Example (continued): 2 Load suitable module % module load paraview/3.7.0-cvs Start ParaView data+render server processes Start 1 process % pvserver \ server-port=<server port> use-offscreen-rendering Start multiple processes % mpirun -np 16 pvserver \ server-port=<server port> use-offscreen-rendering 4 Run paraview client (GUI) locally and choose Connect to local port <client port> % paraview File - Connect Add server. Name: LiDOng via ssh tunnel Server Type: Client/Server Host: localhost Port: <client port> Ok Select entry and press Connect 12

13 Choosing a suitable module for client/server setup (3) Example (continued): 5 Test with e.g. Sources - Cone - Apply 6 Distribute data to parallel processes (aka partition) by applying D3 filter 7 Visualise partitioning by applying Process Id Scalars filter 13

14 Choosing a suitable module for client/server setup (3) Caution: When running ParaView in client/server mode, do not mix versions! Never use e.g. ParaView in combination with ParaView 3.7 In general do not mix ParaView 3.7 <some date> with ParaView 3.7 <some later date>, either. 14

15 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 15

16 Exporting vector graphics By default, ParaView supports screenshots in file formats PNG, BMP, TIFF, PPM, JPG and PDF. All are bitmap based formats. True vector graphics (PS, EPS and SVG) export is only available in self-compiled ParaView, not in official binary releases (Not sure whether it is possible to convert the required changes into a plugin) Compile time requirements: my Postscript patch plus old versions of class vtkpvscalarbaractor (pre-november 2008) Run time requirement: Disable Interpolate Colors checkbox in Object Inspector, Display tab. Otherwise object will be exported with greyscale colormap! Works only for 3D view (i.e. not for 2D view nor bar char/xy plot/spreadsheet view...) Works only in built-in mode, not in server mode Does support all types of source objects (sole exception: Mandelbrot source) and mesh types Live demonstration 16

17 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 17

18 Converting data to fulfill filter s input requirement Most filters require certain types of mesh as input To convert your data set to required format apply one of Clean to Grid filter Delaunay 2D filter Delaunay 3D filter Extract Surface filter Triangulate filter Tetrahedralize filter Tesselate filter Live demonstration: 2D data set from level 7, Extract Surface, Triangulate, Decimate data set on level 2. Use case: reduce memory imprint of large file series, create animation scene with reduced dataset ( faster prototyping), save state file, edit it to use the original file series, finally use ParaView GUI or pvpython script to create high-resolution animation 18

19 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 19

20 Emulate gmvdiff in ParaView Jens Acker has written gmvdiff, a command line tool that imports two (2D/3D) gmv files (possibly using different meshes) and creates a new gmv file containing data arrays being the difference of data arrays in the input files. Use case: How did velocity and pressure change between T=3.0 and T=3.5? Realisation in ParaView: Load both data sets Apply Calculator filter on each of them to give field of interest a unique name (e.g. velocity vel1 / vel2 ) Apply Append Attributes filter to both Calculator objects Apply Calculator and compute difference between the two field using their unique names (operation: vel1 - vel2 ) In case of different meshes, apply a Resample with dataset filter before using Append Attributes filter. 20

21 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 21

22 Emulate gmvpt in ParaView (1) Jens Acker has written gmvpt, a command line tool that imports a sequence of (2D/3D) gmv files providing velocity data arrays, performs particle tracing based on a ASCII configuration file and creates a sequence of new gmv files containing tracers. Realisation in ParaView for steady state data set: Load data set Apply Stream Tracer filter Position Point Source or Line Source somewhere inside of the geometry 22

23 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 23

24 Emulate gmvpt in ParaView (2) Jens Acker has written gmvpt, a command line tool that imports a sequence of (2D/3D) gmv files providing velocity data arrays, performs particle tracing based on a ASCII configuration file and creates a sequence of new gmv files containing tracers. Realisation in ParaView for transient data: Load in a time-dependent file series In case of GMV files possibly convert them to ExodusII file format first: 1 Loading of ExodusII files faster due to smarter file format (no need to parse all files for time information, see previous tutorial on GMV reader plugin) 2 ParticleTracer filter will cause GMV reader plugin to raise (harmless) warnings of kind vtkfileseriesreader: vtktemporaldataset not fully supported. Apply Temporal Cache filter first (mandatory!), set to cache 2 time steps Create Point Source somewhere inside of the geometry Apply ParticleTracer filter (Input: TemporalCache object, Source: PointSource) Toggle visibility of TemporalCache object in Pipeline browser Optional: Apply Pathlines filter 24

25 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 25

26 Equivalent of gmvmpeg3 / gmvmpeg4 gmvmpeg is a script of mine that automates the process of applying a given attribute file to a gmv file series, creating snapshots and converting them into a movie. ParaView supports loading and saving of state files (equivalent to GMV s attribute files), from GUI and in plain Python scripts Manipulation of state files from within Python possible, but tricky: Following scenario as replacement for gmvmpeg4 in principle possible, but not mastered yet: Load a single file or a small file series, set up a pipeline, save state Load state file from Python script and let script replace file series from state file with a different (complete) file (series) Let Python script generate snapshot/animation Easier: Edit state file (plain XML file) and alter file name references. Automate this in a custom shell script. Additionally: ParaView supports generation of AVIs as well as saving sequences of PNGs, TIFs and JPGs directly from GUI. No replacement for my gmvmpeg scripts available nor planned Bitmap file series can be converted to movie by means of, e.g., Unix command line mencoder "mf://animation*.png" -mf type=png:w=1280:h=444:fps=25 -nosound \ 26 -ovc lavc -lavcopts vcodec=wmv2:vbitrate=1200 -o wmv2.avi

27 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 27

28 Examples Programmable filter Problem: Export data as CSV file, but only some of the data, not all. No ready-to-use solution in ParaView Possible with the following pipeline: Apply Extract Block and Merge Blocks filter for multi-block data set Apply Programmable Filter with the following Python source code: inp = self.getinput() filename = "foo.csv" FILE = open(filename, "w") FILE.write("x,y,z, x-velocity\n") numpoints = inp.getnumberofpoints() vel = inp.getpointdata().getarray("velocity") for i in xrange(0, numpoints): coords = inp.getpoint(i)[:3] FILE.write(str(coords[0]) + ",") FILE.write(str(coords[1]) + ",") FILE.write(str(coords[2]) + ",") FILE.write(str(vel.GetValue(i * 3)) + "\n") FILE.close() 28

29 Examples Programmable filter (cont.) Problem: Node and cell numbers should start at 1, not 0. VTK always starts counting at 0, so does ParaView. No ready-to-use solution in ParaView Possible with the Programmable Filter (apply Extract Block + Merge Blocks filter for multi-block data sets first) inp = self.getinput() out = self.getoutput() # Provide all input data sets ourselves out.shallowcopy(inp) # (continued from left column) # Shift cell number by 1 numcells = inp.getnumberofcells() data = inp.getcelldata().getarray("ids") offset = +1 # Shift point number by 1 numpoints = inp.getnumberofpoints() newdata = vtk.vtkidtypearray() newdata.setname( Point IDs (shifted by 1) ) newdata.setnumberofcomponents(1) for i in xrange(0, numpoints): newdata.insertnextvalue(i+offset) out.getpointdata().addarray(newdata) # (continued in next column) newdata = vtk.vtkidtypearray() newdata.setname( Cell IDs (shifted by 1) ) newdata.setnumberofcomponents(1) for i in xrange(0, numcells): newdata.insertnextvalue(i+offset) out.getcelldata().addarray(newdata) 29

30 Examples Programmable filter (cont.) Problem: Plot number of nodes and cell numbers over time. No ready-to-use solution in ParaView Possible with the Programmable Filter (apply Extract Block + Merge Blocks filter for multi-block data sets first) inp = self.getinput() out = self.getpolydataoutput() newpoint = vtk.vtkpoints() newpoint.insertpoint(0,0,0,0) out.setpoints(newpoint) numpoints = inp.getnumberofpoints() numcells = inp.getnumberofcells() newdata = vtk.vtkunsignedlongarray() newdata.setname("numpoints") newdata.setnumberofcomponents(1) newdata.insertnextvalue(numpoints) out.getpointdata().addarray(newdata) # (continued in next column) # (continued from left column) newdata1 = vtk.vtkunsignedlongarray() newdata1.setname("numcells") newdata1.setnumberofcomponents(1) newdata1.insertnextvalue(numcells) out.getpointdata().addarray(newdata1) Select newly created point Apply Plot Selection over Time filter 30

31 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 31

32 Custom filters Provide a means of grouping and saving a set of filters (a sub-pipeline) for future use as a new filter. Can be exported and imported easily (shareable, XML-based format) Live demonstration (Save last Programmable Filter example as Custom filter, store it as Demo - Plot Number of Points & Cells over Time ) 32

33 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 33

34 Lookmarks Are (more or less) custom filters plus stored color by settings and (optionally) stored camera positions Facilitate reproduction of pipeline for another source (e.g. reader source) Are created via Tools - Create Lookmarks Persist through sessions if saved A lookmark s camera settings are restored if the corresponding check box is enabled in the Lookmarks Inspector (accessible via View menu or by selecting edit in a lookmark s context menu) Live demonstration 34

35 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 35

36 Live Data GMV supports auto-read of new files (through busy-wait 100 % cpu load discouraged use case) Difficult to realise in VTK and ParaView and not available for default ParaView readers Reason: No means in general to guarantee consistency of new file, i.e. has it been written completely yet by external application such that it is safe to read the file? Has been discussed frequently in the past (see e.g. discussion and realised at least once (by Takuya Oshima for his parallel OpenFOAM reader plugin, accepting the fact of the mentioned race conditions). No plans to incorporate this into the GMV reader plugin. 36

37 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 37

38 Stability of ParaView 2,000,000 million lines of code (non-blanks, non-comments) (see rule of thumb in computer science: 1 critical error per 1,000 lines of code many bugs in ParaView! Public bug tracker: (Note: Just don t upload any GMV files as GMV reader plugin is neither yet known nor available outside our department. Try converting to one of VTK file formats if necessary to reproduce bug.) Evaluation of Stability Very stable for standard GMV-like operations Expect frequent segmentation faults when trying to create high gloss animations. Save state files often! Better: Make sure to activate "Crash Recovery" checkbox in Edit - Settings - General. Official releases more stable than CVS HEAD versions. The latter have nice new features, though. (Official release cycle: 4 6 months) 38

39 Outline 1 Choosing suitable ParaView module 2 Client/Server setup 3 Exporting vector graphics (PS, EPS, SVG) 4 Converting data to fulfill filter s input requirement 5 Emulate gmvdiff in ParaView 6 Emulate gmvpt for steady state data set / Stream Tracer 7 Emulate gmvpt for transient data / Particle Tracer 8 Equivalent of gmvmpeg 9 Examples Programmable filter 10 Custom filters 11 Lookmarks 12 Live Data 13 Stability of ParaView 14 Learn more about ParaView 39

40 Learn more about ParaView (mirror: ParaView mailing list Thank you for your attention. 40