ProcMT Gui. Testing Version. make a backup of your data!

Transcription

1 ProcMT Gui Testing Version make a backup of your data!

2 Chapter Table of Contents 1.Preface ProcMT Gui...5 Directories...5 ts... 6 config...7 cal... 9 processings...9 jobs... 9 db Processing...13 About Processing Modules...13 Processing Parameters...15 Create a Processing...16 ATS File Reader...17 Scale, Detrend, Hanning...18 FFT...18 Calibrate Spectra...19 Autocross Parzening...20 Stack All...22 Coherency Threshold...22 Median Processing...22 Run a Processing...24 procmt_cmds CSAMT...25 General...25 Run a CSAMT Processing Linux Terms Appendix...33 Example XML config files ADU EMAP...33 Scripts...35

3 1. P r e f a c e ProcMT (process MT) is a collection of libraries and executables. They are bundled together in several GUIs (graphical user interfaces). ProcMT works on a very specific directory structure where the software reads XML data and ats data (advanced time series) as well as calibration files. The philosophy is to configure all first and then run the program. If configured you can run many jobs the computer does wait wait on mouse clicks. All actions and joblists are stored in XML files and will be executed on request from the GUI or shell. The LINUX user will be able to run procmt_main from script files, in case you have to process hundreds of sites. The script interface can also run on the ADU systems. Activating all parallel options may lead to memory exhaustion on smaller 32bit systems.

4 2. P r o c M T G u i Directories ProcMT expects this directory structure: Fig 2.1: Directory tree Start ProcMTGui, go to File Create Project, then create the new folder in your filemanager

5 Chapter 2. Figure 2.1 Create Folder select the new directory and press ok. You also can populate the directory with help of a shell script. You can create additional folders if you want (for example processings_old or documents). This will not harm ProcMT.

6 Chapter 2. Default Folders ts The timeseries data has to be copied into the /ts/site_nnn directories. Create Site_1 (manually) for site 1 and so on ( ref. Fig 2.1); for multi channel CSAMT create for example Site_140_142_144. Site and site numbers have to be separated by underscore ( _ ). The site will be processed in the given order as the directory name. config sk ip t his for 4-5 cha nnel MT, y ou don' t need it. For CSAMT you MUST have a configuration The configuration describes the XML channel number, the channel type you want (Ex, Ey, Hx, Hy, Hz). In the simple single site MT case Site_1 with 5 channels the relation will be Ex = 0.. Hz = 4. In a multi site setup you create for example: Ex, Ey, Hx, Hy = 7, 8, 2, 3 (1st site no. 140) Ex, Ey, Hx, Hy = 0, 1, 2, 3 (2nd site no. 142) Ex, Ey, Hx, Hy = 5, 6, 2, 3 (3rd site no. 144)

7 Chapter 2. The order of your channels is linked to the directory name ADU-07e ADU-07e Fig 2.2: Create Channel Configurations

8 Chapter 2. In Mappings Configuration you insert the numbers of sites you want to have and press Apply. Configure the channels. Hence: if you process a directory with name Site_140_142_144 you can't configure more than 3 sites! Un-check off to activate the channel. cal Put your MFS06nnn.txt and MFS07nnn.txt files here. processings You find and create your processings here. All processing files are XML files. You can edit them manually. In this case make sure that you supply valid parameters.

9 Chapter 2. jobs A job file contains the processing instructions and the reference to the time series data to be used. This directory contains temporary jobs, manually configured and saved jobs. You can run one or more jobs from the GUI or from the shell "procmt_main jobs/jobfile.xml". Figure 2.2 Select jobs from GUI For batch processing.

10 Chapter 2. db Contains a SQLite database. All tensors and spectra are stored here. After creating a new survey the database must be created by hand. In the database all results ever calculated will be stored. The database can be disabled in the procmt_cmds module (command line options) with de-activate database true.

11 3. P r o c e s s i n g About Processing Modules Each module contains one or more processing instructions and will call the corresponding library which will be linked in realtime to ProcMT. You create them in the same order you want to process the data, e.g. you read scale fft (spectra) calibration auto- cross spectra / parzening one or more MT processings After scheduling an MT processing ProcMT will generate EDI files and, if requested, insert the result into a database for later usage.

12 Chapter 3. ats read scale, detrend pre-whitening FFT calibration spectra OP auto- cross spectra MT module 1 MT module N SQL database EDI files ts filter spc filter ats file

13 Chapter 3. Figure 3.1 Processing Scheme Processing Parameters Some processing parameters appear in almost any MT module: start index: 0 N, first tensor to process; default 0 (should not be set by the user) stop index: 0 N, last tensor to process; default 0 ( first = last = 0 = take all) (should not be set by the u ser) start frequency: limit the frequency range (should no t be set by th e user) stop frequency: limit the frequency range (should no t be set by th e user) upper reject rho: reject ρ if ρ xy and ρ y x exceed this limit lower reject rho: reject ρ if ρ x y and ρ yx fall below this limit upper reject phi reject: φ if φ x y and φ y x exceed this limit lower reject phi: reject φ if φ xy and φ yx fall below this limit Hence: the simple cut-off method works on the xy and yx elements only. Data is not rotated, fixed

14 Chapter 3. rotated or Swift rotated. It can be used to cutoff segments where distortions can be identified by ρ or φ. This can happen if cables where broken and have been re-connected later. use n stdev for rejection, n=0=no calculation: reject tensors outside n the standard deviation. E.g. having a Gauss distribution 3 times would be definitively outside the confidence interval. If 2 for example z x y mean is 5 ± 0.75 and z yx 9 ± 1.2 all tensors with z xy greater than 3.5 and smaller 6.5 and z yx greater 7.8 and smaller 10.2 will be selected. if z xy is 5.2 and z yx is 12.1 this element will NOT be selected. recalculate stddev after reject: if above is selected take 1 here to recalculate the standard deviation only from the selected tensors (again) Should be 1 if rejection by standard deviation is used. The order is: rotation (if applies) reject rho reject phi processing method reject by standard deviation re-calculate the standard deviation on selected data.

15 Chapter 3. Create a Processing The filename of the EDI file will be made from the site number and the processing comment. If your processing comment is mt_std_sharp and coherency threshold is selected for site 99 the resulting EDI file will be: 99_mt_std_sharp_mt_processing_ct.edi ATS File Reader Controls the data input. The ATS file reader will control your fft (window) length. skip_samples: skip samples at the beginning of the file, default 0 skip_last: ship samples at the end of the file, default 0. nslice: for CEA sliced files: n t h slice of the ats file to process. Only active if sliced files are read. Use 1024 if you want to process the last available slice. nslices: for CEA sliced files: use n slices for processing starting with nslice move_ahead: move n samples ahead after each reading, default 0

16 Chapter 3. move_back: move back n samples after reading, default 0 read_samples: how many samples to read at once (e.g. your fft window length) into one block skip_marked_parts: skip marked parts this information is stored in the atm files as a Boolean 0/1 list. overlapping: since a Hanning window is used you can set default 0 or 0.4. Do not use for CSAMT and sliced files. prestacks: pre-stack time segments in the time domain n times before calculating the FFT, default 0 dipole_txtime: n/a skip_tx_bds: n/a Scale, Detrend, Hanning Prepares the data for FFT. scale: scale the electric field to mv/km, default 1 detrend: trend removal from the segment, default 1 hanning: apply a Hanning window to the data, default 1

17 Chapter 3. FFT Executes the FFT with the window length given in the ATS file reader. cut upper spectra: default removes upper 5%; ref. page 33 cut lower spectra: default removes lower 0.1% ref. page 33 dump raw spectra: control the raw spectra of each channel (stacked). Result will be found in the dump directory and can be opened for example with xmgrace or qtgrace, default off Cut upper and cut lower can be used to limit the output spectra. This may only be used together with the various dump options. Calibrate Spectra calibration: auto: uses calibration from the xml file, if not found from the cal directory and finally built in (default). Builtin: uses a master calibration function which fits excellent for sampling rates up to 4 khz and good above. Theoretical: uses a theoretical function.

18 Chapter 3. dump calibration: dumps athe calibration function to the dump directory. dump cal spectra: dumps the calibrated spectra to the dump directory. Autocross Parzening This module controls the generation of auto- crossspectra and the parzening (MT) as well as the target frequencies (from list or calculated = CSAMT). xml MT site config file (empty for default): You MUST give a configuration file in case of CSAMT. In case of standard MT (2, 4, 5 channels) ProcMT can guess the configuration. cut upper spectra: limit upper frequency for processing by n (default 0); 0.1 would remove upper 10% ref. page 33 cut lower spectra: limit lower frequency for processing by n (default 0.001) ref. page 33 parzen radius: 0.05 to 0.8 combines frequencies in the spectral domain. higher values (0.25, 0.5) should create more smooth data. Lower values can be used for more stable MT data as well to get points near interfering lines (e.g. close to 50/60 Hz)

19 Chapter 3. activate MT/CSAMT: Switch to MT, MT scalar, CSAMT, CSAMT scalar. Hence that scalar data is mostly useless. Frequency List: Select one of the pre-defined lists (default, txmsmall, generate (for CSAMT) or select a filename freq_list_[1-7], CEA_TX or CSAM_TX (for CSAMT) or VLF for VLF. These files (ending with.dat) are to be placed in the config directory and are plane ASCII files containing the frequencies you want to use. the default list contains 8 frequencies per decade and is compatible with older processings from ADU-06. If the CSAMT mode is activated following items will be evaluated by ProcMT: base TX frequency: the transmitter frequency for example 8 Hz n odd harmonics : e.g.1, 2, 3 will generate a frequency list of 8 Hz; 8 Hz, 24 Hz and 8 Hz, 24 Hz 40 Hz and so on. dipole 1 TX time[s]: time in seconds. Take even times like 8, 16, 32, 64 and so on in order to fit most possible FFT windows

20 Chapter 3. dipole 2 TX time[s]: must have the same value as above. Dipole transmission times must be allways symmetric. activate online processing: n/a yet dump CSAMT spectra infos: dump CSAMT spectra infos. dump parzen spectra: dump the stacked parzened spectra. After finishing auto- and cross spectra the MT modules are activated. You can select one or more modules at the same time. Depending on the memory and CPU of your computer you can choose the cat all option in the command line module. This will lead to a vast amount of memory usage and parallel execution of the MT modules. Spectra from several bands will be processed overlapped. Stack All Stack all is the most simple method. In most cases this method is used to compare the statistical method against the base. Hence: in case of excellent data all methods will converge. In case of noisy data you will see different results for any

21 Chapter 3. method. It is the responsibility of the geophysicist to decide which result is the most reasonable. This module consist of the basic parameters only, as described in Processing Parameters, page 15 Coherency Threshold Coherency threshold is a classical method to sort out data. If Ex and Hy are correlated the transfer function z xy will have a good estimation (small error). In some cases of correlated spikes or other correlated noise you may want to limit the upper threshold in order to sort out what is normally the best data. In other words: if Ex and Hy are outliers at the same time, the data seems to be good but is an outlier. ref. Processing Parameters, page 15, and: upper threshold: min 0.02, max=1.0 (default=1.0). 1.0 corresponds to perfect correlated data. Data with better coherency will be rejected. lower threshold: min , max=0.99 (default=0.8). In case of 0.8 data with a coherency less than 0.8 for z x y and z y x will be rejected

22 Chapter 3. upper threshold %: adaptive coherency n/a yet lower threshold %: adaptive coherency n/a yet Median Processing The median processing calculates the median of the the z xy and z y x component of your tensor array. Choosing upper and lower bound median values will be rejected if outside. Hence: you will not get the median itself if upper & lower = 1 (it will likely crash). Reason: The processing evaluates xy and yx. In general the median is not biased by outliers as we have it using the mean (Stack All ). ref. Processing Parameters, page 15, and: upper bound median: min=1.05 max=100, default 1.2 upper bound median: min=1.05 max=100, default 1.2 Values from lower 1.2 to upper 1.2 will include tensors where z x y and z y x are inside this range (z / 1.2 and z * 1.2). If Z is Gauss distributed this is a valid assumption and will remove outliers. If Z has a different distribution the result can be biased. The same applies if you select 2.2 and 1.6 which is

23 Chapter 3. not symmetric as above. Here you give preference to lower resistivity values. Fig 3.1: Create processing Make sure that Window Length (top right) and Window Length FFT are same = last slice even if less available

24 Chapter 3. All processing modules will be processed in the order as they appear. Run a Processing Select a processing file and select a meas directory and press Run Job. procmt_cmds You can set parameters using the command line shell, for example -queue_len 8. With additional disable -procmt_cmds false the procmt_cmd XML module will be completely ignored if existing. Otherwise (if existing) values will be taken from XML file.

25 4. C S A M T General Using CSAMT data the processing must fit the transmitter data. You can NOT FILTER the data. The amount of samples for each dipole transmitted is calculated by the dipole TX time [s]. The TX time must be the same for each dipole. All CSAMT processings for each TX dipole must contain the same MT/CSAMT processing types. If you configure a median processing, median processing must appear in all configured jobs. The CSAMT processings also must contain the same processing comments in each XML in order that ProcMT combines all timeseries from all transmissions and all sampling rates to a single EDI file. Use the Merge inputs only if you make EMAP. In this case your site directory may contain more than one xml meas file.

26 Chapter 4. Run a CSAMT Processing A good naming convention for your CSAMT processing is: csamt_2048hz_16s_16s. That is processing_txfreq_dipoletime1_dipoletime2. For tensor CSAMT dipole time 1 must be equal dipole time 2. Figure 4.1 same processing comments, different processings

27 Chapter 4. If the processing was already configured during data recording (a name like mt_std_1024 or csamt_1hz_64s_64s) and a corresponding XML file with same name is found in the processings directory you simply double click either on the Site or on a specific run.

28 5. L i n u x On a shell procmt_createjob ts/site_132_134_136_138/*/*xml >jobs/132.xml

29 6. Te r m s Anisotropy: Zyx/Zxy, assumed do be greater 1. An anisotropy limit of 1.8 prevents Swift or rotation of tensors with an anisotropy less than 1.8. This prevents from unnecessary rotation if required. EMAP: is a configuration where multiple electric fields are mapped to one magnetic field. Limit Frequency ranges: In general the frequency spectra wants to be limited. Especially at the lower end spectral density is low and may not deliver good results. In this case it can be better to limit the lower end and to continue with the next lower band (sampling rate). If you filter only by 8 x instead of 32 x and additionally limit the fft range 0.8 to 0.2 you get almost constant bandwidth similar to cascade decimation. odd harmonics: when transmitting rectangular waves you see in the resulting spectra odd harmonics of the base frequency. They can be used to CSAMT processing as well. However: the 3 rd harmonic (=

30 Chapter 6. 2 nd odd) may be already too small to deliver good results in the processing.

31 7. A p p e n d i x Example XML config files 2 ADU EMAP Here you see 2 ADU recording in EMAP mode, 71 is for E only, 69 for E & H. This config file needs channel number and serial number. <?xml version="1.0" encoding="utf-8"?> <!-- this is an emap file for a multiple ADU (many systems in use--> <!-- this file contains serial numbers! -- > <!-- the " Merge Inputs" must be active -- > <!-- the file contains TWO mt_site_config so directory name must be Site_1_2--> <channel_ map> <mt_ site_ config> <Ex>5</Ex> <Ex_acin>-1</Ex_acin> <Ey>6</Ey> <Ey_acin>-1</Ey_acin> <Hx>7</Hx> <Hx_acin>-1</Hx_acin> <Hy>8</Hy> <Hy_acin>-1</Hy_acin> <Hz>-1</Hz> <Hz_acin>-1</Hz_acin> <REx>-1</REx> <REx_acin>-1</REx_acin> <REy>-1</REy> <REy_acin>-1</REy_acin> <RHx>-1</RHx> <RHx_acin>-1</RHx_acin> <RHy>-1</RHy> <RHy_acin>-1</RHy_acin> <RHz>-1</RHz>

32 Chapter 7. <RHz_acin>-1</RHz_acin> <local_ adu_ serial> 0</ local_ adu_ serial> <local_ site_ no> 0</ local_ site_ no> <remote_ adu_ serial> 0</ remote_ adu_ serial> <ser_ex>71</ser_ex> <ser_ey>71</ser_ey> <ser_hx>69</ser_hx> <ser_hy>69</ser_hy> <ser_hz>0</ser_hz> <ser_rex>0</ser_rex> <ser_rey>0</ser_rey> <ser_rhx>0</ser_rhx> <ser_rhy>0</ser_rhy> <ser_rhz>0</ser_rhz> <site_no>-1</site_no> <sitedoc_ channel_ no> 0</ sitedoc_ channel_ no> </ mt_ site_ config> <mt_ site_ config> <Ex>5</Ex> <Ex_acin>-1</Ex_acin> <Ey>6</Ey> <Ey_acin>-1</Ey_acin> <Hx>7</Hx> <Hx_acin>-1</Hx_acin> <Hy>8</Hy> <Hy_acin>-1</Hy_acin> <Hz>-1</Hz> <Hz_acin>-1</Hz_acin> <REx>-1</REx> <REx_acin>-1</REx_acin> <REy>-1</REy> <REy_acin>-1</REy_acin> <RHx>-1</RHx> <RHx_acin>-1</RHx_acin> <RHy>-1</RHy> <RHy_acin>-1</RHy_acin> <RHz>-1</RHz> <RHz_acin>-1</RHz_acin> <local_ adu_ serial> 0</ local_ adu_ serial> <local_ site_ no> 0</ local_ site_ no> <remote_ adu_ serial> 0</ remote_ adu_ serial> <ser_ex>69</ser_ex> <ser_ey>69</ser_ey> <ser_hx>69</ser_hx> <ser_hy>69</ser_hy> <ser_hz>0</ser_hz> <ser_rex>0</ser_rex> <ser_rey>0</ser_rey> <ser_rhx>0</ser_rhx> <ser_rhy>0</ser_rhy> <ser_rhz>0</ser_rhz> <site_no>-1</site_no> <sitedoc_ channel_ no> 0</ sitedoc_ channel_ no> </ mt_ site_ config> </ channel_ map>

33 Chapter 7. Scripts #!/bin/bash mpwd =`pwd `; echo $mpwd cd jobs for pdir in `ls `; do echo ' preparing' $pdir cd $mpwd procmt_ main./jobs/$pdir done

34 Chapter 7.