Fwd: Re: Stuck in Compute_cc step - MSNoise

Sébastien Carniato

15 May 15 May

11:07 a.m.

New subject: Stuck in Compute_cc step

Hi Thomas, before talking about my problems, I’d like to introduce my data, and the configurator’s “configuration” I use. I study mining data. The scale is totally different from the initial purpose of Msnoise, and I think is one of the reasons of a lot of problem I’m facing.. There are 3 stations, which are around 200 meters away from each other, and at the same altitude. They record data with a 200 Hz samplerate. In my database, I’ve chosen for each station 2 days of recording (the end of the first day does not correspond exactly to the beginning of the second day), with X,Y,Z channels. The length of each records is approximatively 24 hours. Here’s the structure of my database folder (all files are SAC files) : LF1.AN..X.2011.003 LF1.AN..Y.2011.003 LF1.AN..Z.2011.003 LF1 LF1.AN..X.2011.004 LF1.AN..Y.2011.004 LF1.AN..Z.2011.004 LF2.AN..X.2011.003 LF2.AN..Y.2011.003 LF2.AN..Z.2011.003 AN LF2 LF2.AN..X.2011.004 LF2.AN..Y.2011.004 LF2.AN..Z.2011.004 LF3.AN..X.2011.003 LF3.AN..Y.2011.003 LF3 LF3.AN..Z.2011.003 LF3.AN..X.2011.004 LF3.AN..Y.2011.004 LF3.AN..Z.2011.004 The configurator I use (only the inserts which are (I guess) important for compute_cc : startdate : 1970-01-01 enddate : 2100-01-01 analysis duration : 86400 cc_sampling_rate : 200 resampling_method : Decimate decimation_factor : 2 preprocess_lowpass : 1 preprocess_highpass : 80 maxlag : 120 corr_duration : 1800 windsorizing : 3 crondays : -1 autocorr : N keep_all : Y keep_days : Y ref_begin : -100 ref_end : 0 export_format : BOTH sac_format : doublets The filter configuration I use : low : 10 mwcs_low : 10 high : 80 mwcs_high : 80 rms_treshold : 0 mwcs_len : 10 mwcs_step : 5 My problems : The folder CROSS_CORRELATIONS is created, as well as the STACK folder. Problem 1 : When I plot the mseed files in CROSS_CORRELATION, i get nothing… the files are not empty (324 Kio), but there’s nothing in the plot window. Problem 2 : When I plot the SAC (or MSEED) file in STACKS/00/001_DAYS folder, here is what I get : It’s indeed a symmetric signal… which looks more like an autocorrelation, doesn’t it ? Here’s what I get if I zoom on it : Thanks for your help Thomas. Sébastien. PS : I had to change for each file in its header, the name of the station, and the component, which were false (thanks to the mining data). Do you think it can be the source of any bug ? On 14 mai 2014, at 09:19, Thomas Lecocq <thomas.lecocq(a)oma.be> wrote:

...

Sébastien, (FW to the ML, as this could be useful for many): Le 13/05/2014 16:10, Sébastien Carniato a écrit :

Hi Thomas, thank you for your quick answer ! Indeed, it works when I use only Decimate. I have to check again, but I think indeed that my data stream are using the same sample rate.

You can also check that with the SQLiteManager (group data_availability by "sampling_rate")

I have a bench of other questions : *Question 1 : *So now that compute_cc works :), _I would like to know if the correlations computed are stored_, and in that case where ? I put the output folder on my desk with the configurator, but nothing was created so far...

If you set "keep_all" to "Y", the 30 minutes CC are stored in the CROSS_CORRELATION folder, named "hh_mm.cc", and these are miniseed files (I know, bad naming, this is going to change in the future).

*Question 2 :* My data are not continuous. Indeed they are a lot of days missing, and the maximum length of recording is one day and a half. _Is it a problem for running MSNoise_ ?

Well, MSNoise does correlate M minutes (30 minutes by default) windows, if data is missing, the CC will be corrupted (NaN or infs) and will not be stored. So, normally no problem. BUT, expect results to be quite strange if you have ony like 20% of the day filled with good data.

*Question 3 :* The only way to get a filter is to create one with the sqlite manager. Is it normal ?

No, there is a bug in the Configurator which doesn't show any empty line.

* * *Question 4 : *and the last one : Now when I launch compute_cc, i get the error : 2014-05-13 16:04:21,314 [INFO] *** Starting: Compute CC *** 2014-05-13 16:04:21,354 [INFO] Will compute ZZ 2014-05-13 16:04:21,479 [INFO] New CC Job: 2011-01-18 (6 pairs with 3 stations) AN.LF1:AN.LF1 2014-05-13 16:04:21,595 [DEBUG] Processing pair: AN.LF1 vs AN.LF1 <msnoise_table_def.Station object at 0x4a06ed0> <msnoise_table_def.Station object at 0x4a06ed0> s03compute_cc.py:407: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future trames2hWb[i] = np.zeros(Nfft) s03compute_cc.py:407: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future trames2hWb[i] = np.zeros(Nfft) /home/stag01/Bureau/MSNoise-1.2.3/myCorr.py:68: RuntimeWarning: invalid value encountered in divide corr /= np.real(normFact) Traceback (most recent call last): File "s03compute_cc.py", line 412, in <module> "%Y-%m-%d", time.gmtime(basetime + itranche * min30 / fe)) NameError: name 'basetime' is not defined Do you know what can be the source of the problem ?

Well, first, you should not do Autocorrelation with this version of MSNoise. It'll work normally, but the result is wrong. I'll push a new release very soon that corrects that. But, this problem seems independent. Reading your next mail, it could be related to the file reading part, but it does look strange... Best regards, Thomas _______________________________________________ MSNoise mailing list MSNoise(a)mailman-as.oma.be http://mailman-as.oma.be/mailman/listinfo/msnoise

Reply

Sébastien Carniato

11:59 a.m.

New subject: Stuck in Compute_cc step

Thomas, I changed my configurator which looks like this now : startdate : 1970-01-01 enddate : 2100-01-01 analysis duration : 86400 cc_sampling_rate : 200 resampling_method : Decimate decimation_factor : 2 preprocess_lowpass : 45 preprocess_highpass : 1 maxlag : 120 corr_duration : 1800 windsorizing : 3 crondays : -1 autocorr : N keep_all : Y keep_days : Y ref_begin : -100 ref_end : 0 export_format : BOTH sac_format : doublets And my filter : low : 45 mwcs_low : 45 high : 1 mwcs_high : 1 rms_treshold : 0 mwcs_len : 10 mwcs_step : 5 I’m not sure of getting what you meant by “same for the following”. Now, I get the error : 2014-05-15 11:52:01,333 [DEBUG] Processing pair: AN.LF1 vs AN.LF2 <msnoise_table_def.Station object at 0x4a94950> <msnoise_table_def.Station object at 0x4a9f110> Traceback (most recent call last): File "s03compute_cc.py", line 405, in <module> trame2h[i], Nfft, dt, low, high, plot=False) File "/home/stag01/Bureau/MSNoise-1.2.3/whiten.py", line 58, in whiten low = J[0] - Napod IndexError: index out of bounds Thanks. Sébastien On 15 mai 2014, at 11:22, Thomas Lecocq <thomas.lecocq(a)oma.be> wrote:

...

Sébastien, Le 15/05/2014 11:07, Sébastien Carniato a écrit :

Hi Thomas, before talking about my problems, I’d like to introduce my data, and the configurator’s “configuration” I use. I study mining data. The scale is totally different from the initial purpose of Msnoise, and I think is one of the reasons of a lot of problem I’m facing.. There are 3 stations, which are around 200 meters away from each other, and at the same altitude. They record data with a 200 Hz samplerate. In my database, I’ve chosen for each station 2 days of recording (the end of the first day does not correspond exactly to the beginning of the second day), with X,Y,Z channels. The length of each records is approximatively 24 hours. Here’s the structure of my database folder (all files are SAC files) : LF1.AN..X.2011.003 LF1.AN..Y.2011.003 LF1.AN..Z.2011.003 LF1 LF1.AN..X.2011.004 LF1.AN..Y.2011.004 LF1.AN..Z.2011.004 LF2.AN..X.2011.003 LF2.AN..Y.2011.003 LF2.AN..Z.2011.003 AN LF2 LF2.AN..X.2011.004 LF2.AN..Y.2011.004 LF2.AN..Z.2011.004 LF3.AN..X.2011.003 LF3.AN..Y.2011.003 LF3 LF3.AN..Z.2011.003 LF3.AN..X.2011.004 LF3.AN..Y.2011.004 LF3.AN..Z.2011.004 The configurator I use (only the inserts which are (I guess) important for compute_cc : startdate : 1970-01-01 enddate : 2100-01-01 analysis duration : 86400 cc_sampling_rate : 200 resampling_method : Decimate decimation_factor : 2 preprocess_lowpass : 1 preprocess_highpass : 80 maxlag : 120 corr_duration : 1800 windsorizing : 3 crondays : -1 autocorr : N keep_all : Y keep_days : Y ref_begin : -100 ref_end : 0 export_format : BOTH sac_format : doublets

OK, here there is a big problem ! The lowpass freq should be bigger than the highpass freq. You inverted the two. Then, if you want your CC to occur up to 80Hz, you cannot decimate the data at 100Hz. If you want to process at 100Hz (the output CC will be 100Hz sampled), you have to set up: preprocess_highpass: 1 preprocess_lowpass: 45 (so, somewhat smaller than the resulting Nyquist) Same for the following, if you have 100Hz CC, you'll have information up to 50Hz, not 80. Hope this helps, Thomas

The filter configuration I use : low : 10 mwcs_low : 10 high : 80 mwcs_high : 80 rms_treshold : 0 mwcs_len : 10 mwcs_step : 5 My problems : The folder CROSS_CORRELATIONS is created, as well as the STACK folder. Problem 1 : When I plot the mseed files in CROSS_CORRELATION, i get nothing… the files are not empty (324 Kio), but there’s nothing in the plot window. Problem 2 : When I plot the SAC (or MSEED) file in STACKS/00/001_DAYS folder, here is what I get : It’s indeed a symmetric signal… which looks more like an autocorrelation, doesn’t it ? Here’s what I get if I zoom on it : Thanks for your help Thomas. Sébastien. PS : I had to change for each file in its header, the name of the station, and the component, which were false (thanks to the mining data). Do you think it can be the source of any bug ? On 14 mai 2014, at 09:19, Thomas Lecocq <thomas.lecocq(a)oma.be> wrote: