hello MSnoise group
i have a problem,after installing Msnoise, i tried to run MSNOISE TEST but
there is some problem.
I tried to install pyqt, pyside but no success.
Please, if someone have an idea to solve this problem.
cheers
(base) vanessa@NANA:~$ msnoise test
2019-06-28 13:51:35 [DEBUG] CACHEDIR=/home/vanessa/.cache/matplotlib
2019-06-28 13:51:35 [DEBUG] Using fontManager instance from
/home/vanessa/.cache/matplotlib/fontlist-v310.json
2019-06-28 13:51:36 [DEBUG] Loaded backend qt5agg version unknown.
2019-06-28 13:51:36 [DEBUG] Loaded backend qt5agg version unknown.
2019-06-28 13:51:36 [DEBUG] Loaded backend Qt5Agg version unknown.
2019-06-28 13:51:36 [DEBUG] Loaded backend agg version unknown.
2019-06-28 13:51:36 [DEBUG] Loaded backend agg version unknown.
Directory is not empty, can't run tests here!
Hello MSnoise Group,
I am trying to use windows version of msnoise with mysql but, I am having
problem during data import. Being a beginer at both python and mysql.
All steps are clear till creating empty database in mysql but, after that I
am facing problem even after reading documentation for several times. Even,
I tried reading codes of Data_structure, Scan_archive and even one extra
code given for custom data structure and station population but could not
get anything.
I can open the admin interface but there is no data to show. The sac file i
am trying to import is attached herewith.
May i get some demo for the steps involved during data loading and
populating the stations.
Thanking you
yours sincerely,
Shiv Mangal Gupta
IIT (ISM), Dhanbad , India
ᐧ
Dear Joseph,
Thanks for using MSNoise.
If there is only one core running it means probably that you have only 1
day of data to correlate. The multi-thread/processing of MSNoise works
based on "day" jobs, which are independent from the other days. With 1
day of data for N stations, you'll have N (N-1)/2 jobs "T"odo in the
database, but they will all be processed by the same thread.
Best regards,
Thomas
Le 24/06/2019 à 02:32, Thangraj, Joseph a écrit :
> Dear Thomas,
> I am using MSNoise to process a 250 Hz sample rate data set with
> 131 stations. I was trying to run my cross-correlations (compute-cc)
> in parallel to make the computations faster. However, inspite of using
> msnoise -t 14 compute-cc on a 70 cpu computer (256 Gb ram), the
> cross-correlations still run on one cpu. The parallel instances of
> 'compute-cc' start and finish without doing any jobs. Is it because
> the first instance sets all the jobs as 'I' meaning in progress, or
> should I change something in my config to make it run in parallel. I
> would appreciate your input about this problem. Please let me know if
> you require any additional information.
> Have a good week ahead!
>
> Joseph Thangraj
>
> Graduate student
>
> Department of Geosciences
>
> Baylor University
>
Hi MSNoise group,
I run a shell script which downloads data for each day, processes it to
produce an updated DVV plot, then removes the original data to save space
on my computer. However, MSNoise does not perform 'compute-cc', until I
manually remove all files in the 'Database-Data Availability' tab in the
MSNoise admin GUI (FileNotFoundError: [Errno 2] No such file or directory) .
Is there a command I can run in my script to not have to make a manual
adjustment to the data availability tab?
Regards,
Caitlin Smith,
University of Auckland
csmi310(a)aucklanduni.ac.nz
Hello MSNoise group,
I faced with a problem with 'msnoise -t 2 scan-archive'.
Below are the responses when I run 'msnoise -t 2 scan-archive' and with
'--init'.
PS>msnoise -t 2 scan-archive --init
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
2019-06-13 20:38:41 [INFO] *** Starting: Scan Archive ***
2019-06-13 20:38:41 [INFO] Initializing (should be run only once)
2019-06-13 20:38:41 [INFO] Will work on 2 thread(s)
2019-06-13 20:38:41 [INFO] Will search for files between 2017-01-01 and
2019-12-30
2019-06-13 20:38:41 [DEBUG] Will search for channels: ['*']
2019-06-13 20:38:41 [DEBUG] Data Folder: E:\00.msnoise\02.SeimicData
2019-06-13 20:38:41 [DEBUG] Folders to glob:
E:\00.msnoise\02.SeimicData\2017/KG/TJN/*.D,E:\00.msnoise\02.SeimicData\2017/KS/JEJB/*.D,E:\00.msnoise\02.SeimicData\2017/KS/YOCB/*.D,E:\00.msnoise\02.SeimicData\2018/KG/TJN/*.D,E:\00.msnoise\02.SeimicData\2018/KS/JEJB/*.D,E:\00.msnoise\02.SeimicData\2018/KS/YOCB/*.D,E:\00.msnoise\02.SeimicData\2019/KG/TJN/*.D,E:\00.msnoise\02.SeimicData\2019/KS/JEJB/*.D,E:\00.msnoise\02.SeimicData\2019/KS/YOCB/*.D
2019-06-13 20:38:41 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KG/TJN\HHE.D folder: 1
2019-06-13 20:38:41 [INFO] E:\00.msnoise\02.SeimicData\2019/KG/TJN\HHE.D:
Started
2019-06-13 20:38:42 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHE.D folder: 4
2019-06-13 20:38:42 [INFO] E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHE.D:
Started
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
2019-06-13 20:38:54 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHN.D folder: 3
2019-06-13 20:38:54 [INFO] E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHN.D:
Started
2019-06-13 20:38:55 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHZ.D folder: 3
2019-06-13 20:38:55 [INFO] E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHZ.D:
Started
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
2019-06-13 20:39:12 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHE.D folder: 1
2019-06-13 20:39:12 [INFO] E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHE.D:
Started
2019-06-13 20:39:12 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHN.D folder: 1
2019-06-13 20:39:12 [INFO] E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHN.D:
Started
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
2019-06-13 20:39:25 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHZ.D folder: 1
2019-06-13 20:39:25 [INFO] E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHZ.D:
Started
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
2019-06-13 20:39:34 [INFO] *** Finished: Scan Archive ***
2019-06-13 20:39:34 [INFO] It took 52.61 seconds
PS>msnoise -t 2 scan-archive
c:\users\jeon\anaconda3\lib\site-packages\pymysql\cursors.py:170: Warning:
(1366, "Incorrect string value: '\\xB4\\xEB\\xC7\\xD1\\xB9\\xCE...' for
column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
2019-06-17 21:09:28 [INFO] *** Starting: Scan Archive ***
2019-06-17 21:09:28 [INFO] Will work on 2 thread(s)
2019-06-17 21:09:28 [INFO] Will search for files between 2017-01-01 and
2019-12-30
2019-06-17 21:09:28 [DEBUG] Will search for channels: ['*']
2019-06-17 21:09:28 [DEBUG] Data Folder: E:\00.msnoise\02.SeimicData
2019-06-17 21:09:28 [DEBUG] Folders to glob:
E:\00.msnoise\02.SeimicData\2017/KG/TJN/*.D,E:\00.msnoise\02.SeimicData\2017/KS/JEJB/*.D,E:\00.msnoise\02.SeimicData\2017/KS/YOCB/*.D,E:\00.msnoise\02.SeimicData\2018/KG/TJN/*.D,E:\00.msnoise\02.SeimicData\2018/KS/JEJB/*.D,E:\00.msnoise\02.SeimicData\2018/KS/YOCB/*.D,E:\00.msnoise\02.SeimicData\2019/KG/TJN/*.D,E:\00.msnoise\02.SeimicData\2019/KS/JEJB/*.D,E:\00.msnoise\02.SeimicData\2019/KS/YOCB/*.D
2019-06-17 21:09:28 [DEBUG] Will search for files more recent than
2019-06-16T12:09:28.717502Z
2019-06-17 21:09:28 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KG/TJN\HHE.D folder: 0
2019-06-17 21:09:28 [DEBUG] Will search for files more recent than
2019-06-16T12:09:28.717502Z
2019-06-17 21:09:28 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHE.D folder: 0
2019-06-17 21:09:28 [DEBUG] Will search for files more recent than
2019-06-16T12:09:28.717502Z
2019-06-17 21:09:28 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHN.D folder: 0
2019-06-17 21:09:28 [DEBUG] Will search for files more recent than
2019-06-16T12:09:28.717502Z
2019-06-17 21:09:28 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/JEJB\HHZ.D folder: 0
2019-06-17 21:09:28 [DEBUG] Will search for files more recent than
2019-06-16T12:09:28.717502Z
2019-06-17 21:09:28 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHE.D folder: 0
2019-06-17 21:09:28 [DEBUG] Will search for files more recent than
2019-06-16T12:09:28.717502Z
2019-06-17 21:09:28 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHN.D folder: 0
2019-06-17 21:09:28 [DEBUG] Will search for files more recent than
2019-06-16T12:09:28.717502Z
2019-06-17 21:09:28 [DEBUG] Files detected in
E:\00.msnoise\02.SeimicData\2019/KS/YOCB\HHZ.D folder: 0
2019-06-17 21:09:28 [INFO] *** Finished: Scan Archive ***
2019-06-17 21:09:28 [INFO] It took 0.16 seconds
I have don't idea why it do not work.
Please give me a help.
Thank you!
Sincerely,
YoungJun Jeon
Hi MSNoise group,
I was faced with a new problem in computing MWCS with multiprocess operation. I defined different Mwcs_Wlen for different filters. Then I started 10 instances of the code using command “msnoise -t 10 compute-mwcs”. When the Mwcs_Wlen was 10s, the code was working properly. But when the Mwcs_Wlen was more than 10s such as 20s, 30s, the 10 processes occupied all my 40 physical cores. And the computational speed became very slow.
I don’t know what happened, so I stoped the code and reseted the DTT jobs. And I recalculated them in one process using command “msnoise compute-mwcs”.I found when computed 10s window length, the process occupied 100% of one core. But when computed others, the process occupied 4000% of one core.
The cpu utilization percentage was obtained from the top command in Linux，and I computed with msnoise 1.5.1.
Looking forward to know the reason of this problem.
Thanks!
Weiyu
Hi Thomas,
Thanks for your reply! And you are right，I adjusted the parameters of my filter according to your reply. It worked without any warnings and the outputs were normal. Thank you very much!
And the width between mwcs_low and mwcs_high is at least 0.03 in my own project.I hope this will be helpful to other users who have the same problems as me(￣.￣)
weiyu
Hi MSNoise group,
I recently ran into a problem when compute MWCS. I have 9 filters in my project, when I ran “msnoise compute-mwcs”,it worked successfully, but when calculated filter 8 and filter 9 something happened, the parameters were “0.04 0.05 0.0667 0.1” and “0.025 0.033 0.05 0.1” respectively. The code reported some Warnings:
2019-06-10 15:39:13 [INFO] We will recompute all MWCS based on the new REF for SX.YDH:SX.YSG
2019-06-10 15:39:13 [DEBUG] Ready to compute
2019-06-10 15:39:14 [INFO] There are MWCS jobs for some days to recompute for CQ.CHK:CQ.CHS
2019-06-10 15:39:14 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 05 days
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/scipy/optimize/minpack.py:779: OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:349: RuntimeWarning: invalid value encountered in double_scalars
e = np.sum((phi - m * v) ** 2) / (np.size(v) - 1)
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:349: RuntimeWarning: divide by zero encountered in double_scalars
e = np.sum((phi - m * v) ** 2) / (np.size(v) - 1)
2019-06-10 15:39:16 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 15 days
2019-06-10 15:39:16 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 30 days
2019-06-10 15:39:17 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 45 days
With these warnings, the code still worked, and then new warnings occured:
2019-06-10 15:42:33 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 15 days
2019-06-10 15:42:33 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 30 days
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:327: RuntimeWarning: divide by zero encountered in true_divide
w = 1.0 / (1.0 / (coh[index_range] ** 2) - 1.0)
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:327: RuntimeWarning: invalid value encountered in true_divide
w = 1.0 / (1.0 / (coh[index_range] ** 2) - 1.0)
2019-06-10 15:42:34 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 45 days
2019-06-10 15:42:34 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 60 days
And the code was still running, the output was as follows:
-3.300000000000000000e+02 -7.905952284653005258e+00 nan 9.937050757172795068e-01
-3.250000000000000000e+02 -8.138657988098142226e+00 nan 9.976592601521057091e-01
-3.200000000000000000e+02 -7.632117540664275168e+00 nan 9.894723420074960085e-01
-3.150000000000000000e+02 -4.285274200620775353e+00 nan 8.972202676215104145e-01
-3.100000000000000000e+02 -1.489405237146531613e+00 nan 9.666564114380370443e-01
-3.050000000000000000e+02 -1.821892139101924268e+00 nan 9.604215787607996990e-01
-3.000000000000000000e+02 8.124339913942842273e+00 nan 9.809556426874297408e-01
-2.950000000000000000e+02 1.436912500259247771e+00 nan 9.710551233524897796e-01
-2.900000000000000000e+02 6.558446922684406655e-01 nan 9.932296151167433473e-01
-2.850000000000000000e+02 5.593336746118490188e-01 inf 9.946785904215598162e-01
-2.800000000000000000e+02 1.623208176830735067e-02 inf 9.997050009388863678e-01
-2.750000000000000000e+02 -8.591600654862702413e+00 inf 9.924982060141236406e-01
-2.700000000000000000e+02 5.866839658532229329e+00 nan 9.421224113061066330e-01
-2.650000000000000000e+02 4.197498248311506863e+00 nan 9.094520333511715293e-01
-2.600000000000000000e+02 6.883655670067920340e+00 nan 9.634636129273534966e-01
-2.550000000000000000e+02 6.902610666221385749e+00 nan 9.689572985482811651e-01
-2.500000000000000000e+02 -2.224539350609665789e+00 nan 9.149593016370045628e-01
-2.450000000000000000e+02 -2.569921624096909163e+00 nan 9.193301735266846686e-01
-2.400000000000000000e+02 -1.787990836057278932e+00 inf 9.549244033726577152e-01
-2.350000000000000000e+02 -7.489977845697655123e-01 nan 9.923064807292454326e-01
I don’t know why this problem only occured at filter 8 and 9.
looking forward to know what’s going on
very very thanks!
weiyu
