Hi MSNoise group,
I was faced with a new problem in computing MWCS with multiprocess operation. I defined different Mwcs_Wlen for different filters. Then I started 10 instances of the code using command “msnoise -t 10 compute-mwcs”. When the Mwcs_Wlen was 10s, the code was working properly. But when the Mwcs_Wlen was more than 10s such as 20s, 30s, the 10 processes occupied all my 40 physical cores. And the computational speed became very slow.
I don’t know what happened, so I stoped the code and reseted the DTT jobs. And I recalculated them in one process using command “msnoise compute-mwcs”.I found when computed 10s window length, the process occupied 100% of one core. But when computed others, the process occupied 4000% of one core.
The cpu utilization percentage was obtained from the top command in Linux，and I computed with msnoise 1.5.1.
Looking forward to know the reason of this problem.
Thanks!
Weiyu
Hi Thomas,
Thanks for your reply! And you are right，I adjusted the parameters of my filter according to your reply. It worked without any warnings and the outputs were normal. Thank you very much!
And the width between mwcs_low and mwcs_high is at least 0.03 in my own project.I hope this will be helpful to other users who have the same problems as me(￣.￣)
weiyu
Hi MSNoise group,
I recently ran into a problem when compute MWCS. I have 9 filters in my project, when I ran “msnoise compute-mwcs”,it worked successfully, but when calculated filter 8 and filter 9 something happened, the parameters were “0.04 0.05 0.0667 0.1” and “0.025 0.033 0.05 0.1” respectively. The code reported some Warnings:
2019-06-10 15:39:13 [INFO] We will recompute all MWCS based on the new REF for SX.YDH:SX.YSG
2019-06-10 15:39:13 [DEBUG] Ready to compute
2019-06-10 15:39:14 [INFO] There are MWCS jobs for some days to recompute for CQ.CHK:CQ.CHS
2019-06-10 15:39:14 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 05 days
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/scipy/optimize/minpack.py:779: OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:349: RuntimeWarning: invalid value encountered in double_scalars
e = np.sum((phi - m * v) ** 2) / (np.size(v) - 1)
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:349: RuntimeWarning: divide by zero encountered in double_scalars
e = np.sum((phi - m * v) ** 2) / (np.size(v) - 1)
2019-06-10 15:39:16 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 15 days
2019-06-10 15:39:16 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 30 days
2019-06-10 15:39:17 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-01-01 - 45 days
With these warnings, the code still worked, and then new warnings occured:
2019-06-10 15:42:33 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 15 days
2019-06-10 15:42:33 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 30 days
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:327: RuntimeWarning: divide by zero encountered in true_divide
w = 1.0 / (1.0 / (coh[index_range] ** 2) - 1.0)
/data1/dell/Software/anaconda3/envs/msnoise/lib/python3.6/site-packages/msnoise/move2obspy.py:327: RuntimeWarning: invalid value encountered in true_divide
w = 1.0 / (1.0 / (coh[index_range] ** 2) - 1.0)
2019-06-10 15:42:34 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 45 days
2019-06-10 15:42:34 [DEBUG] Processing MWCS for: CQ_CHK_CQ_CHS.ZZ.08 - 2016-04-25 - 60 days
And the code was still running, the output was as follows:
-3.300000000000000000e+02 -7.905952284653005258e+00 nan 9.937050757172795068e-01
-3.250000000000000000e+02 -8.138657988098142226e+00 nan 9.976592601521057091e-01
-3.200000000000000000e+02 -7.632117540664275168e+00 nan 9.894723420074960085e-01
-3.150000000000000000e+02 -4.285274200620775353e+00 nan 8.972202676215104145e-01
-3.100000000000000000e+02 -1.489405237146531613e+00 nan 9.666564114380370443e-01
-3.050000000000000000e+02 -1.821892139101924268e+00 nan 9.604215787607996990e-01
-3.000000000000000000e+02 8.124339913942842273e+00 nan 9.809556426874297408e-01
-2.950000000000000000e+02 1.436912500259247771e+00 nan 9.710551233524897796e-01
-2.900000000000000000e+02 6.558446922684406655e-01 nan 9.932296151167433473e-01
-2.850000000000000000e+02 5.593336746118490188e-01 inf 9.946785904215598162e-01
-2.800000000000000000e+02 1.623208176830735067e-02 inf 9.997050009388863678e-01
-2.750000000000000000e+02 -8.591600654862702413e+00 inf 9.924982060141236406e-01
-2.700000000000000000e+02 5.866839658532229329e+00 nan 9.421224113061066330e-01
-2.650000000000000000e+02 4.197498248311506863e+00 nan 9.094520333511715293e-01
-2.600000000000000000e+02 6.883655670067920340e+00 nan 9.634636129273534966e-01
-2.550000000000000000e+02 6.902610666221385749e+00 nan 9.689572985482811651e-01
-2.500000000000000000e+02 -2.224539350609665789e+00 nan 9.149593016370045628e-01
-2.450000000000000000e+02 -2.569921624096909163e+00 nan 9.193301735266846686e-01
-2.400000000000000000e+02 -1.787990836057278932e+00 inf 9.549244033726577152e-01
-2.350000000000000000e+02 -7.489977845697655123e-01 nan 9.923064807292454326e-01
I don’t know why this problem only occured at filter 8 and 9.
looking forward to know what’s going on
very very thanks!
weiyu
* Python 3.6 w/ latest pull of msnoise from git.
* first run through the complete workflow with on a new project
* CC step seems to complete fine, although I should note I'm not saving the
full CC files
Problem. When I run the stack (msnoise stack -r -m) for the first time
everything just gets flags as in progress, but never actually completes
before the code exits. No stack folder is made, and no data written. I get
one of these messages for each CC-pair;
2019-05-12 04:56:05.085438 msnoise [pid 11389][INFO]: There are STACKS jobs
for some days to recompute for HV.MLOD:HV.TRAD
any idea whats going on?
Thanks!
ashton
--
Ashton F. Flinders, Ph.D
Presidential Management Fellow
U.S. Geological Survey
Hawaiian Volcano Observatory
(808) 224 4367
Hi, msnoise group,
I have trouble for the original data input. I define directly in the data_structure configuration bit using the the format:
STA/YEAR/NET_STA_CHAN_DAY.SAC. For example, the data is SC_EMS_BHE_227.SAC. But, the code told me the “unknown format for file”.
In order to describe, I give 2 figures in the attachment.
Interesting, I use the same data_structure for the other data, it is ok. The sac head files between the two sets of data are different.
Jinwu.
On 2019-05-07 17:08, Thomas Lecocq wrote:
> Dear Navid,
>
> Each column of the image is a color-coded representation of individual daily cross-correlations (amplitude: negative: blue, positive:red).
>
> Thomas
>
> Le 07/05/2019 à 14:27, navid.sirous a écrit :
>
>> Hi Msnoise group,
>>
>> could you please explain more about the command "Interferogram Plot",
>> and what parameters we can check with this command?
>>
>> In the msnoise documentation, it says that "shows an image instead of
>> wiggles"?
>> _______________________________________________
>> MSNoise mailing list
>> MSNoise(a)mailman-as.oma.be
>> http://mailman-as.oma.be/mailman/listinfo/msnoise
> _______________________________________________
> MSNoise mailing list
> MSNoise(a)mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
Thank you for your answer
>From ljw627(a)whu.edu.cn Thu May 9 02:38:03 2019
Return-Path: <ljw627(a)whu.edu.cn>
X-Original-To: msnoise(a)mailman-as.oma.be
Delivered-To: msnoise(a)mailman-as.oma.be
Received: from zg8tmtu5ljy1ljeznc42.icoremail.net
(zg8tmtu5ljy1ljeznc42.icoremail.net [159.65.134.6])
by mailman-as.oma.be (Postfix) with SMTP id BC9A420643
for <msnoise(a)mailman-as.oma.be>; Thu, 9 May 2019 02:38:02 +0000 (UTC)
Received: from [192.168.15.219] (unknown [60.30.77.18])
by email1 (Coremail) with SMTP id AQBjCgD3WKZ9ktNcdidUAA--.42494S2;
Thu, 09 May 2019 10:37:52 +0800 (CST)
From: ljw627 <ljw627(a)whu.edu.cn>
Content-Type: text/plain;
charset=gb2312
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\))
Date: Thu, 9 May 2019 10:37:49 +0800
References: <4D1B0EEF-757C-4195-847E-DC3041815879(a)whu.edu.cn>
To: "msnoise(a)mailman-as.oma.be" <msnoise(a)mailman-as.oma.be>
In-Reply-To: <4D1B0EEF-757C-4195-847E-DC3041815879(a)whu.edu.cn>
Message-Id: <2DC7DE17-F9E4-47B9-A73E-8AE3B05FBEC2(a)whu.edu.cn>
X-Mailer: Apple Mail (2.3445.4.7)
X-CM-TRANSID: AQBjCgD3WKZ9ktNcdidUAA--.42494S2
X-Coremail-Antispam: 1UD129KBjDUn29KB7ZKAUJUUUUU529EdanIXcx71UUUUU7v73
VFW2AGmfu7bjvjm3AaLaJ3UjIYCTnIWjp_UUUYQ7k0a2IF6F4UM7kC6x804xWl14x267AK
xVWUJVW8JwAFc2x0x2IEx4CE42xK8VAvwI8IcIk0rVWrJVCq3wAFIxvE14AKwVWUJVWUGw
A2ocxC64kIII0Yj41l84x0c7CEw4AK67xGY2AK021l84ACjcxK6xIIjxv20xvE14v26w1j
6s0DM28EF7xvwVC0I7IYx2IY6xkF7I0E14v26r4UJVWxJr1l84ACjcxK6I8E87Iv67AKxV
W0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_GcCE3s1lnxkEFVAIw20F6cxK64vIFxWl
e2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx0E2Ix0cI
8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkEbVWUJVW8JwAC
jcxG0xvY0x0EwIxGrwCY02Avz4vE14v_Gr1l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x
0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2
zVA0820Y0xCF62I06xkIj41lx4CE17CEb7AF67AKxVWUJVWUXwCIc40Y0x0EwIxGrwCI42
IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVWUJVW8JwCI42IY
6xAIw20EY4v20xvaj40_Wr1j6rW3Jr1lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z2
80aVCY1x0267AKxVWUJVW8JbIYCTnIWIevJa73UjIFyTuYvjxUxlksDUUUU
X-CM-SenderInfo: qsqrliaqsrkiqqqvq4lkxovvfxof0/
Subject: [MSNoise] Data structure
X-BeenThere: msnoise(a)mailman-as.oma.be
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Python Package for Monitoring Seismic Velocity Changes using Ambient
Seismic Noise <msnoise.mailman-as.oma.be>
List-Unsubscribe: <http://mailman-as.oma.be/mailman/options/msnoise>,
<mailto:msnoise-request@mailman-as.oma.be?subject=unsubscribe>
List-Archive: <http://mailman-as.oma.be/pipermail/msnoise/>
List-Post: <mailto:msnoise@mailman-as.oma.be>
List-Help: <mailto:msnoise-request@mailman-as.oma.be?subject=help>
List-Subscribe: <http://mailman-as.oma.be/mailman/listinfo/msnoise>,
<mailto:msnoise-request@mailman-as.oma.be?subject=subscribe>
X-List-Received-Date: Thu, 09 May 2019 02:38:03 -0000
>=20
> Hi, msnoise group,
>=20
> I have trouble for the original data input. I define directly in the =
data_structure configuration bit using the the format:=20
>=20
> STA/YEAR/NET_STA_CHAN_DAY.SAC. For example, the data is =
SC_EMS_BHE_227.SAC. But, the code told me the =A1=B0unknown format for =
file=A1=B1.
>=20
> In order to describe, I give 2 figures in the attachment.=20
>=20
> Interesting, I use the same data_structure for the other data, it is =
ok. The sac head files between the two sets of data are different.=20
>=20
> Jinwu.
> =20
> <3.jpeg><da.jpeg>
Hi Msnoise group,
could you please explain more about the command "Interferogram Plot",
and what parameters we can check with this command?
In the msnoise documentation, it says that "shows an image instead of
wiggles"?
>From Thomas.Lecocq(a)seismology.be Tue May 7 12:38:32 2019
Return-Path: <Thomas.Lecocq(a)seismology.be>
X-Original-To: msnoise(a)mailman-as.oma.be
Delivered-To: msnoise(a)mailman-as.oma.be
Received: from 2.mo178.mail-out.ovh.net (2.mo178.mail-out.ovh.net
[46.105.39.61])
by mailman-as.oma.be (Postfix) with ESMTP id 2FF4A23C99
for <msnoise(a)mailman-as.oma.be>; Tue, 7 May 2019 12:38:32 +0000 (UTC)
Received: from player793.ha.ovh.net (unknown [10.109.146.175])
by mo178.mail-out.ovh.net (Postfix) with ESMTP id CDA145EA6F
for <msnoise(a)mailman-as.oma.be>; Tue, 7 May 2019 14:38:31 +0200 (CEST)
Received: from asktom.be (unknown [62.119.166.9])
(Authenticated sender: tom(a)asktom.be)
by player793.ha.ovh.net (Postfix) with ESMTPSA id 260E4593F240
for <msnoise(a)mailman-as.oma.be>; Tue, 7 May 2019 12:38:31 +0000 (UTC)
To: msnoise(a)mailman-as.oma.be
References: <fee9cac5f6f6288410da0216ce5f13ea(a)ut.ac.ir>
From: Thomas Lecocq <Thomas.Lecocq(a)seismology.be>
Message-ID: <e6fb9c40-bae7-0773-66b8-3dfbae7346bf(a)seismology.be>
Date: Tue, 7 May 2019 14:38:29 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
Thunderbird/60.6.1
MIME-Version: 1.0
In-Reply-To: <fee9cac5f6f6288410da0216ce5f13ea(a)ut.ac.ir>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: fr
X-Ovh-Tracer-Id: 8955126384448180510
X-VR-SPAMSTATE: OK
X-VR-SPAMSCORE: 0
X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeduuddrkedtgdehjecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfqggfjpdevjffgvefmvefgnecuuegrihhlohhuthemucehtddtnecu
Subject: Re: [MSNoise] Interferogram Plot
X-BeenThere: msnoise(a)mailman-as.oma.be
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Python Package for Monitoring Seismic Velocity Changes using Ambient
Seismic Noise <msnoise.mailman-as.oma.be>
List-Unsubscribe: <http://mailman-as.oma.be/mailman/options/msnoise>,
<mailto:msnoise-request@mailman-as.oma.be?subject=unsubscribe>
List-Archive: <http://mailman-as.oma.be/pipermail/msnoise/>
List-Post: <mailto:msnoise@mailman-as.oma.be>
List-Help: <mailto:msnoise-request@mailman-as.oma.be?subject=help>
List-Subscribe: <http://mailman-as.oma.be/mailman/listinfo/msnoise>,
<mailto:msnoise-request@mailman-as.oma.be?subject=subscribe>
X-List-Received-Date: Tue, 07 May 2019 12:38:32 -0000
Dear Navid,
Each column of the image is a color-coded representation of individual
daily cross-correlations (amplitude: negative: blue, positive:red).
Thomas
Le 07/05/2019 à 14:27, navid.sirous a écrit :
> Hi Msnoise group,
>
> could you please explain more about the command "Interferogram Plot",
> and what parameters we can check with this command?
>
> In the msnoise documentation, it says that "shows an image instead of
> wiggles"?
> _______________________________________________
> MSNoise mailing list
> MSNoise(a)mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
Hi Gaelle,
It's not really and MSNoise problem. I suggest you ask the obspy mailing
list or the obspy Gitter chat for help, and please make sure to include
the traceback of your error.
Thomas
On 25/04/2019 11:51, gaelle vanessa Nana wrote:
> hello,
> I use this python code to download data, but unfortunatelly, i cannot
> download. I need your help.
>
>
> import obspy
> from obspy.clients.fdsn.mass_downloader import RectangularDomain, \
> Restrictions, MassDownloader
>
> # Rectangular domain containing parts of southern Germany.
> domain = RectangularDomain(minlatitude=2, maxlatitude=12,
> minlongitude=6, maxlongitude=16)
>
> restrictions = Restrictions(
> # Get data for a whole year.
> starttime=obspy.UTCDateTime(2006, 1, 1),
> endtime=obspy.UTCDateTime(2007, 1, 1),
> # Chunk it to have one file per day.
> chunklength_in_sec=86400,
> # Considering the enormous amount of data associated with continuous
> # requests, you might want to limit the data based on SEED identifiers.
> # If the location code is specified, the location priority list is not
> # used; the same is true for the channel argument and priority list.
> network="XB", station="CM*", location="", channel="BH*",
> # The typical use case for such a data set are noise correlations where
> # gaps are dealt with at a later stage.
> reject_channels_with_gaps=False,
> # Same is true with the minimum length. All data might be useful.
> minimum_length=0.0,
> # Guard against the same station having different names.
> minimum_interstation_distance_in_m=100.0)
>
> # Restrict the number of providers if you know which serve the desired
> # data. If in doubt just don't specify - then all providers will be
> # queried.
> mdl = MassDownloader(providers=["IRIS" ])
> mdl.download(domain, restrictions, mseed_storage="waveforms",
> stationxml_storage="stations")
> _______________________________________________
> MSNoise mailing list
> MSNoise(a)mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
--
Dr. Thomas Lecocq
Geologist - Seismologist
Seismology - Gravimetry
Royal Observatory of Belgium
*
* * * * *
* * * *
---------
http://www.seismology.behttp://msnoise.orghttp://twitter.com/#!/Seismologie_behttps://www.facebook.com/seismologie.be
Dear Msnoise group,
I can run some of plot commands successfully, such as plotting of dvv,
but whenever plot "interferogram" and " ccftime" between my stations my
plots are empty something like attachments. Whenever I use the command
msnoise plot mwcs DHR LIN -m1, I find an error of below.
Traceback (most recent call last):
File "/usr/local/bin/msnoise", line 9, in <module>
load_entry_point('msnoise==1.5.1', 'console_scripts', 'msnoise')()
File
"/usr/local/lib/python3.5/dist-packages/msnoise/scripts/msnoise.py",
line 717, in run
cli(obj={})
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 764,
in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 717,
in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line
1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line
1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 956,
in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 555,
in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/click/decorators.py",
line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File
"/usr/local/lib/python3.5/dist-packages/msnoise/scripts/msnoise.py",
line 597, in mwcs
main(sta1, sta2, filterid, comp, mov_stack, show, outfile)
File "/usr/local/lib/python3.5/dist-packages/msnoise/plots/mwcs.py",
line 58, in main
station1 = get_station(db, station1[0], station1[1])
IndexError: list index out of range
I will grateful if you help me
regards,
Navid Sirous
University of Tehran
