18 Oct
2018
18 Oct
'18
6:54 p.m.
I get a strange crash part way through my MWCS step (see below), and
compute_MWCS is not finishing. E.g. I have 5 frequency bands, but for bands
2-4 only 1 of 10 station pair MWCS's get calculated, even though all the
data is there in the stacks. I have tried rerunning comute_mwcs by changing
the flag back to 'T' for the station pairs where mwcs did not get
calculated, but it still crashes. This crash is repeatable.
Any thoughts?
(p.s. I also initially tried remaking the stacks, but it crashed at the
same point. The data looks good in the stacks)
-ashton
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py",
line 1139, in _execute_context
context)
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py",
line 450, in do_execute
cursor.execute(statement, parameters)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 165, in execute
result = self._query(query)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 321, in _query
conn.query(q)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 859, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1096, in _execute_command
self._write_bytes(packet)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1048, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away
(BrokenPipeError(32, 'Broken pipe'))")
The above exception was the direct cause of the following exception:
--
Ashton F. Flinders, Ph.D
U.S. Geological Survey
345 Middlefield Road
Menlo Park, CA 94025
(650) 329-5050
19 Oct
19 Oct
8:15 a.m.
Hi Ashton
it seems your MWCS computation took a looooooong time and the MySQL
connection was killed during that time. Can you confirm ?
Thomas
Le 18/10/2018 à 18:54, Flinders, Ashton a écrit :
I get a strange crash part way through my MWCS step
(see below), and
compute_MWCS is not finishing. E.g. I have 5 frequency bands, but for bands
2-4 only 1 of 10 station pair MWCS's get calculated, even though all the
data is there in the stacks. I have tried rerunning comute_mwcs by changing
the flag back to 'T' for the station pairs where mwcs did not get
calculated, but it still crashes. This crash is repeatable.
Any thoughts?
(p.s. I also initially tried remaking the stacks, but it crashed at the
same point. The data looks good in the stacks)
-ashton
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py",
line 1139, in _execute_context
context)
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py",
line 450, in do_execute
cursor.execute(statement, parameters)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 165, in execute
result = self._query(query)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 321, in _query
conn.query(q)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 859, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1096, in _execute_command
self._write_bytes(packet)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1048, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away
(BrokenPipeError(32, 'Broken pipe'))")
The above exception was the direct cause of the following exception:
6:53 p.m.
New subject: [EXTERNAL] Re: MWCS sql crash
Hi Thomas, I actually think this was related to the PR I submitted the
other day. Since I have a mix of stations (some 3-comp some only Z), when
mwcs_compute tried to calculate RR for a station-pair that only had ZZ, and
it couldnt find the reference function it crashed/hanged. Then after a
while hanging it threw the SQL error.
On Thu, Oct 18, 2018 at 11:15 PM Thomas Lecocq <Thomas.Lecocq(a)seismology.be>
wrote:
Hi Ashton
it seems your MWCS computation took a looooooong time and the MySQL
connection was killed during that time. Can you confirm ?
Thomas
Le 18/10/2018 à 18:54, Flinders, Ashton a écrit :
--
Ashton F. Flinders, Ph.D
U.S. Geological Survey
345 Middlefield Road
Menlo Park, CA 94025
(650) 329-5050
I get a strange crash part way through my MWCS
step (see below), and
compute_MWCS is not finishing. E.g. I have 5 frequency bands, but for
bands
2-4 only 1 of 10 station pair MWCS's get
calculated, even though all the
data is there in the stacks. I have tried rerunning comute_mwcs by
changing
the flag back to 'T' for the station
pairs where mwcs did not get
calculated, but it still crashes. This crash is repeatable.
Any thoughts?
(p.s. I also initially tried remaking the stacks, but it crashed at the
same point. The data looks good in the stacks)
-ashton
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py",
line 1139, in _execute_context
context)
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py",
line 450, in do_execute
cursor.execute(statement, parameters)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 165, in execute
result = self._query(query)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 321, in _query
conn.query(q)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 859, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1096, in _execute_command
self._write_bytes(packet)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1048, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away
(BrokenPipeError(32, 'Broken pipe'))")
The above exception was the direct cause of the following exception:
_______________________________________________
MSNoise mailing list
MSNoise(a)mailman-as.oma.be
http://mailman-as.oma.be/mailman/listinfo/msnoise
8 p.m.
New subject: [EXTERNAL] Re: MWCS sql crash
Ashton,
No, I don't think it's linked. If the REF file is not available, the
code should crash and not hang.
How long did your MWCS ran ? How many MWCS jobs are there ? How many
stations / stations-pairs ?
what is the content of your my.cnf / or mysql configuration file ?
Thomas
Le 19/10/2018 à 18:53, Flinders, Ashton a écrit :
Hi Thomas, I actually think this was related to the PR
I submitted the
other day. Since I have a mix of stations (some 3-comp some only Z), when
mwcs_compute tried to calculate RR for a station-pair that only had ZZ, and
it couldnt find the reference function it crashed/hanged. Then after a
while hanging it threw the SQL error.
On Thu, Oct 18, 2018 at 11:15 PM Thomas Lecocq <Thomas.Lecocq(a)seismology.be>
wrote:
Hi Ashton
it seems your MWCS computation took a looooooong time and the MySQL
connection was killed during that time. Can you confirm ?
Thomas
Le 18/10/2018 à 18:54, Flinders, Ashton a écrit :
I get a strange crash part way through my MWCS
step (see below), and
compute_MWCS is not finishing. E.g. I have 5 frequency bands, but for
bands
2-4 only 1 of 10 station pair MWCS's get
calculated, even though all the
data is there in the stacks. I have tried rerunning comute_mwcs by
changing
the flag back to 'T' for the station
pairs where mwcs did not get
calculated, but it still crashes. This crash is repeatable.
Any thoughts?
(p.s. I also initially tried remaking the stacks, but it crashed at the
same point. The data looks good in the stacks)
-ashton
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py",
line 1139, in _execute_context
context)
File
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py",
line 450, in do_execute
cursor.execute(statement, parameters)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 165, in execute
result = self._query(query)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
line 321, in _query
conn.query(q)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 859, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1096, in _execute_command
self._write_bytes(packet)
File
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1048, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away
(BrokenPipeError(32, 'Broken pipe'))")
The above exception was the direct cause of the following exception:
_______________________________________________
MSNoise mailing list
MSNoise(a)mailman-as.oma.be
http://mailman-as.oma.be/mailman/listinfo/msnoise
9:13 p.m.
New subject: [EXTERNAL] Re: MWCS sql crash
It ran for less than a day. Approximately ~12k MWCS jobs, but not all of
those ran because of the REF file issue. I reran it last night, and it
ended just in the last hour - and while it did compute all the jobs, it
gave me the same error at the end. So it seems like an exit error once the
compute_mwcs job is done (no exit code, and then SQL hangs?).
Of course this is could all be complicated since I am using a Slurm job
scheduler to hand processor assignment.
On Fri, Oct 19, 2018 at 11:01 AM Thomas Lecocq <Thomas.Lecocq(a)seismology.be>
wrote:
Ashton,
No, I don't think it's linked. If the REF file is not available, the
code should crash and not hang.
How long did your MWCS ran ? How many MWCS jobs are there ? How many
stations / stations-pairs ?
what is the content of your my.cnf / or mysql configuration file ?
Thomas
Le 19/10/2018 à 18:53, Flinders, Ashton a écrit :
_______________________________________________
MSNoise mailing list
MSNoise(a)mailman-as.oma.be
http://mailman-as.oma.be/mailman/listinfo/msnoise
--
Ashton F. Flinders, Ph.D
U.S. Geological Survey
345 Middlefield Road
Menlo Park, CA 94025
(650) 329-5050
Hi Thomas, I actually think this was related to
the PR I submitted the
other day. Since I have a mix of stations (some 3-comp some only Z), when
mwcs_compute tried to calculate RR for a station-pair that only had ZZ,
and
it couldnt find the reference function it
crashed/hanged. Then after a
while hanging it threw the SQL error.
On Thu, Oct 18, 2018 at 11:15 PM Thomas Lecocq <
Thomas.Lecocq(a)seismology.be>
wrote:
> Hi Ashton
>
> it seems your MWCS computation took a looooooong time and the MySQL
> connection was killed during that time. Can you confirm ?
>
> Thomas
>
>
> Le 18/10/2018 à 18:54, Flinders, Ashton a écrit :
>> I get a strange crash part way through my MWCS step (see below), and
>> compute_MWCS is not finishing. E.g. I have 5 frequency bands, but for
> bands
>> 2-4 only 1 of 10 station pair MWCS's get calculated, even though all
the
>> data is there in the stacks. I have tried
rerunning comute_mwcs by
> changing
>> the flag back to 'T' for the station pairs where mwcs did not get
>> calculated, but it still crashes. This crash is repeatable.
>>
>> Any thoughts?
>>
>> (p.s. I also initially tried remaking the stacks, but it crashed at the
>> same point. The data looks good in the stacks)
>>
>> -ashton
>>
>> During handling of the above exception, another exception occurred:
>>
>>
>> Traceback (most recent call last):
>>
>> File
>>
>
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py",
>> line 1139, in _execute_context
>>
>> context)
>>
>> File
>>
>
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py",
>> line 450, in do_execute
>>
>> cursor.execute(statement, parameters)
>>
>> File
>>
>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
>> line 165, in execute
>>
>> result = self._query(query)
>>
>> File
>>
>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
>> line 321, in _query
>>
>> conn.query(q)
>>
>> File
>>
>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
>> line 859, in query
>>
>> self._execute_command(COMMAND.COM_QUERY, sql)
>>
>> File
>>
>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
>> line 1096, in _execute_command
>>
>> self._write_bytes(packet)
>>
>> File
>>
>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
line 1048, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away
(BrokenPipeError(32, 'Broken pipe'))")
The above exception was the direct cause of the following exception:
_______________________________________________
MSNoise mailing list
MSNoise(a)mailman-as.oma.be
http://mailman-as.oma.be/mailman/listinfo/msnoise
9:18 p.m.
New subject: [EXTERNAL] Re: MWCS sql crash
Ahston
I'm pretty sure the issue #1 is linked to your mysql server not being
able to handle long connections/too numerous connections.
Thomas
Le 19/10/2018 à 21:13, Flinders, Ashton a écrit :
It ran for less than a day. Approximately ~12k MWCS
jobs, but not all of
those ran because of the REF file issue. I reran it last night, and it
ended just in the last hour - and while it did compute all the jobs, it
gave me the same error at the end. So it seems like an exit error once the
compute_mwcs job is done (no exit code, and then SQL hangs?).
Of course this is could all be complicated since I am using a Slurm job
scheduler to hand processor assignment.
On Fri, Oct 19, 2018 at 11:01 AM Thomas Lecocq <Thomas.Lecocq(a)seismology.be>
wrote:
Ashton,
No, I don't think it's linked. If the REF file is not available, the
code should crash and not hang.
How long did your MWCS ran ? How many MWCS jobs are there ? How many
stations / stations-pairs ?
what is the content of your my.cnf / or mysql configuration file ?
Thomas
Le 19/10/2018 à 18:53, Flinders, Ashton a écrit :
Hi Thomas, I actually think this was related to
the PR I submitted the
other day. Since I have a mix of stations (some 3-comp some only Z), when
mwcs_compute tried to calculate RR for a station-pair that only had ZZ,
and
it couldnt find the reference function it
crashed/hanged. Then after a
while hanging it threw the SQL error.
On Thu, Oct 18, 2018 at 11:15 PM Thomas Lecocq <
Thomas.Lecocq(a)seismology.be>
wrote:
> Hi Ashton
>
> it seems your MWCS computation took a looooooong time and the MySQL
> connection was killed during that time. Can you confirm ?
>
> Thomas
>
>
> Le 18/10/2018 à 18:54, Flinders, Ashton a écrit :
>> I get a strange crash part way through my MWCS step (see below), and
>> compute_MWCS is not finishing. E.g. I have 5 frequency bands, but for
> bands
>> 2-4 only 1 of 10 station pair MWCS's get calculated, even though all
the
>> data is there in the stacks. I have tried
rerunning comute_mwcs by
> changing
>> the flag back to 'T' for the station pairs where mwcs did not get
>> calculated, but it still crashes. This crash is repeatable.
>>
>> Any thoughts?
>>
>> (p.s. I also initially tried remaking the stacks, but it crashed at the
>> same point. The data looks good in the stacks)
>>
>> -ashton
>>
>> During handling of the above exception, another exception occurred:
>>
>>
>> Traceback (most recent call last):
>>
>> File
>>
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py",
>> line 1139, in _execute_context
>>
>> context)
>>
>> File
>>
"/home/ashton/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py",
>> line 450, in do_execute
>>
>> cursor.execute(statement, parameters)
>>
>> File
>>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
>> line 165, in execute
>>
>> result = self._query(query)
>>
>> File
>>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/cursors.py",
>> line 321, in _query
>>
>> conn.query(q)
>>
>> File
>>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
>> line 859, in query
>>
>> self._execute_command(COMMAND.COM_QUERY, sql)
>>
>> File
>>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
>> line 1096, in _execute_command
>>
>> self._write_bytes(packet)
>>
>> File
>>
"/home/ashton/anaconda3/envs/msnoise/lib/python3.5/site-packages/pymysql/connections.py",
>> line 1048, in _write_bytes
>>
>> "MySQL server has gone away (%r)" % (e,))
>>
>> pymysql.err.OperationalError: (2006, "MySQL server has gone away
>> (BrokenPipeError(32, 'Broken pipe'))")
>>
>>
>> The above exception was the direct cause of the following exception:
>>
>>
>>
> _______________________________________________
> MSNoise mailing list
> MSNoise(a)mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
>
_______________________________________________
MSNoise mailing list
MSNoise(a)mailman-as.oma.be
http://mailman-as.oma.be/mailman/listinfo/msnoise
2794
days inactive
2795
days old
5 comments
2 participants
participants (2)
-
Flinders, Ashton -
Thomas Lecocq