Hello,
I periodically receive this error when running (very large) cross correlation jobs:
“pymysql.err.OperationalError 2006, "MySQL server has gone away (error(32, 'Broken pipe'))")”
I have modified wait_timeout to 24 hours and increased the max_allowed_packet variable in the mySQL server (the two suggested sources of the error I’ve found through Google,) but this error is still occurring.
Any recommendations?
Thanks,
Chrysm
Hello,
I am trying to look for reference for msnoise, but when I opened
http://msnoise.org/doc it shows "Page not found". Is there a problem with
the msnoise website?
Warm regards,
Mochammad Husni Rizal
Technician of geophysics equipments for DOMERAPI -- Research Engineer
Phone : (+62) 857-151-220-90
Hi Shuye,
there are some possible causes of this:
1/ the start & end dates are not properly defined --> edit config
2/ the ref start & end dates are not properly defined --> edit config
3/ the CC jobs have been done more than 1 day ago --> run "msnoise stack
-r -m -i 9999 " to include all computations,
Let us know how it goes,
Thomas
Le 06/09/2016 à 12:08, hsy119 a écrit :
> Hi,
>
> I've do the cross-correlation jobs, and now stacking, but there's a
> problem. I have data in my data folder from 2015-07-01 to 2015-12-01
> of 3 stations, and 100+ cross-correlation functions for each pair have
> been computed(stored in 001_DAYS I guess)
> But when I stacking, only a few days' stacking have been done. The
> folder 010_DAYS/station1_station2/ only have tens of .mseed files.
> What has happend?
>
> Everything is right when I only populate 2 stations. Stacking is all
> right and 100+ files is in 010_DAYS/station1_station2/.
>
> MSNoise is very nice tool, even for the beginner of python like me.
> But I'm stucking in this step.
> I'm appreciated for your help
>
> Shuye
>
Hello,
I have successfully run 'msnoise compute_cc' for my data so that I now have a file of 1 day long cross-correlations. I would now like to stack them for the whole time period (~2 years). However, when I try to run 'msnoise stack -r' it fails and I get an error message. I have pasted this below. Would you be able to shed some light onto it? How can I resolve this?
Thanks in advance,
Emily
root@COPS-023603-L:/home/rpgsbs/r01ec15/Documents/BASS_DATA/sorted# msnoise stack -r
Lets STACK !
2016-08-25 15:02:55 [DEBUG] Starting the ref stack
[]
2016-08-25 15:02:55 [DEBUG] Processing BA_BA01:BA_BA02-TT-1
Traceback (most recent call last):
File "/root/anaconda2/bin/msnoise", line 11, in <module>
sys.exit(run())
File "/root/anaconda2/lib/python2.7/site-packages/msnoise/scripts/msnoise.py", line 614, in run
cli(obj={})
File "/root/anaconda2/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/root/anaconda2/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/root/anaconda2/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/anaconda2/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/anaconda2/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/root/anaconda2/lib/python2.7/site-packages/msnoise/scripts/msnoise.py", line 301, in stack
main('ref', interval)
File "/root/anaconda2/lib/python2.7/site-packages/msnoise/s04stack.py", line 203, in main
updated_days = updated_days_for_dates(db, start, end, pair.replace('_', '.'), jobtype='CC', interval=datetime.timedelta(days=interval),returndays=True)
File "/root/anaconda2/lib/python2.7/site-packages/msnoise/api.py", line 1189, in updated_days_for_dates
days = session.query(Job).filter(Job.pair == pair).filter(Job.day >= date1).filter(Job.day <= date2).filter(Job.jobtype == jobtype).filter(Job.lastmod >= lastmod).group_by(Job.day).order_by(Job.day).all()
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2613, in all
return list(self)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
return self._execute_and_instances(context)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2776, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
exc_info
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/root/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/cursors.py", line 161, in execute
result = self._query(query)
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/cursors.py", line 317, in _query
conn.query(q)
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/connections.py", line 837, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/connections.py", line 1021, in _read_query_result
result.read()
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/connections.py", line 1304, in read
first_packet = self.connection._read_packet()
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/connections.py", line 983, in _read_packet
packet.check_error()
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/connections.py", line 395, in check_error
err.raise_mysql_exception(self._data)
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
_check_mysql_exception(errinfo)
File "/root/anaconda2/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
raise InternalError(errno, errorvalue)
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1055, u"Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'msnoise.jobs.ref' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by") [SQL: u'SELECT jobs.ref AS jobs_ref, jobs.day AS jobs_day, jobs.pair AS jobs_pair, jobs.jobtype AS jobs_jobtype, jobs.flag AS jobs_flag, jobs.lastmod AS jobs_lastmod \nFROM jobs \nWHERE jobs.pair = %(pair_1)s AND jobs.day >= %(day_1)s AND jobs.day <= %(day_2)s AND jobs.jobtype = %(jobtype_1)s AND jobs.lastmod >= %(lastmod_1)s GROUP BY jobs.day ORDER BY jobs.day'] [parameters: {u'day_2': datetime.date(2013, 4, 29), u'lastmod_1': datetime.datetime(2016, 8, 24, 15, 2, 55, 233936), u'day_1': datetime.date(2011, 5, 22), u'jobtype_1': 'CC', u'pair_1': 'BA.BA01:BA.BA02'}]
~~~~~~~~~~~~~~~~~~~~~~~~~
Emily Crowder MSc BSc
PhD Student
School of Geosciences
University of Aberdeen
Aberdeen AB24 3UE
Scotland
Em: emily.crowder(a)abdn.ac.uk
~~~~~~~~~~~~~~~~~~~~~~~~~
The University of Aberdeen is a charity registered in Scotland, No SC013683.
Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.
Hello,
I have used MSNoise's new feature PWS to get some excellent-looking cross-correlations of ambient noise. My problem is that the zero time mark is located at the left edge of the CC graph, rather than at the centre. I wanted to have + to - 300 s but instead I have 0 to 600 s. Has anyone else come across this problem? Is there any fix you can recommend to solve it?
Kind regards,
Emily
~~~~~~~~~~~~~~~~~~~~~~~~~
Emily Crowder MSc BSc
PhD Student
School of Geosciences
University of Aberdeen
Aberdeen AB24 3UE
Scotland
Em: emily.crowder(a)abdn.ac.uk
~~~~~~~~~~~~~~~~~~~~~~~~~
The University of Aberdeen is a charity registered in Scotland, No SC013683.
Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.
can you please give a step by step instructions regarding how to input data
into the database. I have setup everything (phpmyadmin, mysql database,
msnoise web interface) . I have data from stations in ASCII format and are
in separate files of one hour time intervals. Data files have an extension
of seisan and seis_saf. Also i would like to know how to add stations and
populate station list.
Dear Thomas and msnoise users,
I'm using msnoise for the first time and so far I have been able to compute dtt without any trouble. However, when I'm trying to plot "mwcs" and "dtt" I have the following errors.
I already check the msnoise bugreport -m and it seems I have everything in order, so I really don't know what seems to be the problem.
I appreciate in advance for any help.
Kind regards,
Jose Castillo
josecastillo@josecastillo-550P5C-550P7C:~/Desktop/Project_msnoise/MS_Noise$ msnoise plot mwcs YH_DA07 YH_DB07
7.79420817017
New Data for YH_DA07_YH_DB07-ZZ-1-1
47
/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/plots/mwcs.py:146: FutureWarning: how in .resample() is deprecated
the new syntax is .resample(...).mean()
alldt = alldt.resample('D', how='mean')
/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/plots/mwcs.py:147: FutureWarning: how in .resample() is deprecated
the new syntax is .resample(...).mean()
allcoh = allcoh.resample('D', how='mean')
Traceback (most recent call last):
File "/home/josecastillo/anaconda3/bin/msnoise", line 11, in <module>
sys.exit(run())
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/scripts/msnoise.py", line 614, in run
cli(obj={})
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/scripts/msnoise.py", line 502, in mwcs
main(sta1, sta2, filterid, comp, mov_stack, show, outfile)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/plots/mwcs.py", line 159, in main
plt.imshow(alldt.T, extent=xextent,aspect="auto",interpolation='none',origin='lower',cmap=cmap_center_point_adjust(cm.seismic,crange,0))
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/plots/mwcs.py", line 80, in cmap_center_point_adjust
abs(center - range[0]) / abs(range[1] - range[0]))
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/plots/mwcs.py", line 68, in cmap_center_adjust
return cmap_powerlaw_adjust(cmap, a)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/plots/mwcs.py", line 54, in cmap_powerlaw_adjust
cdict[key].sort()
AttributeError: 'map' object has no attribute 'sort'
josecastillo@josecastillo-550P5C-550P7C:~/Desktop/Project_msnoise/MS_Noise$ msnoise plot dtt YH_DA07 YH_DB07 2012-10-11
MWCS/01/001_DAYS/ZZ/YH_DA07_YH_DB07/2012-10-11.txt
Empty DataFrame
Columns: [Date, A, EA, EM, EM0, M, M0, Pairs]
Index: []
Traceback (most recent call last):
File "/home/josecastillo/anaconda3/bin/msnoise", line 11, in <module>
sys.exit(run())
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/scripts/msnoise.py", line 614, in run
cli(obj={})
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/scripts/msnoise.py", line 559, in dtt
main(sta1, sta2, filterid, comp, day, mov_stack, show, outfile)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/msnoise/plots/dtt.py", line 91, in main
plt.plot(xlineM0, ylineM0, 'r', label='M0')
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/matplotlib/pyplot.py", line 3154, in plot
ret = ax.plot(*args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/matplotlib/__init__.py", line 1812, in inner
return func(ax, *args, **kwargs)
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/matplotlib/axes/_axes.py", line 1424, in plot
for line in self._get_lines(*args, **kwargs):
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 386, in _grab_next_args
for seg in self._plot_args(remaining, kwargs):
File "/home/josecastillo/anaconda3/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 374, in _plot_args
seg = func(x[:, j % ncx], y[:, j % ncy], kw, kwargs)
ZeroDivisionError: integer division or modulo by zero
Let's Bug Report MSNoise !
************* Computer Report *************
----------------+PYTHON+-------------------
Python:3.5.1 |Anaconda 4.0.0 (64-bit)| (default, Dec 7 2015, 11:16:01)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
---------------+MODULES+-------------------
Required:
[X] setuptools: 20.3
[X] click: 6.6
[X] numpy: 1.10.4
[X] scipy: 0.17.0
[X] pandas: 0.18.0
[X] matplotlib: 1.5.1
[X] statsmodels: 0.6.1
[X] sqlalchemy: 1.0.12
[X] traitsui: 5.1.0
[X] traits: 4.5.0
[X] scikits.samplerate: present (no version)
[X] obspy: 1.0.1
[X] flask: 0.10.1
[X] flask.ext.admin: 0.10.1
[X] flask_admin: 1.4.2
[X] wtforms: 2.1
[X] bokeh: 0.11.1
Only necessary if you plan to build the doc locally:
[X] sphinx: 1.3.5
[X] jinja2: 2.8
Graphical Backends: (at least one is required)
[ ] wx: not found (install via None)
[X] PyQt4: present (no version)
[ ] PySide: not found (install via None)
Not required, just checking:
[X] json: 2.0.9
[X] psutil: 4.1.0
[ ] reportlab: not found (install via None)
[X] configobj: 5.0.6
[X] pkg_resources: present (no version)
[ ] paramiko: not found (install via None)
[X] ctypes: 1.1.0
[X] pyparsing: 2.0.3
[X] distutils: 3.5.1
[X] IPython: 4.1.2
[ ] vtk: not found (install via None)
[ ] enable: not found (install via None)
Hi all,
This is an annoucement for a small bugfix release: MSNoise 1.4.1
contains corrections for:
* Typos
* Hardcode version to prevent pypi errors
* dynamic loading of CDN or local jquery
* fixed raw_input vs input (Py2 vs Py3) at "installation step"
Nothing 'core' business, so these "bugs" will not have affected your
results using MSNoise!
By the way: here are two recent really cool papers using MSNoise !
* Mordret, Aurélien, T. Dylan Mikesell, Christopher Harig, Bradley P.
Lipovsky, et Germán A. Prieto. « Monitoring Southwest Greenland’s Ice
Sheet Melt with Ambient Seismic Noise ». /Science Advances/ 2, n^o 5 (1
mai 2016): e1501538. doi:10.1126/sciadv.1501538.
* Wu, Chunquan, Andrew Delorey, Florent Brenguier, Celine Hadziioannou,
Eric G Daub, et Paul Johnson. « Constraining Depth Range of S-Wave
Velocity Decrease after Large Earthquakes near Parkfield, California. »
/Geophysical Research Letters/, 1 mai 2016, 2016GL069145.
doi:10.1002/2016GL069145.
Cheers !
Thomas
--
Dr. Thomas Lecocq
Geologist - Seismologist
Seismology - Gravimetry
Royal Observatory of Belgium
*
* * * * *
* * * *
---------
http://www.seismology.behttp://twitter.com/#!/Seismologie_behttps://www.facebook.com/seismologie.be
Hi Thomas,
As far as I understand you haven't changed anything since then in the code
concerning the pre-processing when computing autocorrelations. Right?
Greetings from Athens,
Christos
From: Thomas Lecocq <thomas.lecocq@...
<http://gmane.org/get-address.php?address=thomas.lecocq%2d7ewb5K7nbqnl%2fIpH…>
>
Subject: Re: Autocorrelations in MSNoise
<http://news.gmane.org/find-root.php?message_id=54DB931F.5080705%40seismolog…>
Newsgroups: gmane.science.geophysics.msnoise
<http://news.gmane.org/gmane.science.geophysics.msnoise>
Date: 2015-02-11 17:36:31 GMT (1 year, 14 weeks, 6 days, 11 hours and 16
minutes ago)
Hello Aurélien,
Le 11/02/2015 18:30, Aurélien Mordret a écrit :
> Hi Thomas,
>
> What is the pre-processing of the noise when computing autocorrelations? Is
> it the same as for the cross-correlations?
Yeah, currently it's the same, WHICH IS NOT GOOD ! (that's why it's
written somewhere NOT TO USE AUTOCORR), but that might not be trivial to
find...
Your suggestion makes absolutely sense ! Are you used to github etc ? If
yes, please fork and PR an updated version of whiten ! (including a
autocorr=False flag in the def, maybe?)!
Regards from Brussels
Thomas
>
> If yes, would it be better to not fully whiten the data before
> autocorrelation as the only information comes from the amplitude spectrum
> (and not the phase)?
>
> I suggest to modify a bit the whiten.py function to keep the structure of
> the whole code similar. Following Gorbatov et al. (2013), instead of
> putting the amplitude spectrum =1 (with tapper) I would use a normalisation
> with a water-level in the form
>
> A(freq) = A(freq)/max(A(freq),c*max(A(freq)))
>
> With A the amplitude spectrum and c the water-level factor (which is
> typically on the order of 1%).
>
> It should not change the results for the cross-correlations and improve the
> autocorrelations.
>
> Does it make sense or did I missed something?
>
> Cheers,
>
> Aurelien
>
> REF:
>
> Gorbatov, A., Saygin, E., & Kennett, B. L. N. (2013). Crustal properties
> from seismic station autocorrelograms. *Geophysical Journal International*,
> **192**(2), 861-870.
>
