Thomas,
When trying to use s04stack.py, I keep getting the message of "Found 000 updated days" when it is not true. I have tried changing the "–i" interval to no avail (e.g. s04stack.py -m -r -i 100). I think I have traced the problem to the following queries in database_tools.py:
if pair == '%':
days = session.query(Job).filter(Job.day >= date1).filter(Job.day <= date2).filter(Job.type == type).filter(Job.lastmod >= lastmod).group_by(Job.day).order_by(Job.day).all()
else:
days = session.query(Job).filter(Job.pair == pair).filter(Job.day >= date1).filter(Job.day <= date2).filter(Job.type == type).filter(Job.lastmod >= lastmod).group_by(Job.day).order_by(Job.day).all()
The call from s04stack.py goes to the ELSE conditions, but days == [ ] after the query.
If I manually set 'pair = '%' on the line before, then 'days' returns what looks to be correct. As the only difference in the two queries is filtering by Job.pair, maybe that is the problem. I am attaching a screenshot of my jobs table. I manually changed 'lastmod' at one point to yesterday in case the –i switch was not working correctly.
I am on Mac OS X.
Thanks,
Rob
+----------------------------+
Robert E. Abbott, Ph.D.
Sandia National Laboratories
Geophysics Department MS 0750
P.O. Box 5800
Albuquerque, NM 87185-0750
(505) 845-0266
+----------------------------+
+----------------------------+
Robert E. Abbott, Ph.D.
Sandia National Laboratories
Geophysics Department MS 0750
P.O. Box 5800
Albuquerque, NM 87185-0750
(505) 845-0266
+----------------------------+
Dear MsNoise-ers:
I managed to run the CC's on the EH* components on 11 stations with 3 days
(Julian days 102,103,104) worth of data initialized with
python s01scan_archive.py -i -t 1
New jobs were created, and life was good. Then, I added more days (days
105-106):
[kasper@localhost ABAZ]$ ls
ABAZ.NZ..EHE.2014.102 ABAZ.NZ..EHZ.2014.106 ABAZ.NZ..VEM.2014.105
ABAZ.NZ..EHE.2014.103 ABAZ.NZ..LCQ.2014.102 ABAZ.NZ..VEM.2014.106
ABAZ.NZ..EHE.2014.104 ABAZ.NZ..LCQ.2014.103 ABAZ.NZ..VEP.2014.102
ABAZ.NZ..EHE.2014.105 ABAZ.NZ..LCQ.2014.104 ABAZ.NZ..VEP.2014.103
ABAZ.NZ..EHE.2014.106 ABAZ.NZ..LCQ.2014.105 ABAZ.NZ..VEP.2014.104
ABAZ.NZ..EHN.2014.102 ABAZ.NZ..LCQ.2014.106 ABAZ.NZ..VEP.2014.105
ABAZ.NZ..EHN.2014.103 ABAZ.NZ..VEC.2014.102 ABAZ.NZ..VEP.2014.106
ABAZ.NZ..EHN.2014.104 ABAZ.NZ..VEC.2014.103 ABAZ.NZ..VKI.2014.102
ABAZ.NZ..EHN.2014.105 ABAZ.NZ..VEC.2014.104 ABAZ.NZ..VKI.2014.103
ABAZ.NZ..EHN.2014.106 ABAZ.NZ..VEC.2014.105 ABAZ.NZ..VKI.2014.104
ABAZ.NZ..EHZ.2014.102 ABAZ.NZ..VEC.2014.106 ABAZ.NZ..VKI.2014.105
ABAZ.NZ..EHZ.2014.103 ABAZ.NZ..VEM.2014.102 ABAZ.NZ..VKI.2014.106
ABAZ.NZ..EHZ.2014.104 ABAZ.NZ..VEM.2014.103ABAZ.NZ..EHZ.2014.105
ABAZ.NZ..VEM.2014.104
and ran:
python s01scan_archive.py -t 1
Which I thought should flag the data base that there is work to do, but no
new jobs are created! In fact, the data is not added to the data base (see
attached screenshot), and hence there is nothing to CC.
Any ideas to try would be appreciated.
Best,
kasper
Hi Thom:
The errors happen both as -i (needs a fix in the documentation: no longer
"init") and in chron. Here is an excerpt of the output with the print
statement added as you suggested:
[kasper@kvan637-277769 MSNoise-master]$ python -i s01scan_archive.py
2014-04-01 13:18:11,043 [INFO] *** Starting: Scan Archive ***
2014-04-01 13:18:11,130 [INFO] Will work on 1 threads
['',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..EHE.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..EHE.2014.090',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..EHN.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..EHN.2014.090',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..EHZ.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..EHZ.2014.090',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..LCQ.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..LCQ.2014.090',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VEC.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VEC.2014.090',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VEM.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VEM.2014.090',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VEP.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VEP.2014.090',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VKI.2014.089',
'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/ABAZ.NZ..VKI.2014.090']
2014-04-01 13:18:11,252 [INFO] Started:
/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ
2014-04-01 13:18:11,276 [INFO] child process calling self.run()
Problem [Errno 21] Is a directory:
u'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/ABAZ/'
2014-04-01 13:18:13,215 [INFO] process shutting down
2014-04-01 13:18:13,215 [INFO] process exiting with exitcode 0
And maybe it is not too bad, but I don't know enough about the sqlite to
understand if the attached screenshot means the data_availability table is
filled or not, but
[kasper@kvan637-277769 MSNoise-master]$ python s02new_jobs.py
2014-04-01 13:21:01,630 [INFO] *** Starting: New Jobs ***
2014-04-01 13:21:01,664 [DEBUG] Found 0 new jobs to do
2014-04-01 13:21:01,839 [INFO] *** Finished: New Jobs ***
there are no new jobs..
Cheers,
Kasper
On 31 March 2014 21:04, Thomas Lecocq <thomas.lecocq(a)oma.be> wrote:
> Kasper,
>
> Does this happen when initialising the scan ? (- i argument) ? or in cron ?
> Is the data_availability table filled, even with all the errors ?
>
> It looks like, when listing files in a directory, the "." or "/" line
> gets returned, which leads to the read-error... This said, the ExitCode
> 0 means it sucessfully finished the process.
>
> one thing to test:
>
> on Line 179, add :
>
> print files
>
>
> with the same indentation as the following line (L180: if len(files) !=
> 0:).
>
> Best regards
>
> Thom
>
>
> Le 31/03/2014 02:52, Kasper van Wijk a écrit :
> > Dear Thomas,
> >
> > python s01scan_archive.py leads to many errors of this type:
> >
> > 2014-03-31 13:39:36,535 [INFO] Started:
> > /home/kasper/Downloads/MSNoise-master/AUCK/NZ/WTAZ
> > 2014-03-31 13:39:36,538 [INFO] child process calling self.run()
> > Problem [Errno 21] Is a directory:
> > u'/home/kasper/Downloads/MSNoise-master/AUCK/NZ/WTAZ/'
> > 2014-03-31 13:39:36,603 [INFO] process shutting down
> > 2014-03-31 13:39:36,603 [INFO] process exiting with exitcode 0
> >
> > My data is in the following BUD structure:
> > [kasper@kvan637-277769 AWAZ]$ pwd
> >
> > /home/kasper/Downloads/MSNoise-master/AUCK/NZ/
> >
> > [kasper@kvan637-277769 NZ]$ ls -ltr
> > total 44
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 RVAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 AWAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 WIAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 MKAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 MBAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 HBAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 ETAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 ABAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 WTAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 KBAZ
> > drwxrwxr-x. 2 kasper kasper 4096 Mar 31 13:00 EPAZ
> > [kasper@kvan637-277769 NZ]$ cd AWAZ/
> > [kasper@kvan637-277769 AWAZ]$ ls
> > AWAZ.NZ..EH1.2014.078 AWAZ.NZ..LCQ.2014.078 AWAZ.NZ..VEP.2014.078
> > AWAZ.NZ..EH1.2014.089 AWAZ.NZ..LCQ.2014.089 AWAZ.NZ..VEP.2014.089
> > AWAZ.NZ..EH2.2014.078 AWAZ.NZ..VEC.2014.078 AWAZ.NZ..VKI.2014.078
> > AWAZ.NZ..EH2.2014.089 AWAZ.NZ..VEC.2014.089 AWAZ.NZ..VKI.2014.089
> > AWAZ.NZ..EHZ.2014.078 AWAZ.NZ..VEM.2014.078
> > AWAZ.NZ..EHZ.2014.089 AWAZ.NZ..VEM.2014.089
> >
> > I am only intending to xcorrelate the EHZ (and EHN and EHE, if they
> exist).
> >
> > Config screen attached.
> >
> > Cheers,
> > kasper
> >
> >
> > _______________________________________________
> > MSNoise mailing list
> > MSNoise(a)mailman-as.oma.be
> > http://mailman-as.oma.be/mailman/listinfo/msnoise
>
> _______________________________________________
> MSNoise mailing list
> MSNoise(a)mailman-as.oma.be
> http://mailman-as.oma.be/mailman/listinfo/msnoise
>
