Retrieve External Bulletins

Author: Jennifer Eakins <jeakins@ucsd.edu>

How do I retrieve external bulletins to use in real-time processing?

There are many ways to collect bulletins from other institutions. BRTT has written some programs to do this and others have been written by individual users and submitted to the contrib sections (available via the Antelope User's Group website). While your system may be collecting data in real-time, most bulletins have some lag-time due to reviewing needs and can be considered either near real-time or not real-time at all.

Some of the bulletin import programs available:

PROGRAM NAME BRTT/contrib Latency Comments
qedd BRTT hours finger quake
mail_parser/mail_parser_wrapper contrib varies e-mail input
update_weekly_qed contrib days-weeks cronjob
scec2db contrib days cronjob
update_cmt contrib months cronjob
update_pde contrib many months cronjob
reflectd BRTT user specified server/client
orb2orb BRTT none not reviewed

The qedd, scec2db, update_weekly_QED, update_pde, and reflectd programs can all be run through a single rtexec.pf. Although I have not tested it, I suspect that the mail_parser could also be run from within the same rtexec.pf. Running orb2orb requires a local orbserver and is not discussed in this faq.

Take advantage of rtexec.pf!

Managing all of these individual programs can be a chore, but the task can be simplified if you group them into an rtexec file. Among other benefits: cronjob execution is simplified; after a system reboot catalog retrieval can be automatically restarted; all processes are run from one machine; ease of adding additional processes.

A typical PASSCAL experiment will have a single orb running on a SUN workstation at a remote site. This orb collects data via the ipd program or through the orb2orb. At that remote site, additional processes such as orbtrigger, orbdetect, and orbassoc may be running. Cron jobs, such as rtdbclean might also be running. It is unlikely, however, that an analyst is at that remote location.

Normally, data is sent to a second orb (at the PIs lab?) where further processing is done on the data. This could include combining the PASSCAL data with regional network data, associating the triggered events against a near-realtime catalog, doing a full review of the data including event comparisons against other published bulletins. The methods of retrieval for these bulletins may vary, but most do not require a running orbserver (except orb2orb).

How to setup bulletin retrieval.

Before you read this, you should have some basic knowledge about the ANTELOPE software. I would suggest reading the manpages for rtsetup, rt, and rtexec.

At the PIs lab, I would create a real-time user and separate directories for data and catalog retrieval. Two separate rtexec programs will be started under these two directories: the processes to run in each of the rtexec.pf files will be very different.

This modified rtexec.pf file needs no orbserver to run, and has dummy $ORB and $dB values. In my example below, the PI has a Sun workstation called rthost with a user, rt. There is one orbserver receiving data from a remote PASSCAL telemetered deployment (via orb2orb): the database for this orb is "swus".

rthost% cd ~rt 
rthost% ls swus/ rthost% ls swus bin/ pf/ dB/ rtexec.pf dbmaster/ 
rtsys/ logs/ state/ orb/ rthost% mkdir catalogs rthost% cd catalogs 
rthost% mkdir bin dB dbmaster logs orb 
pf rtsys state idaho utah mtech usbr scec ncec unr alaska canada 
rthost% CD dB rthost% vi QED

Put the following two lines in the "QED" descriptor file:

rt1.0 /export/rt/catalogs/dbmaster/{QED}

At this point, things get a little tricky. You can start/stop rtexec without the GUI interface program, rtm, but rtm can give users a more interactive look at what processes are currently running. However, with this "catalogs" database you don't have the standard set of database tables that rtm expects to see. You need to fudge some tables (network and site) and create fake records to make rtm happy.

rthost% CD dbmaster rthost% dbe dbmaster/QED

Now choose "File > Create new table > network". You'll see a network table with one blank row, and a window which allows you to edit all fields of the network table. You only need to change the net and netname.

Figure 1: The netname you choose will be used as the default title of your rtm display

Figure 2: Only one station is needed. You could use a pre-existing site table, but the information in this table is not used during the catalog retrieval processes in this example.

Once you have these tables made, click Add/Done and they should be saved in the dbmaster directory.

Next, copy the default rtexec.pf file from $ANTELOPE/data/pf/rtexec.pf to your ~/catalogs directory

rthost% cp $ANTELOPE/data/pf/rtexec.pf ~/catalogs/

Make the following edits to your local rtexec.pf:

  1. Remove unneeded commands from the "Processes" and "Run" arrays.
    This includes orbserver, orb2orb, orb2db, orb2dbt, orbdetect, orbtrigger, orbassoc, etc.
  2. Change $dB from dB/nw' to dB/QED
  3. Add the following lines to the "Processes" array:
  4. reflect_anza 
    reflect -d /export/rt/catalogs/anza bbarray.ucsd.edu
    /export/rt/anza/dB/anza.origin /export/rt/anza/dB/anza.assoc /export/rt/anza/dB/anza.event
    finger_quake finger_quake -p mk_finger_swus -e 75 swus/swus qedd qedd QED/QED_2001 qedd_Idaho qedd -ipfinger quake@sisyphus.idbsu.edu -author IDAHO_QED Idaho/Idaho qedd_Utah qedd -ipfinger quake@eqinfo.seis.utah.edu -author UTAH_QED Utah/Utah qedd_mtech qedd -ipfinger quake@mbmgsun.mtech.edu -author MTECH_QED mtech/mtech qedd_usbr qedd -ipfinger quake@seismo.do.usbr.gov -author USBR_QED usbr/usbr qedd_scec qedd -l -ipfinger quake@scec.gps.caltech.edu -author SCEC_QED scec/scec qedd_ncec qedd -l -ipfinger quake@quake.geo.berkeley.edu -author NCEC_QED ncec/ncec qedd_unr qedd -ipfinger quake@seismo.unr.edu -author UNR_QED unr/unr qedd_Alaska qedd -l -ipfinger quake@giseis.alaska.edu -author AK_QED Alaska/Alaska qedd_Canada qedd -l -ipfinger quake@seismo.nrcan.gc.ca -author CAN_QED Canada/Canada

    Note that some of these finger quake sites need to have the "-l" option specified, while others do not. When adding a new qedd request to the Processes array, you may wish to try "finger quake@somewhere.else.net" from the command line to see if the -l option is needed.

  5. Add the following lines to the "Run" array:
  6. reflect_anza 
    1 finger_quake 1 qedd 1 qedd_Utah 1 qedd_Idaho 1 qedd_mtech 
    1 qedd_usbr 1 qedd_scec 1 qedd_ncec 1 qedd_unr 1 qedd_Alaska 
    1 qedd_Canada 1
  7. Add to the crontab:
  8. cmt_update 
    UTC 15 22 1,18 * * cmt_update -v cmt_update weekly_QED 
    UTC 50 13 * * * update_weekly_QED -v pf/update_weekly_QED 
    get_scec 
    UTC 22 02 * * * scec2db -v -m -c hypo71 cit/cit
  9. Remove all other values from the "Buttons" array and replace with:
  10. QED dbe QED/QED 
    QED_weekly dbe QED/QED_weekly QED_cit dbe cit/cit QED_scec 
    dbe scec/scec QED_ncec dbe ncec/ncec QED_unr dbe unr/unr QED_usbr 
    dbe usbr/usbr QED_mtech dbe mtech/mtech QED_Idaho dbe Idaho/Idaho 
    QED_Utah dbe Utah/Utah QED_Alaska dbe Alaska/Alaska QED_Canada 
    dbe Canada/Canada cmt dbe cmt/cmt dbevents dbevents QED/QED 
    

Explanation of modifications – What have I just asked rtexec to do for me?

You have set up your system to:

  1. Retrieve the anza catalog from bbarray.ucsd.edu via the reflect mechanism (this assumes that bbarray.ucsd.edu is serving these files and you have been allowed to access them.)
  2. Create your own "finger quake" bulletin that others can retrieve.
  3. Retrieve 10 different "finger quake" bulletins.
  4. Look for new Harvard CMT bulletins on the 1st and 18th of every month
  5. Update your QED_weekly catalog with new origins once a day (at 13:50 UTC)
  6. Look for new origins from the TRINET catalog every day (at 02:22 UTC)

Testing – But does it work?

Now comes the moment of truth. See if you can get rtm to run. If so, try starting your rtexec processes.

rthost% pwd /export/rt/catalogs rthost% rtm

If you see:

problems

you have something wrong with your setup. Verify that you have a descriptor file (dB/QED), a proper database name specified in rtexec.pf, all needed directories, both site and network tables in the database you specified. Fix any problems and retry.

If all goes well, you should see:

closed

Click on the "Processing Tasks" button/bar.

open down

Hit the "Start" button. Eventually you will see something similar to:

running

Note that the "reflect_anza" process is highlighted in orange, has no Pid (process id) and is taking no memory or CPU. This means that there is either a problem in the command-line or that the server that it is trying to grab files from is down or not accepting connections from you. You should contact the network manager at the remote site if you wish to gain access to those tables. Note that reflectd is not the preferred means of obtaining a catalog as it replaces the entire local catalog soon after it was updated at the remote site.

How to use bulletins retrieved – So, I'm getting catalogs, now what?

Now that you are retrieving all of these catalogs, use them in your daily processing. You can now run dbassoc_rt on the orb running on the "swus" database. The current distribution of Antelope (4.3u as of Jan 2001), can't handle multiple instances of dbassoc_rt, so choose one authoritative catalog to use with this program. Future versions of dbassoc_rt will allow the user to have multiple dbassoc_rt programs with different external catalogs with a user-specified "preferred" catalog. You can run multiple dbassoc_rt processes now, but the most recent origin that dbassoc_rt encounters will be chosen as preferred.

rthost% CD ~/swus/ rthost% vi rtexec.pf

Add the following line to "Processes":

dbassoc dbassoc_rt -v /export/rt/catalogs/QED/QED $dB

And add to "Run":

dbassoc 1

If you are using dbloc2 to review your data add the catalogs to your local dbloc2.pf file

rthost% cp $ANTELOPE/data/pf/dbloc2.pf /export/rt/swus/pf/ 
rthost% vi /export/rt/swus/pf/dbloc2.pf

Modify the reference_dB table:

reference_dB 
&Tbl{ 
	# list of reference databases (catalogs) for association 
	/export/rt/catalogs/QED/QED /export/rt/catalogs/QED/QED_weekly 
	/export/rt/catalogs/cmt/cmt /export/rt/catalogs/cit/cit /export/rt/catalogs/ncec/ncec 
	/export/rt/catalogs/scec/scec /export/rt/catalogs/unr/unr 
	}

You should now be able to review your data and associate against these external bulletins.

How do I get the catalogs rtexec to start on reboot?

Read the manpage on S99_antelope. You will need to have root access to modify the /etc/init.d/antelope file. Add your /export/rt/catalogs to the "@dirs" array.

URL: http://eqinfo.ucsd.edu/faq/antelope_retrieve.php [Last updated: 2015-10-22 (295) 23:07:44 UTC]