Using the nexsciTAP Server with the PyVO Client to Access LRIS Data At KOA In Asynchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, including cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform asynchronous TAP-based queries for public raw science and calibation data acquired with the Low Resolution Imaging Spectrograph (LRIS), hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (September, 2021)

Set up

In [10]:
from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")
import time

import sys
import os
import time
import warnings
warnings.filterwarnings("ignore")
In [3]:
instrument = 'LRIS'

instr = 'lris'
koatable = 'koa_lris'    
outdir = './outputLR/'
koaid_date = '20150123'

startdate = '2015-01-01 00:00:00'
enddate   = '2015-03-01 23:59:59'
    
date='20150123'

program = 'C305'


circle_pos = '174.27 29.8 1.0'
box_pos = '174.27 29.8 1 1'
polygon_pos = '173.27 28.8 173.27 30.8 176.27 29.8 '

query_keys = "koaid, object, koaimtyp, frameno, ra, dec, to_char(date_obs,'YYYY-MM-DD') as date_obs, elaptime, progid, proginst,  progpi, progtitl, semester, ofname, filehand"

Create output directory

In [5]:
try:
    os.mkdir(outdir)
except:
    print(instrument + " Directory exists already", flush=True)

Query by date

In [6]:
# write results to VOTable format
sql = "select koaid, filehand from " + koatable + " where koaid like '%" + koaid_date + "%'"
results = koa.run_async(sql)
print(results)
table=results.to_table()
table.write (outdir + instr + "_table_ipacascii.vot",format='votable',overwrite=True)
<Table length=7>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
LR.20150123.00166.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00166.fits
LR.20150123.00300.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00300.fits
LR.20150123.00538.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00538.fits
LR.20150123.00792.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00792.fits
LR.20150123.01035.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01035.fits
LR.20150123.01289.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01289.fits
LR.20150123.01603.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01603.fits

PyVO by default returns results in VOTable format. Convert them to IPAC ASCII and CSV format.

In [7]:
# write results to an IPAC ASCII table
sql = "select koaid, filehand from " + koatable + " where koaid like '%" + koaid_date + "%' "
results = koa.run_async(sql)
print(results)
table=results.to_table()
table.write (outdir + instr + "_table_ipacascii.tbl",format='ascii.ipac',overwrite=True)
<Table length=7>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
LR.20150123.00166.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00166.fits
LR.20150123.00300.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00300.fits
LR.20150123.00538.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00538.fits
LR.20150123.00792.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00792.fits
LR.20150123.01035.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01035.fits
LR.20150123.01289.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01289.fits
LR.20150123.01603.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01603.fits
In [8]:
#  write results to CSV file
sql = "select koaid, filehand from " + koatable + " where koaid like '%" + koaid_date + "%' "
results = koa.run_async(sql)
print(results)
table=results.to_table()
table.write (outdir + instr + "_table_csv.csv",format='csv', overwrite=True)
<Table length=7>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
LR.20150123.00166.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00166.fits
LR.20150123.00300.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00300.fits
LR.20150123.00538.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00538.fits
LR.20150123.00792.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00792.fits
LR.20150123.01035.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01035.fits
LR.20150123.01289.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01289.fits
LR.20150123.01603.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01603.fits

Select all keywords for data acquired on UT Date

In [11]:
sql = "select * from " + koatable + " where koaid like '%" + koaid_date + "%' "
results = koa.run_async(sql)
print (results)
table_selectall=results.to_table()
table_selectall.write (outdir + instr + "_table_selectall.vot",format='votable',overwrite=True)
<Table length=7>
        koaid          propint instrume ...    y        z     spt_ind 
                                        ...                           
        object          int32   object  ... float64  float64   int32  
---------------------- ------- -------- ... -------- ------- ---------
LR.20150123.00166.fits      18     LRIS ... -0.92293     0.0 210210000
LR.20150123.00300.fits      18     LRIS ... -0.92293     0.0 210210000
LR.20150123.00538.fits      18     LRIS ... -0.92293     0.0 210210000
LR.20150123.00792.fits      18     LRIS ... -0.92293     0.0 210210000
LR.20150123.01035.fits      18     LRIS ... -0.92293     0.0 210210000
LR.20150123.01289.fits      18     LRIS ... -0.92293     0.0 210210000
LR.20150123.01603.fits      18     LRIS ... -0.92293     0.0 210210000

Select columns acquired on UT date

In [12]:
sql = "select koaid, filehand from " + koatable + " where koaid like '%" + koaid_date + "%'"
results = koa.run_async(sql)
print(results)
table   = results.to_table()
table.write(outdir + instr + "_select_columns.tbl",format='ascii.ipac',overwrite=True)
<Table length=7>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
LR.20150123.00166.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00166.fits
LR.20150123.00300.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00300.fits
LR.20150123.00538.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00538.fits
LR.20150123.00792.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00792.fits
LR.20150123.01035.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01035.fits
LR.20150123.01289.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01289.fits
LR.20150123.01603.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01603.fits

Select a maximum number of records

In [13]:
sql = "select koaid, filehand, frameno from " +  koatable + " where koaid like '%" + koaid_date + "%'"
results = koa.run_async(sql, maxrec=20)
print (results)

table   = results.to_table()
table.write(outdir + instr + "_select_maxrec.tbl",format='ascii.ipac',overwrite=True)
<Table length=7>
        koaid          ... frameno
        object         ...  int32 
---------------------- ... -------
LR.20150123.00166.fits ...       8
LR.20150123.00300.fits ...       9
LR.20150123.00538.fits ...      10
LR.20150123.00792.fits ...      11
LR.20150123.01035.fits ...      12
LR.20150123.01289.fits ...      13
LR.20150123.01603.fits ...      14

Select records by time range

In [14]:
sql = "select koaid, filehand from " + koatable + " where (utdatetime >= to_date('" + startdate + "', 'yyyy-mm-dd HH24:MI:SS') and utdatetime <= to_date('" + enddate + "', 'yyyy-mm-dd HH24:MI:SS'))"
results  = koa.run_async(sql)
print (results)
table_daterange=results.to_table()
table.write (outdir + instr + "_table_daterange.vot",format='votable',overwrite=True)
<Table length=2454>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
LB.20150119.72427.fits /koadata25/LRIS/20150120/lev0/LB.20150119.72427.fits
LB.20150119.72771.fits /koadata25/LRIS/20150120/lev0/LB.20150119.72771.fits
LB.20150119.78800.fits /koadata25/LRIS/20150120/lev0/LB.20150119.78800.fits
LB.20150120.02709.fits /koadata25/LRIS/20150120/lev0/LB.20150120.02709.fits
LB.20150120.03172.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03172.fits
LB.20150120.03492.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03492.fits
LB.20150120.03563.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03563.fits
LB.20150120.03635.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03635.fits
LB.20150120.03704.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03704.fits
LB.20150120.03774.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03774.fits
                   ...                                                  ...
LR.20150222.61846.fits /koadata25/LRIS/20150222/lev0/LR.20150222.61846.fits
LR.20150222.62009.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62009.fits
LR.20150222.62140.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62140.fits
LR.20150222.62272.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62272.fits
LR.20150222.62402.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62402.fits
LR.20150222.62533.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62533.fits
LR.20150224.75522.fits /koadata25/LRIS/20150225/lev0/LR.20150224.75522.fits
LR.20150224.76599.fits /koadata25/LRIS/20150225/lev0/LR.20150224.76599.fits
LR.20150224.76732.fits /koadata25/LRIS/20150225/lev0/LR.20150224.76732.fits
LR.20150224.77046.fits /koadata25/LRIS/20150225/lev0/LR.20150224.77046.fits

Select records by program ID

In [15]:
sql= ("select koaid, filehand, progid from " + koatable + " where (progid = '" + program + "')")
results  = koa.run_async(sql)
print(results)
table_progid=results.to_table()
table.write (outdir + instr + "_progid.vot",format='votable',overwrite=True)
<Table length=140>
        koaid          ... progid
        object         ... object
---------------------- ... ------
LB.20180512.01660.fits ...   C305
LB.20180512.03116.fits ...   C305
LB.20180512.03186.fits ...   C305
LB.20180512.03390.fits ...   C305
LB.20180512.03473.fits ...   C305
LB.20180512.04976.fits ...   C305
LB.20180512.05041.fits ...   C305
LB.20180512.05101.fits ...   C305
LB.20180512.05161.fits ...   C305
LB.20180512.05221.fits ...   C305
                   ... ...    ...
LR.20180512.40161.fits ...   C305
LR.20180512.41491.fits ...   C305
LR.20180512.42818.fits ...   C305
LR.20180512.44145.fits ...   C305
LR.20180512.45472.fits ...   C305
LR.20180512.46800.fits ...   C305
LR.20180512.48127.fits ...   C305
LR.20180512.49454.fits ...   C305
LR.20180512.50781.fits ...   C305
LR.20180512.52109.fits ...   C305

Spatial cone search with column selection

In [16]:
sql=("select koaid, filehand, ra, dec from " + koatable + " where contains(point('J2000', ra, dec), circle('J2000', " + circle_pos + ")) = 1")
results  = koa.run_async(sql)
print(results)
table_spatial_1=results.to_table()
table.write (outdir + instr + "_table_spatial_circle.vot",format='votable',overwrite=True)
<Table length=101>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
LB.20010529.20873.fits ... 30.00314
LB.20010529.21063.fits ... 30.00314
LB.20010529.21331.fits ... 29.99631
LB.20010529.21738.fits ...   29.996
LB.20010529.21996.fits ... 29.99589
LB.20010529.22312.fits ... 29.99603
LB.20010529.23664.fits ... 29.99608
LB.20010529.25022.fits ...   29.996
LB.20011211.57154.fits ... 29.79737
LB.20020109.59153.fits ... 29.80264
                   ... ...      ...
LR.20100121.57605.fits ... 29.78861
LR.20120428.22909.fits ... 29.79953
LR.20120428.23105.fits ... 29.79953
LR.20180512.20205.fits ... 29.79947
LR.20180512.20631.fits ... 29.79947
LR.20180512.20857.fits ... 29.79947
LR.20180512.21080.fits ... 29.79947
LR.20181229.56431.fits ... 29.79942
LR.20181229.56539.fits ... 29.79942
LR.20190304.33130.fits ... 29.94239
In [17]:
sql=("select koaid, filehand, ra, dec from " + koatable + " where contains(point('J2000', ra, dec), box('J2000', " + box_pos + ")) = 1")
results  = koa.run_async(sql)
print(results)
table_spatial_1=results.to_table()
table.write (outdir + instr + "_table_spatial_box.vot",format='votable',overwrite=True)
<Table length=95>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
LB.20010529.20873.fits ... 30.00314
LB.20010529.21063.fits ... 30.00314
LB.20010529.21331.fits ... 29.99631
LB.20010529.21738.fits ...   29.996
LB.20010529.21996.fits ... 29.99589
LB.20010529.22312.fits ... 29.99603
LB.20010529.23664.fits ... 29.99608
LB.20010529.25022.fits ...   29.996
LB.20011211.57154.fits ... 29.79737
LB.20020109.59153.fits ... 29.80264
                   ... ...      ...
LR.20100121.57605.fits ... 29.78861
LR.20120428.22909.fits ... 29.79953
LR.20120428.23105.fits ... 29.79953
LR.20180512.20205.fits ... 29.79947
LR.20180512.20631.fits ... 29.79947
LR.20180512.20857.fits ... 29.79947
LR.20180512.21080.fits ... 29.79947
LR.20181229.56431.fits ... 29.79942
LR.20181229.56539.fits ... 29.79942
LR.20190304.33130.fits ... 29.94239
In [18]:
sql=("select koaid, filehand, ra, dec from " + koatable + " where contains(point('icrs', ra, dec), polygon('J2000', " + polygon_pos + ")) = 1")
results  = koa.run_async(sql)
print(results)
table_spatial_1=results.to_table()
table.write (outdir + instr + "_table_spatial_polygon.vot",format='votable',overwrite=True)
<Table length=101>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
LB.20010529.20873.fits ... 30.00314
LB.20010529.21063.fits ... 30.00314
LB.20010529.21331.fits ... 29.99631
LB.20010529.21738.fits ...   29.996
LB.20010529.21996.fits ... 29.99589
LB.20010529.22312.fits ... 29.99603
LB.20010529.23664.fits ... 29.99608
LB.20010529.25022.fits ...   29.996
LB.20011211.57154.fits ... 29.79737
LB.20020109.59153.fits ... 29.80264
                   ... ...      ...
LR.20100121.57605.fits ... 29.78861
LR.20120428.22909.fits ... 29.79953
LR.20120428.23105.fits ... 29.79953
LR.20180512.20205.fits ... 29.79947
LR.20180512.20631.fits ... 29.79947
LR.20180512.20857.fits ... 29.79947
LR.20180512.21080.fits ... 29.79947
LR.20181229.56431.fits ... 29.79942
LR.20181229.56539.fits ... 29.79942
LR.20190304.33130.fits ... 29.94239

Count number of records in date range

In [19]:
sql=("select count(*) as count from " + koatable +  " where date_obs=to_date('" + date + "', 'YYYY-MM-DD')")
results = koa.run_async(sql)
print(results)
<Table length=1>
count
int32
-----
    7

Count total number of records in table

In [20]:
sql="select count(*) as total from " + koatable
results = koa.run_async(sql)
print(results)
<Table length=1>
total 
int32 
------
548756

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA