Using the nexsciTAP Server with the PyVO Client to Access Data From KOA In Asynchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform asynchronous TAP-based queries for public raw science and calibation data acquired with the OSIRIS integral field spectrograph these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (March 2021)

Set up

In [3]:
from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")

import sys
import os
import time

Query by date

In [4]:
sql = "select koaid, filehand from koa_osiris where koaid like '%20150529%' "

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsAS = job.fetch_result()
print(resultsAS)

# write output file
table=resultsAS.to_table()
table.write ('./table_select_AStime.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu/TAP/async/tap_y6bh2jrk
job_id:  tap_y6bh2jrk
phase:   PENDING
phase:   COMPLETED
<Table length=123>
        koaid                                 filehand                       
        object                                 object                        
---------------------- ------------------------------------------------------
OI.20150529.41901.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.41901.fits
OI.20150529.42096.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42096.fits
OI.20150529.42327.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42327.fits
OI.20150529.46047.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.46047.fits
OS.20150529.01240.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01240.fits
OS.20150529.01400.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01400.fits
OS.20150529.01554.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01554.fits
OS.20150529.01606.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01606.fits
OS.20150529.01737.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01737.fits
OS.20150529.01820.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01820.fits
                   ...                                                    ...
OS.20150529.54981.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.54981.fits
OS.20150529.55131.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55131.fits
OS.20150529.55209.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55209.fits
OS.20150529.55541.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55541.fits
OS.20150529.55592.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55592.fits
OS.20150529.55667.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55667.fits
OS.20150529.55735.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55735.fits
OS.20150529.83374.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83374.fits
OS.20150529.83590.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83590.fits
OS.20150529.83772.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83772.fits

PyVO by default returns results in VOTable format. Write to IPAC ASCII and CSV format for storage.

In [5]:
# IPAC ASCII
table=resultsAS.to_table()
table.write ('./table_ipacasciiAS_time.tbl',format='ascii.ipac', \
             overwrite=True)

# CSV
table_csv=resultsAS.to_table()
table.write ('./table_csvAS_time.csv',format='csv',overwrite=True)

Select all keywords for data acquired on UT Date

In [6]:
sql = "select * from koa_osiris where koaid like '%20150529%' "
job = koa.submit_job(sql)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsALL= job.fetch_result()

print(resultsALL)

# write output file in VTtable format
table_selectall=resultsALL.to_table()
table_selectall.write ('./table_select_asynchALL.vot', \
                        format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu/TAP/async/tap_0s8wlpfu
job_id:  tap_0s8wlpfu
phase:   PENDING
phase:   COMPLETED
<Table length=123>
        koaid          propint        ofname        ...     z      spt_ind 
        object          int32         object        ...  float64    int32  
---------------------- ------- -------------------- ... --------- ---------
OI.20150529.41901.fits      18 i150529_a024016.fits ...  0.026658 202000111
OI.20150529.42096.fits      18 i150529_a024017.fits ...  0.026658 202000111
OI.20150529.42327.fits      18 i150529_a024018.fits ...  0.026749 202000111
OI.20150529.46047.fits      18 i150529_a026019.fits ...  0.026722 202000111
OS.20150529.01240.fits      18 s150529_a001002.fits ... -0.001497 132000000
OS.20150529.01400.fits      18 s150529_a001003.fits ... -0.001497 132000000
OS.20150529.01554.fits      18 s150529_a001004.fits ... -0.001497 132000000
OS.20150529.01606.fits      18 s150529_a001005.fits ... -0.001497 132000000
OS.20150529.01737.fits      18 s150529_a001006.fits ... -0.001497 132000000
OS.20150529.01820.fits      18 s150529_a001007.fits ... -0.001497 132000000
                   ...     ...                  ... ...       ...       ...
OS.20150529.54981.fits      18 s150529_s033001.fits ...  0.070675 202001131
OS.20150529.55131.fits      18 s150529_s034001.fits ...  0.070675 202001131
OS.20150529.55209.fits      18 s150529_s034002.fits ...  0.070665 202001131
OS.20150529.55541.fits      18 s150529_s035001.fits ... -0.102996 130033020
OS.20150529.55592.fits      18 s150529_s035002.fits ... -0.102986 130033020
OS.20150529.55667.fits      18 s150529_s036001.fits ... -0.103003 130033020
OS.20150529.55735.fits      18 s150529_s036002.fits ... -0.102989 130033020
OS.20150529.83374.fits      30 s150529_a001001.fits ...       0.0 200020010
OS.20150529.83590.fits      30 s150529_a001002.fits ...       0.0 200020010
OS.20150529.83772.fits      30 s150529_a001003.fits ...       0.0 200020010

Select columns on UT date

In [7]:
sql = "select koaid, filehand from koa_osiris where koaid like '%20150529%' "

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsASK = job.fetch_result()

print(resultsASK)

## write output file
table=resultsASK.to_table()
table.write ('./table_asynch_cols.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu/TAP/async/tap_5mzvcw01
job_id:  tap_5mzvcw01
phase:   PENDING
phase:   COMPLETED
<Table length=123>
        koaid                                 filehand                       
        object                                 object                        
---------------------- ------------------------------------------------------
OI.20150529.41901.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.41901.fits
OI.20150529.42096.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42096.fits
OI.20150529.42327.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42327.fits
OI.20150529.46047.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.46047.fits
OS.20150529.01240.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01240.fits
OS.20150529.01400.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01400.fits
OS.20150529.01554.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01554.fits
OS.20150529.01606.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01606.fits
OS.20150529.01737.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01737.fits
OS.20150529.01820.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01820.fits
                   ...                                                    ...
OS.20150529.54981.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.54981.fits
OS.20150529.55131.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55131.fits
OS.20150529.55209.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55209.fits
OS.20150529.55541.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55541.fits
OS.20150529.55592.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55592.fits
OS.20150529.55667.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55667.fits
OS.20150529.55735.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55735.fits
OS.20150529.83374.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83374.fits
OS.20150529.83590.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83590.fits
OS.20150529.83772.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83772.fits

Select maximum number of records to return from a query

In [8]:
sql = "select koaid, filehand, frameno from koa_osiris where koaid like '%20150529%'"
job = koa.submit_job(sql,maxrec=20)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsASK20 = job.fetch_result( )

print(resultsASK20)

## write output file
table=resultsASK20.to_table()
table.write ('./table_asynch_max20.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu/TAP/async/tap_0gkum4bu
job_id:  tap_0gkum4bu
phase:   PENDING
phase:   COMPLETED
<Table length=123>
        koaid          ... frameno
        object         ...  int32 
---------------------- ... -------
OI.20150529.41901.fits ...      16
OI.20150529.42096.fits ...      17
OI.20150529.42327.fits ...      18
OI.20150529.46047.fits ...      19
OS.20150529.01240.fits ...       2
OS.20150529.01400.fits ...       3
OS.20150529.01554.fits ...       4
OS.20150529.01606.fits ...       5
OS.20150529.01737.fits ...       6
OS.20150529.01820.fits ...       7
                   ... ...     ...
OS.20150529.54981.fits ...       1
OS.20150529.55131.fits ...       1
OS.20150529.55209.fits ...       2
OS.20150529.55541.fits ...       1
OS.20150529.55592.fits ...       2
OS.20150529.55667.fits ...       1
OS.20150529.55735.fits ...       2
OS.20150529.83374.fits ...       1
OS.20150529.83590.fits ...       2
OS.20150529.83772.fits ...       3

Query records by time range

In [9]:
sql="select koaid, filehand from koa_osiris \
   where (utdatetime >= to_date('2019-05-29 00:00:00', 'yyyy-mm-dd HH24:MI:SS') \
   and utdatetime <= to_date('2019-05-31 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"


job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsDATERANGE = job.fetch_result()
print(resultsDATERANGE)

## write to output file
table=resultsDATERANGE.to_table()
table.write ('./table_asynch_daterange.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu/TAP/async/tap_3p33aeo_
job_id:  tap_3p33aeo_
phase:   PENDING
phase:   COMPLETED
<Table length=770>
        koaid                                 filehand                       
        object                                 object                        
---------------------- ------------------------------------------------------
OI.20190529.03140.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.03140.fits
OI.20190529.03275.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.03275.fits
OI.20190529.09103.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.09103.fits
OI.20190529.09810.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.09810.fits
OI.20190529.10083.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.10083.fits
OI.20190529.10304.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.10304.fits
OI.20190529.10373.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.10373.fits
OI.20190529.10484.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.10484.fits
OI.20190529.10526.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.10526.fits
OI.20190529.10568.fits /koadata23/OSIRIS/20190529/lev0/OI.20190529.10568.fits
                   ...                                                    ...
OS.20190531.30532.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.30532.fits
OS.20190531.32019.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.32019.fits
OS.20190531.32077.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.32077.fits
OS.20190531.32435.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.32435.fits
OS.20190531.33107.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.33107.fits
OS.20190531.33822.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.33822.fits
OS.20190531.34492.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.34492.fits
OS.20190531.35167.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.35167.fits
OS.20190531.35915.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.35915.fits
OS.20190531.36614.fits /koadata23/OSIRIS/20190531/lev0/OS.20190531.36614.fits

Query by program ID

In [ ]:
sql=("select koaid, filehand, progid from \
      koa_osiris where (progid = 'N021')")

job = koa.submit_job(sql)
print('url:    ', job.url)

print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsPROGID= job.fetch_result( )

print(resultsPROGID)

## write to output file
table_progID_C307=resultsPROGID.to_table()
table.write ('./table_progID.vot',format='votable',overwrite=True)
In [13]:
sql=("select koaid, filehand, ra, dec from koa_osiris \
    where contains(point('icrs', ra, dec), circle('icrs',270.4 -27.2 1.0))   = 1")

    
job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA= job.fetch_result( )

print(resultsSPA)

# write to output file
table=resultsSPA.to_table()
table.write ('./table_asynch_spatial_cone.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu//TAP/async/tap_rkc1uoi1
job_id:  tap_rkc1uoi1
phase:   PENDING
phase:   COMPLETED
<Table length=122>
        koaid          ...    dec   
        object         ...  float64 
---------------------- ... ---------
OI.20190421.48934.fits ...   -27.547
OI.20190421.49094.fits ...   -27.547
OI.20190421.49287.fits ...   -27.547
OI.20190421.49344.fits ...   -27.547
OI.20190421.49394.fits ...   -27.547
OI.20190421.49442.fits ...   -27.547
OI.20190421.49493.fits ...   -27.547
OI.20190421.49560.fits ... -27.54716
OI.20190421.49610.fits ... -27.54716
OI.20190421.49659.fits ... -27.54716
                   ... ...       ...
OI.20190513.43202.fits ... -27.54727
OI.20190513.43252.fits ... -27.54727
OI.20190513.43301.fits ... -27.54727
OI.20190513.43350.fits ... -27.54727
OS.20160813.24913.fits ... -26.52847
OS.20160813.24983.fits ... -26.52819
OS.20160813.25147.fits ... -26.52847
OS.20160813.25218.fits ... -26.52819
OS.20160813.25335.fits ... -26.52847
OS.20160813.25407.fits ... -26.52819
In [15]:
sql=("select koaid, filehand, ra, dec from koa_osiris where \
     contains(point('icrs', ra, dec), \
     box('icrs',23.4, 30.60,1.0,120)) = 1")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA_BOX= job.fetch_result( )

print(resultsSPA_BOX)

## write to output file
table=resultsSPA_BOX.to_table()
table.write ('./table_spatial_Box.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu//TAP/async/tap_vb1kvgd2
job_id:  tap_vb1kvgd2
phase:   PENDING
phase:   COMPLETED
<Table length=2645>
        koaid          ...    dec   
        object         ...  float64 
---------------------- ... ---------
OI.20051012.12061.fits ...  89.96783
OI.20051012.12087.fits ...  89.96783
OI.20051012.12111.fits ...  89.96783
OI.20051012.12135.fits ...  89.96783
OI.20051012.12160.fits ...  89.96783
OI.20051012.12957.fits ...  89.96783
OI.20051012.13651.fits ...  89.96783
OI.20051012.13676.fits ...  89.96783
OI.20051012.13692.fits ...  89.96783
OI.20051012.13717.fits ...  89.96783
                   ... ...       ...
OS.20210114.24190.fits ... -13.08002
OS.20210114.24245.fits ... -13.08002
OS.20210114.24299.fits ... -13.08002
OS.20210114.24357.fits ... -13.08002
OS.20210114.24411.fits ... -13.08002
OS.20210114.24464.fits ... -13.08002
OS.20210114.24523.fits ... -13.08002
OS.20210114.24578.fits ... -13.08002
OS.20210114.24632.fits ... -13.08002
OS.20210114.24690.fits ... -13.08002
In [16]:
sql=("select koaid, filehand, ra, dec from koa_osiris where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA_POLY= job.fetch_result( )

print(resultsSPA_POLY)

## write to output file
table=resultsSPA_POLY.to_table()
table.write ('./table_spatial_P.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu//TAP/async/tap_22rywf9q
job_id:  tap_22rywf9q
phase:   PENDING
phase:   COMPLETED
<Table length=215>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
OS.20160416.49503.fits ... 54.41594
OS.20160416.49605.fits ... 54.41594
OS.20160416.49631.fits ... 54.41594
OS.20160416.49658.fits ... 54.41594
OS.20160416.49683.fits ... 54.41594
OS.20160416.49709.fits ... 54.41594
OS.20160416.49744.fits ... 54.41594
OS.20160416.49769.fits ... 54.41594
OS.20160416.49796.fits ... 54.41594
OS.20160416.49826.fits ... 54.41594
                   ... ...      ...
OS.20170509.43806.fits ... 54.23825
OS.20170509.43836.fits ... 54.23825
OS.20170509.43864.fits ... 54.23825
OS.20170509.43890.fits ... 54.23784
OS.20170509.43920.fits ... 54.23784
OS.20170509.43948.fits ... 54.23825
OS.20170509.43974.fits ... 54.23825
OS.20170509.44001.fits ... 54.23825
OS.20170509.44027.fits ... 54.23825
OS.20170509.44083.fits ... 54.23825

Count number of records in a time range

In [17]:
sql=("select count(*) from koa_osiris where \
     (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') and  \
     utdatetime <= to_date('2009-01-31 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)
url:     https://koa.ipac.caltech.edu//TAP/async/tap_xu5eoivv
job_id:  tap_xu5eoivv
phase:   PENDING
phase:   COMPLETED
<Table length=1>
count(*)
 int32  
--------
     306

Count total number of records

In [18]:
sql="select count(*) as total from koa_osiris"

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
      time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)
url:     https://koa.ipac.caltech.edu/TAP/async/tap_ugmsh04v
job_id:  tap_ugmsh04v
phase:   PENDING
phase:   COMPLETED
<Table length=1>
total 
int32 
------
239106

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA