Using the nexsciTAP Server with the PyVO Client to Access Data From KOA In Asynchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform asynchronous TAP-based queries for public raw science and calibation data acquired with the HIRES instrument; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (March, 2021)

Set up

In [1]:
from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")

import sys
import os
import time

Query by date

In [2]:
sql = "select koaid, filehand from koa_hires where koaid like '%20040824%' "


job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsAS = job.fetch_result()

print(resultsAS)

# write output file
table=resultsAS.to_table()
table.write ('./table_select_AS20040824.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_it_5jati
job_id:  tap_it_5jati
phase:   PENDING
phase:   COMPLETED
<Table length=105>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
HI.20040824.08204.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08204.fits
HI.20040824.08544.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08544.fits
HI.20040824.08591.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08591.fits
HI.20040824.08638.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08638.fits
HI.20040824.08685.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08685.fits
HI.20040824.08732.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08732.fits
HI.20040824.08779.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08779.fits
HI.20040824.08826.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08826.fits
HI.20040824.08873.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08873.fits
HI.20040824.08919.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08919.fits
                   ...                                                  ...
HI.20040824.54803.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54803.fits
HI.20040824.54849.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54849.fits
HI.20040824.54896.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54896.fits
HI.20040824.54944.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54944.fits
HI.20040824.54990.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54990.fits
HI.20040824.55037.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55037.fits
HI.20040824.55084.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55084.fits
HI.20040824.55131.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55131.fits
HI.20040824.55177.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55177.fits
HI.20040824.55224.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55224.fits

PyVO by default returns results in VOTable format. Write to IPAC ASCII and CSV format for storage.

In [4]:
# IPAC ASCII
table=resultsAS.to_table()
table.write ('./table_ipacasciiAS_20040824.tbl',format='ascii.ipac', \
             overwrite=True)

# CSV
table_csv=resultsAS.to_table()
table.write ('./table_csvAS_20040824.csv',format='csv',overwrite=True)

Select all keywords for data acquired on UT Date

In [5]:
sql = "select * from koa_hires where koaid like '%20040824%' "
job = koa.submit_job(sql)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsALL= job.fetch_result()

print(resultsALL)

# write output file in VTtable format
table_selectall=resultsALL.to_table()
table_selectall.write ('./table_select_asynchALL.vot', \
                        format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_1f06caa7
job_id:  tap_1f06caa7
phase:   PENDING
phase:   COMPLETED
<Table length=105>
   el   elaptime equinox eramode ...      z        spt_ind      ofname    
float64  int32   float64  object ...   float64      int32       object    
------- -------- ------- ------- ... ------------ --------- --------------
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2300.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2301.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2302.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2303.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2304.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2305.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2306.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2307.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2308.fits
   45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2309.fits
    ...      ...     ...     ... ...          ...       ...            ...
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2395.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2396.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2397.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2398.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2399.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2400.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2401.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2402.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2403.fits
   0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2404.fits

Select columns on UT date and order by signal-to-noise (sig2nois)

In [6]:
sql = "select koaid, filehand, sig2nois from koa_hires \
      where koaid like '%20040824%'order by sig2nois"

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsASK = job.fetch_result()

print(resultsASK)

## write output file
table=resultsASK.to_table()
table.write ('./table_asynch_orderbysig2nois.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_qpe_qenv
job_id:  tap_qpe_qenv
phase:   PENDING
phase:   COMPLETED
<Table length=105>
        koaid          ... sig2nois
        object         ... float64 
---------------------- ... --------
HI.20040824.09677.fits ...      2.0
HI.20040824.09724.fits ...      2.0
HI.20040824.54001.fits ...      2.0
HI.20040824.53947.fits ...      2.0
HI.20040824.45942.fits ...      2.0
HI.20040824.44695.fits ...      2.0
HI.20040824.44395.fits ...      2.0
HI.20040824.44343.fits ...      2.0
HI.20040824.43054.fits ...      2.0
HI.20040824.41806.fits ...      2.0
                   ... ...      ...
HI.20040824.09011.fits ...     11.0
HI.20040824.08965.fits ...     11.0
HI.20040824.08873.fits ...     11.0
HI.20040824.09105.fits ...     11.0
HI.20040824.08779.fits ...     11.0
HI.20040824.08732.fits ...     11.0
HI.20040824.08826.fits ...     11.0
HI.20040824.08591.fits ...     11.0
HI.20040824.08204.fits ...     11.0
HI.20040824.08685.fits ...     15.0

Select maximum number of records to return from a query

In [7]:
sql = "select koaid, filehand, frameno from koa_hires where koaid like '%20040824%'"
job = koa.submit_job(sql,maxrec=20)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsASK20 = job.fetch_result( )

print(resultsASK20)

## write output file
table=resultsASK20.to_table()
table.write ('./table_asynch_max20.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_xtckhb39
job_id:  tap_xtckhb39
phase:   PENDING
phase:   COMPLETED
<Table length=105>
        koaid          ... frameno
        object         ...  int32 
---------------------- ... -------
HI.20040824.08204.fits ...    2300
HI.20040824.08544.fits ...    2301
HI.20040824.08591.fits ...    2302
HI.20040824.08638.fits ...    2303
HI.20040824.08685.fits ...    2304
HI.20040824.08732.fits ...    2305
HI.20040824.08779.fits ...    2306
HI.20040824.08826.fits ...    2307
HI.20040824.08873.fits ...    2308
HI.20040824.08919.fits ...    2309
                   ... ...     ...
HI.20040824.54803.fits ...    2395
HI.20040824.54849.fits ...    2396
HI.20040824.54896.fits ...    2397
HI.20040824.54944.fits ...    2398
HI.20040824.54990.fits ...    2399
HI.20040824.55037.fits ...    2400
HI.20040824.55084.fits ...    2401
HI.20040824.55131.fits ...    2402
HI.20040824.55177.fits ...    2403
HI.20040824.55224.fits ...    2404

Query records by time range

In [8]:
sql="select koaid, filehand from koa_hires \
   where (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') \
   and utdatetime <= to_date('2009-01-05 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"


job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsDATERANGE = job.fetch_result()
print(resultsDATERANGE)

## write to output file
table=resultsDATERANGE.to_table()
table.write ('./table_asynch_daterange.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_r8sbq5ra
job_id:  tap_r8sbq5ra
phase:   PENDING
phase:   COMPLETED
<Table length=61>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
HI.20090101.09133.fits /koadata4/HIRES/20090101/lev0/HI.20090101.09133.fits
HI.20090101.09658.fits /koadata4/HIRES/20090101/lev0/HI.20090101.09658.fits
HI.20090101.09784.fits /koadata4/HIRES/20090101/lev0/HI.20090101.09784.fits
HI.20090104.02004.fits /koadata4/HIRES/20090104/lev0/HI.20090104.02004.fits
HI.20090104.02222.fits /koadata4/HIRES/20090104/lev0/HI.20090104.02222.fits
HI.20090104.02477.fits /koadata4/HIRES/20090104/lev0/HI.20090104.02477.fits
HI.20090105.04559.fits /koadata4/HIRES/20090105/lev0/HI.20090105.04559.fits
HI.20090105.05148.fits /koadata4/HIRES/20090105/lev0/HI.20090105.05148.fits
HI.20090105.06188.fits /koadata4/HIRES/20090105/lev0/HI.20090105.06188.fits
HI.20090105.08445.fits /koadata4/HIRES/20090105/lev0/HI.20090105.08445.fits
                   ...                                                  ...
HI.20090105.10587.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10587.fits
HI.20090105.10632.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10632.fits
HI.20090105.10678.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10678.fits
HI.20090105.10724.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10724.fits
HI.20090105.10770.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10770.fits
HI.20090105.10816.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10816.fits
HI.20090105.10862.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10862.fits
HI.20090105.10908.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10908.fits
HI.20090105.10954.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10954.fits
HI.20090105.11000.fits /koadata4/HIRES/20090105/lev0/HI.20090105.11000.fits

Query by program ID

In [9]:
sql=("select koaid, filehand, progid, iodin, koaimtyp from \
      koa_hires where (progid = 'C307')")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsPROGID= job.fetch_result( )

print(resultsPROGID)

## write to output file
table_progID_C307=resultsPROGID.to_table()
table.write ('./table_progID_C307.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_gmgadxry
job_id:  tap_gmgadxry
phase:   PENDING
phase:   COMPLETED
<Table length=538>
        koaid          ... koaimtyp
        object         ...  object 
---------------------- ... --------
HI.20180331.02525.fits ...    focus
HI.20180331.03342.fits ...    focus
HI.20180331.03598.fits ...    focus
HI.20180331.03786.fits ...    focus
HI.20180331.03943.fits ...    focus
HI.20180331.04060.fits ...    focus
HI.20180331.04218.fits ...    focus
HI.20180331.04338.fits ...    focus
HI.20180331.04502.fits ...  arclamp
HI.20180331.04578.fits ...  arclamp
                   ... ...      ...
HI.20180801.45058.fits ...   object
HI.20180801.45222.fits ...   object
HI.20180801.45356.fits ...   object
HI.20180801.45644.fits ...   object
HI.20180801.45867.fits ...   object
HI.20180801.45945.fits ...   object
HI.20180801.46157.fits ... flatlamp
HI.20180801.46246.fits ... flatlamp
HI.20180801.46399.fits ...  arclamp
HI.20180801.46463.fits ...  arclamp
In [10]:
sql=("select koaid, filehand, ra, dec from koa_hires \
    where contains(point('icrs', ra, dec), circle('icrs',262.0,17.0,1.0)) = 1")
    
job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA= job.fetch_result( )

print(resultsSPA)

# write to output file
table=resultsSPA.to_table()
table.write ('./table_asynch_spatial_cone.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_kquddjfx
job_id:  tap_kquddjfx
phase:   PENDING
phase:   COMPLETED
<Table length=42>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
HI.19980718.34975.fits ... 16.82711
HI.19980912.26279.fits ... 16.82489
HI.19980913.27523.fits ... 16.82269
HI.19980914.23928.fits ... 16.82186
HI.19980914.24204.fits ... 16.82186
HI.19980914.29108.fits ... 16.82222
HI.19980915.26653.fits ...  16.8225
HI.19980916.25837.fits ... 16.82369
HI.19980918.24196.fits ... 16.82519
HI.19990421.49245.fits ...  17.5081
                   ... ...      ...
HI.20040824.18925.fits ... 17.50994
HI.20040825.18800.fits ... 17.51019
HI.20041002.18948.fits ... 17.50994
HI.20041002.19291.fits ... 17.50994
HI.20041002.19935.fits ... 17.50994
HI.20041002.20579.fits ... 17.50994
HI.20050301.49856.fits ... 16.82503
HI.20160126.56113.fits ...  16.8245
HI.20160714.38961.fits ... 16.82319
HI.20180702.37596.fits ... 16.82131
In [11]:
sql=("select koaid, filehand, ra, dec from koa_hires where \
     contains(point('icrs', ra, dec), \
     box('icrs',262.0,17.0,2.0,120)) = 1")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA_BOX= job.fetch_result( )

print(resultsSPA_BOX)

## write to output file
table=resultsSPA_BOX.to_table()
table.write ('./table_spatial_Box.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_kp0i56nl
job_id:  tap_kp0i56nl
phase:   PENDING
phase:   COMPLETED
<Table length=1031>
        koaid          ...    dec   
        object         ...  float64 
---------------------- ... ---------
HI.19941214.78458.fits ...  29.92281
HI.19950716.29399.fits ... -37.29384
HI.19950716.29658.fits ... -37.29384
HI.19950716.29859.fits ... -37.29384
HI.19950716.30097.fits ... -37.29384
HI.19951129.15228.fits ...      25.0
HI.19951129.15327.fits ...      25.0
HI.19960523.16106.fits ...   5.33211
HI.19960523.16363.fits ...   5.33211
HI.19960523.16460.fits ...   5.33211
                   ... ...       ...
HI.20190614.36658.fits ...  34.27092
HI.20190616.31037.fits ...  67.30686
HI.20190715.29454.fits ...  60.05008
HI.20190715.29515.fits ...  60.05008
HI.20190715.29575.fits ...  60.05008
HI.20190715.29654.fits ...  67.30667
HI.20190715.29719.fits ...  67.30667
HI.20190715.29786.fits ...  67.30667
HI.20190715.29853.fits ...  67.30667
HI.20190715.29924.fits ...  67.30667
In [12]:
sql=("select koaid, filehand, ra, dec from koa_hires where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA_POLY= job.fetch_result( )

print(resultsSPA_POLY)

## write to output file
table=resultsSPA_POLY.to_table()
table.write ('./table_spatial_P.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_e33dhjl4
job_id:  tap_e33dhjl4
phase:   PENDING
phase:   COMPLETED
<Table length=28>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
HI.20040114.51275.fits ... 54.35381
HI.20040114.53552.fits ... 54.35381
HI.20040114.55829.fits ... 54.35381
HI.20040114.58182.fits ... 54.35383
HI.20060420.33559.fits ... 54.35292
HI.20060420.35412.fits ... 54.35292
HI.20060420.37265.fits ... 54.35292
HI.20060420.39117.fits ... 54.35292
HI.20060420.41232.fits ... 54.31703
HI.20060420.43085.fits ... 54.31703
                   ... ...      ...
HI.20110905.19585.fits ... 54.27675
HI.20110909.17581.fits ... 54.27289
HI.20110909.18064.fits ... 54.27289
HI.20110909.18415.fits ... 54.27289
HI.20110909.19067.fits ... 54.27289
HI.20110909.19719.fits ... 54.27289
HI.20110920.17641.fits ... 54.27347
HI.20110923.18047.fits ... 54.27414
HI.20110923.18126.fits ... 54.27414
HI.20110923.18478.fits ... 54.27414

Count number of records in a time range

In [13]:
sql=("select count(*) from koa_hires where \
     (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') \
     and utdatetime <= to_date('2009-01-05 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_c5f71cr2
job_id:  tap_c5f71cr2
phase:   PENDING
phase:   COMPLETED
<Table length=1>
count(*)
 int32  
--------
      61

Count total number of records

In [14]:
sql="select count(*) as total from koa_hires"

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
      time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_sl5m6l09
job_id:  tap_sl5m6l09
phase:   PENDING
phase:   COMPLETED
<Table length=1>
total 
int32 
------
414960

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA

In [ ]: