Using the nexsciTAP Server with the PyVO Client to Access DEIMOS Data From KOA In Asynchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform asynchronous TAP-based queries for public raw science and calibation data acquired with the DEep Imaging Multi-Object Spectrograph (DEIMOS) these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (March, 2021)

Set up

In [13]:
from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")

import sys
import os
import time

Query by date

In [14]:
sql = "select koaid, filehand from koa_deimos where koaid like '%20150901%' "


job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsAS = job.fetch_result()
print(resultsAS)

# write output file
table=resultsAS.to_table()
table.write ('./table_select_AS20040824.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_86z77pw6
job_id:  tap_86z77pw6
phase:   PENDING
phase:   COMPLETED
<Table length=7>
        koaid                                 filehand                       
        object                                 object                        
---------------------- ------------------------------------------------------
DE.20150901.07424.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.07424.fits
DE.20150901.08740.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08740.fits
DE.20150901.08811.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08811.fits
DE.20150901.08879.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08879.fits
DE.20150901.08950.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08950.fits
DE.20150901.09052.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09052.fits
DE.20150901.09522.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09522.fits

PyVO by default returns results in VOTable format. Write to IPAC ASCII and CSV format for storage.

In [4]:
# IPAC ASCII
table=resultsAS.to_table()
table.write ('./table_ipacasciiAS_20040824.tbl',format='ascii.ipac', \
             overwrite=True)

# CSV
table_csv=resultsAS.to_table()
table.write ('./table_csvAS_20040824.csv',format='csv',overwrite=True)

Select all keywords for data acquired on UT Date

In [15]:
sql = "select * from koa_deimos where koaid like '%20150901%' "
job = koa.submit_job(sql)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsALL= job.fetch_result()

print(resultsALL)

# write output file in VTtable format
table_selectall=resultsALL.to_table()
table_selectall.write ('./table_select_asynchALL.vot', \
                        format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_ccq_bpd2
job_id:  tap_ccq_bpd2
phase:   PENDING
phase:   COMPLETED
<Table length=7>
        koaid               ofname     filesize_mb ...  spt_ind   cntr 
        object              object       float64   ...   int32   int32 
---------------------- --------------- ----------- ... --------- ------
DE.20150901.07424.fits d0827_0037.fits          -- ... 132000000 150853
DE.20150901.08740.fits d0827_0038.fits          -- ... 132000000 150854
DE.20150901.08811.fits d0827_0039.fits          -- ... 132000000 150855
DE.20150901.08879.fits d0827_0040.fits          -- ... 132000000 150856
DE.20150901.08950.fits d0827_0041.fits          -- ... 132000000 150857
DE.20150901.09052.fits d0827_0042.fits          -- ... 132000000 150858
DE.20150901.09522.fits d0827_0043.fits          -- ... 132000000 150859

Select columns on UT date

In [6]:
sql = "select koaid, filehand from koa_deimos where koaid like '%20150901%' "

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsASK = job.fetch_result()

print(resultsASK)

## write output file
table=resultsASK.to_table()
table.write ('./table_asynch_orderdoesnot.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_bf6idfu5
job_id:  tap_bf6idfu5
phase:   PENDING
phase:   COMPLETED
<Table length=7>
        koaid                                 filehand                       
        object                                 object                        
---------------------- ------------------------------------------------------
DE.20150901.07424.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.07424.fits
DE.20150901.08740.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08740.fits
DE.20150901.08811.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08811.fits
DE.20150901.08879.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08879.fits
DE.20150901.08950.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08950.fits
DE.20150901.09052.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09052.fits
DE.20150901.09522.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09522.fits

Select maximum number of records to return from a query

In [7]:
sql = "select koaid, filehand, frameno from koa_deimos where koaid like '%%20150901%'"
job = koa.submit_job(sql,maxrec=5)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsASK5 = job.fetch_result( )

print(resultsASK5)

## write output file
table=resultsASK5.to_table()
table.write ('./table_asynch_max5.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_b4rtbdac
job_id:  tap_b4rtbdac
phase:   PENDING
phase:   COMPLETED
<Table length=7>
        koaid          ... frameno
        object         ...  int32 
---------------------- ... -------
DE.20150901.07424.fits ...      37
DE.20150901.08740.fits ...      38
DE.20150901.08811.fits ...      39
DE.20150901.08879.fits ...      40
DE.20150901.08950.fits ...      41
DE.20150901.09052.fits ...      42
DE.20150901.09522.fits ...      43

Query records by time range

In [8]:
sql="select koaid, filehand from koa_deimos \
   where (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') \
   and utdatetime <= to_date('2009-09-30 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"


job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsDATERANGE = job.fetch_result()
print(resultsDATERANGE)

## write to output file
table=resultsDATERANGE.to_table()
table.write ('./table_asynch_daterange.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_lyfbax5z
job_id:  tap_lyfbax5z
phase:   PENDING
phase:   COMPLETED
<Table length=10489>
        koaid                                 filehand                       
        object                                 object                        
---------------------- ------------------------------------------------------
DE.20090121.68384.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68384.fits
DE.20090121.68430.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68430.fits
DE.20090121.68479.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68479.fits
DE.20090121.68527.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68527.fits
DE.20090121.68579.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68579.fits
DE.20090121.68620.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68620.fits
DE.20090121.68664.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68664.fits
DE.20090121.68704.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68704.fits
DE.20090121.68750.fits /koadata38/DEIMOS/20090121/lev0/DE.20090121.68750.fits
DE.20090124.04044.fits /koadata38/DEIMOS/20090124/lev0/DE.20090124.04044.fits
                   ...                                                    ...
DF.20090826.14294.fits /koadata38/DEIMOS/20090826/lev0/DF.20090826.14294.fits
DF.20090916.12190.fits /koadata38/DEIMOS/20090916/lev0/DF.20090916.12190.fits
DF.20090917.09973.fits /koadata38/DEIMOS/20090917/lev0/DF.20090917.09973.fits
DF.20090917.10593.fits /koadata38/DEIMOS/20090917/lev0/DF.20090917.10593.fits
DF.20090917.10854.fits /koadata38/DEIMOS/20090917/lev0/DF.20090917.10854.fits
DF.20090918.11868.fits /koadata38/DEIMOS/20090918/lev0/DF.20090918.11868.fits
DF.20090919.09499.fits /koadata38/DEIMOS/20090919/lev0/DF.20090919.09499.fits
DF.20090919.09632.fits /koadata38/DEIMOS/20090919/lev0/DF.20090919.09632.fits
DF.20090920.12222.fits /koadata38/DEIMOS/20090920/lev0/DF.20090920.12222.fits
DF.20090921.07611.fits /koadata38/DEIMOS/20090921/lev0/DF.20090921.07611.fits

Query by program ID

In [9]:
sql=("select koaid, filehand, progid, ha, az, detector, gratenam from \
      koa_deimos where (progid = 'U024')")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsPROGID= job.fetch_result( )

print(resultsPROGID)

## write to output file
table_progID_C307=resultsPROGID.to_table()
table.write ('./table_progID_C307.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_9l4d87l8
job_id:  tap_9l4d87l8
phase:   PENDING
phase:   COMPLETED
<Table length=222>
        koaid          ... gratenam
        object         ...  object 
---------------------- ... --------
DE.20190605.06053.fits ...    600ZD
DE.20190605.10617.fits ...    600ZD
DE.20190605.10691.fits ...    600ZD
DE.20190605.12860.fits ...    600ZD
DE.20190605.12938.fits ...    600ZD
DE.20190605.13014.fits ...    600ZD
DE.20190605.13089.fits ...    600ZD
DE.20190605.13197.fits ...    600ZD
DE.20190605.13275.fits ...    600ZD
DE.20190605.13351.fits ...    600ZD
                   ... ...      ...
DF.20190606.09732.fits ...     830G
DF.20190705.10814.fits ...    600ZD
DF.20190705.10919.fits ...    600ZD
DF.20190705.11248.fits ...    600ZD
DF.20190705.11506.fits ...     830G
DF.20190706.06286.fits ...    600ZD
DF.20190706.06705.fits ...    600ZD
DF.20190706.06915.fits ...    600ZD
DF.20190706.07219.fits ...     830G
DF.20190706.07433.fits ...     830G
In [18]:
sql=("select koaid, filehand, ra, dec from koa_deimos \
    where contains(point('icrs', ra, dec), circle('icrs',23.4, 30.60,1.0))   = 1")

    
job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA= job.fetch_result( )

print(resultsSPA)

# write to output file
table=resultsSPA.to_table()
table.write ('./table_asynch_spatial_cone.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_hgguze_2
job_id:  tap_hgguze_2
phase:   PENDING
phase:   COMPLETED
<Table length=401>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
DE.20031018.19913.fits ... 30.95914
DE.20031018.20064.fits ... 30.95914
DE.20031018.20249.fits ... 30.95881
DE.20031018.20392.fits ... 30.95872
DE.20031018.20535.fits ... 30.95878
DE.20031018.21134.fits ... 30.95325
DE.20031018.21289.fits ... 30.95297
DE.20031018.21429.fits ...   30.953
DE.20031018.21616.fits ...   30.953
DE.20031018.22378.fits ... 30.95336
                   ... ...      ...
DE.20181204.27263.fits ... 30.73564
DE.20181204.28434.fits ... 30.73564
DE.20181204.29603.fits ... 30.73564
DE.20181204.31202.fits ... 30.60358
DE.20181204.31296.fits ... 30.60375
DE.20181204.31462.fits ... 30.60375
DE.20181204.32522.fits ... 30.60375
DE.20181204.33582.fits ... 30.60375
DE.20181204.34642.fits ... 30.60375
DE.20181204.35702.fits ... 30.60375
In [19]:
sql=("select koaid, filehand, ra, dec from koa_deimos where \
     contains(point('icrs', ra, dec), \
     box('icrs',23.4, 30.60,1.0,120)) = 1")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA_BOX= job.fetch_result( )

print(resultsSPA_BOX)

## write to output file
table=resultsSPA_BOX.to_table()
table.write ('./table_spatial_Box.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_07051521
job_id:  tap_07051521
phase:   PENDING
phase:   COMPLETED
<Table length=2464>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
DE.20020810.12784.fits ... 89.98549
DE.20020810.12904.fits ... 89.98549
DE.20020810.13013.fits ... 89.98549
DE.20020810.13122.fits ... 89.98549
DE.20020810.13271.fits ... 89.98549
DE.20020810.13393.fits ... 89.98549
DE.20020810.13504.fits ... 89.98549
DE.20020810.13613.fits ... 89.98549
DE.20020810.13777.fits ... 89.98549
DE.20020810.13913.fits ... 89.98549
                   ... ...      ...
DE.20190825.77507.fits ...    90.01
DE.20190825.77609.fits ...    90.01
DE.20190825.80669.fits ...    90.01
DE.20190825.82643.fits ...    90.01
DE.20190825.82739.fits ...    90.01
DE.20190825.84498.fits ...    90.01
DE.20190825.85012.fits ...    90.01
DE.20190825.85257.fits ...    90.01
DE.20190825.85455.fits ...    90.01
DE.20190825.85569.fits ...    90.01
In [20]:
sql=("select koaid, filehand, ra, dec from koa_deimos where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsSPA_POLY= job.fetch_result( )

print(resultsSPA_POLY)

## write to output file
table=resultsSPA_POLY.to_table()
table.write ('./table_spatial_P.vot',format='votable',overwrite=True)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_e4rlymj_
job_id:  tap_e4rlymj_
phase:   PENDING
phase:   COMPLETED
<Table length=37>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
DE.20100313.56134.fits ... 54.34917
DE.20100313.56233.fits ... 54.34917
DE.20100313.56356.fits ... 54.34917
DE.20100313.56466.fits ... 54.34917
DE.20100313.56584.fits ... 54.34917
DE.20100313.56780.fits ... 54.34917
DE.20100313.56898.fits ... 54.34917
DE.20100313.57009.fits ... 54.34917
DE.20120226.55210.fits ... 54.26819
DE.20120226.55594.fits ... 54.26869
                   ... ...      ...
DE.20160428.48021.fits ...  54.4045
DE.20160704.30504.fits ... 54.45833
DE.20160704.30975.fits ... 54.45833
DE.20160704.31540.fits ... 54.45833
DE.20160704.31910.fits ... 54.45833
DE.20160704.32336.fits ... 54.45833
DE.20160704.32707.fits ... 54.45833
DE.20180705.24528.fits ... 54.43911
DE.20180705.26500.fits ... 54.43772
DE.20180705.26576.fits ... 54.43772

Count number of records in a time range

In [21]:
sql=("select count(*) from koa_deimos where \
     (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') and  \
     utdatetime <= to_date('2009-01-31 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_at_gnca_
job_id:  tap_at_gnca_
phase:   PENDING
phase:   COMPLETED
<Table length=1>
count(*)
 int32  
--------
     404

Count total number of records

In [22]:
sql="select count(*) as total from koa_deimos"

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
      time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)
url:     https://koa.ipac.caltech.edu:443//TAP/async/tap_gr0jgb6o
job_id:  tap_gr0jgb6o
phase:   PENDING
phase:   COMPLETED
<Table length=1>
total 
int32 
------
172291

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA