Using the nexsciTAP Server with the PyVO Client to Access KCWI Data At KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, including cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform asynchronous TAP-based queries for public raw science and calibation data acquired with the KCWI integral field spectrograph; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (May, 2021)

Set up

In [1]:
from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")
import time

import sys
import os
import time

Query by date

In [2]:
sql = "select koaid, filehand from koa_kcwi where koaid like '%20190509%'"

results= koa.run_async(sql)
print(results)


# write results to VOTable format
table=results.to_table()
table.write ('./table_ipacascii.vot',format='ascii.ipac',overwrite=True)
<Table masked=True length=96>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
KB.20190509.07554.fits /koadata28/KCWI/20190509/lev0/KB.20190509.07554.fits
KB.20190509.08692.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08692.fits
KB.20190509.08718.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08718.fits
KB.20190509.08743.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08743.fits
KB.20190509.08770.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08770.fits
KB.20190509.08795.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08795.fits
KB.20190509.08821.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08821.fits
KB.20190509.08846.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08846.fits
KB.20190509.08872.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08872.fits
KB.20190509.08897.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08897.fits
                   ...                                                  ...
KB.20190509.49752.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49752.fits
KB.20190509.49811.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49811.fits
KB.20190509.49876.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49876.fits
KB.20190509.49934.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49934.fits
KF.20190509.07636.fits /koadata28/KCWI/20190509/lev0/KF.20190509.07636.fits
KF.20190509.18675.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18675.fits
KF.20190509.18849.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18849.fits
KF.20190509.18929.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18929.fits
KF.20190509.25327.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25327.fits
KF.20190509.25416.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25416.fits

PyVO by default returns results in VOTable format. Convert them to IPAC ASCII and CSV format.

In [3]:
# write results to an IPAC ASCII table
sql = "select koaid, filehand from koa_kcwi where koaid like '%20190509%' "
results = koa.run_async(sql)
table=results.to_table()
table.write ('./table_ipacascii.tbl',format='ascii.ipac',overwrite=True)
In [4]:
#  write results to CSV file
sql = "select koaid, filehand from koa_kcwi where koaid like '%20190509%' "
results = koa.run_async(sql)
table=results.to_table()
table.write ('./table_csv.csv',format='csv', overwrite=True)

Select all keywords for data acquired on UT Date

In [5]:
sql = "select * from koa_kcwi where koaid like '%20190509%' "
results = koa.run_async(sql)
print (results)

## write to an output file
table_selectall=results.to_table()
table_selectall.write ('./table_selectall.vot',format='votable',overwrite=True)
<Table masked=True length=96>
        koaid          propint filesize_mb ...     y        z      spt_ind 
        object          int32    float64   ...  float64  float64    int32  
---------------------- ------- ----------- ... --------- -------- ---------
KB.20190509.07554.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08692.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08718.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08743.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08770.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08795.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08821.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08846.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08872.fits      18          -- ...  0.599661 0.707107 233211302
KB.20190509.08897.fits      18          -- ...  0.599661 0.707107 233211302
                   ...     ...         ... ...       ...      ...       ...
KB.20190509.49752.fits      18          -- ... -0.420835 0.900349 211211221
KB.20190509.49811.fits      18          -- ... -0.420835 0.900349 211211221
KB.20190509.49876.fits      18          -- ... -0.420835 0.900349 211211221
KB.20190509.49934.fits      18          -- ... -0.420835 0.900349 211211221
KF.20190509.07636.fits      18          -- ...  0.599661 0.707107 233211302
KF.20190509.18675.fits      18          -- ...  0.253495 0.685911 223233033
KF.20190509.18849.fits      18          -- ...  0.253495 0.685911 223233033
KF.20190509.18929.fits      18          -- ...  0.253495 0.685911 223233033
KF.20190509.25327.fits      18          -- ...  0.196024  0.27296 220322212
KF.20190509.25416.fits      18          -- ...  0.196024  0.27296 220322212

Select columns acquired on UT date

In [7]:
sql = "select koaid, filehand  from koa_kcwi where koaid like '%20190509%'"
results = koa.run_async(sql)
print (results)

table   = results.to_table()
table.write('./select_columns.vot',format='ascii.ipac',overwrite=True)
<Table masked=True length=96>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
KB.20190509.07554.fits /koadata28/KCWI/20190509/lev0/KB.20190509.07554.fits
KB.20190509.08692.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08692.fits
KB.20190509.08718.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08718.fits
KB.20190509.08743.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08743.fits
KB.20190509.08770.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08770.fits
KB.20190509.08795.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08795.fits
KB.20190509.08821.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08821.fits
KB.20190509.08846.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08846.fits
KB.20190509.08872.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08872.fits
KB.20190509.08897.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08897.fits
                   ...                                                  ...
KB.20190509.49752.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49752.fits
KB.20190509.49811.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49811.fits
KB.20190509.49876.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49876.fits
KB.20190509.49934.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49934.fits
KF.20190509.07636.fits /koadata28/KCWI/20190509/lev0/KF.20190509.07636.fits
KF.20190509.18675.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18675.fits
KF.20190509.18849.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18849.fits
KF.20190509.18929.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18929.fits
KF.20190509.25327.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25327.fits
KF.20190509.25416.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25416.fits

Select a maximum number of records

In [8]:
sql="select koaid, filehand, frameno from koa_kcwi \
    where koaid like '%20190509%'"
results = koa.run_async(sql, maxrec=20)
print (results)

table   = results.to_table()
table.write('./select_maxrec.tbl',format='ascii.ipac',overwrite=True)
<Table masked=True length=20>
        koaid          ... frameno
        object         ...  int32 
---------------------- ... -------
KB.20190509.07554.fits ...       1
KB.20190509.08692.fits ...       2
KB.20190509.08718.fits ...       3
KB.20190509.08743.fits ...       4
KB.20190509.08770.fits ...       5
KB.20190509.08795.fits ...       6
KB.20190509.08821.fits ...       7
KB.20190509.08846.fits ...       8
KB.20190509.08872.fits ...       9
KB.20190509.08897.fits ...      10
KB.20190509.09347.fits ...      11
KB.20190509.09418.fits ...      12
KB.20190509.09494.fits ...      13
KB.20190509.09525.fits ...      14
KB.20190509.09585.fits ...      15
KB.20190509.09645.fits ...      16
KB.20190509.09705.fits ...      17
KB.20190509.09765.fits ...      18
KB.20190509.09826.fits ...      19
KB.20190509.09921.fits ...      20

Select records by time range

In [9]:
sql="select koaid, filehand from koa_kcwi \
     where (utdatetime >= to_date('2019-05-09 03:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2019-05-09 03:30:00', 'yyyy-mm-dd HH24:MI:SS'))"
results  = koa.run_async(sql)

print (results)

table_daterange=results.to_table()
table.write ('./table_daterange.vot',format='votable',overwrite=True)
<Table masked=True length=17>
        koaid                                filehand                      
        object                                object                       
---------------------- ----------------------------------------------------
KB.20190509.11052.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11052.fits
KB.20190509.11110.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11110.fits
KB.20190509.11158.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11158.fits
KB.20190509.11187.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11187.fits
KB.20190509.11246.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11246.fits
KB.20190509.11305.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11305.fits
KB.20190509.11364.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11364.fits
KB.20190509.11422.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11422.fits
KB.20190509.11481.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11481.fits
KB.20190509.11652.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11652.fits
KB.20190509.11724.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11724.fits
KB.20190509.11796.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11796.fits
KB.20190509.12317.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12317.fits
KB.20190509.12392.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12392.fits
KB.20190509.12460.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12460.fits
KB.20190509.12493.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12493.fits
KB.20190509.12555.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12555.fits

Select records by program ID

In [10]:
sql= ("select koaid, filehand, progid  \
    from koa_kcwi where (progid = 'U216') ")
results  = koa.run_async(sql)
print (results)

table_progid=results.to_table()
table.write ('./progid.vot',format='votable',overwrite=True)
<Table masked=True length=357>
        koaid          ... progid
        object         ... object
---------------------- ... ------
KB.20191029.02565.fits ...   U216
KB.20191029.02592.fits ...   U216
KB.20191029.02621.fits ...   U216
KB.20191029.02648.fits ...   U216
KB.20191029.02676.fits ...   U216
KB.20191029.02704.fits ...   U216
KB.20191029.02732.fits ...   U216
KB.20191029.02761.fits ...   U216
KB.20191029.02789.fits ...   U216
KB.20191029.04150.fits ...   U216
                   ... ...    ...
KB.20191101.47975.fits ...   U216
KB.20191101.49236.fits ...   U216
KB.20191101.50496.fits ...   U216
KB.20191101.51753.fits ...   U216
KB.20191101.53021.fits ...   U216
KF.20191029.01516.fits ...   U216
KF.20191029.16281.fits ...   U216
KF.20191029.16402.fits ...   U216
KF.20191029.79371.fits ...   U216
KF.20191030.83125.fits ...   U216

Spatial cone search with column selection

In [11]:
sql=("select koaid, filehand, ra, dec  from koa_kcwi where \
    contains(point('icrs', ra, dec), \
    circle('icrs', 23.48 ,30.60 1.0)) = 1")

results  = koa.run_async(sql)
print(results)

table_spatial_1=results.to_table()
table.write ('./table_spatial_1.vot',format='votable',overwrite=True)
<Table masked=True length=46>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
KB.20191007.19853.fits ... 30.70153
KB.20191007.20637.fits ... 30.69856
KB.20191007.21303.fits ... 30.73003
KB.20191007.22362.fits ... 30.69642
KB.20191007.23112.fits ... 30.69642
KB.20191007.23772.fits ... 30.73003
KB.20191007.24613.fits ... 30.69108
KB.20191007.25504.fits ... 30.69117
KB.20191007.26159.fits ... 30.73003
KB.20191007.26883.fits ... 30.69333
                   ... ...      ...
KB.20191007.47014.fits ... 30.73294
KB.20191007.47783.fits ... 30.68761
KB.20191007.48477.fits ... 30.68761
KB.20191007.49134.fits ... 30.73294
KB.20191007.49836.fits ... 30.74714
KB.20191007.50501.fits ... 30.74267
KB.20191007.51157.fits ...   30.962
KB.20191007.51815.fits ... 30.74267
KB.20191007.52473.fits ... 30.74769
KB.20191007.53013.fits ... 30.74475
In [12]:
sql=("select koaid, filehand, ra, dec from koa_kcwi where \
     contains(point('icrs', ra, dec), \
     box('icrs',23.48 ,30.60,1.0,120)) = 1")

results = koa.run_async(sql)
print (results)

table_spatial_box=results.to_table()
table.write('./table_spatial_box.vot',format='votable',overwrite=True)
<Table masked=True length=910>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
KB.20170919.17047.fits ... 90.01667
KB.20170919.17126.fits ... 90.01667
KB.20170919.17211.fits ... 90.01667
KB.20170919.17300.fits ... 90.01667
KB.20170919.17411.fits ... 90.01667
KB.20170919.17639.fits ... 90.01667
KB.20170919.17865.fits ... 90.01667
KB.20170919.18233.fits ... 90.01667
KB.20170919.55567.fits ...     90.0
KB.20170919.55942.fits ...     90.0
                   ... ...      ...
KF.20190103.04308.fits ... 90.01719
KF.20190202.22921.fits ... 33.38131
KF.20190202.23172.fits ... 33.33467
KF.20190202.23233.fits ... 33.33467
KF.20190204.04009.fits ... 90.01719
KF.20190205.04782.fits ... 90.01719
KF.20190331.02925.fits ... 90.01719
KF.20190426.70623.fits ...     89.9
KF.20190427.74839.fits ... 90.01719
KF.20190628.04306.fits ...     90.0
In [13]:
sql=("select koaid, filehand, ra, dec from koa_kcwi where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',160.0, 43.2, 160.0, 43.0, 159.8, 43.0, 159.8, 43.2))) = 1")

results = koa.run_async(sql)
print(results)

table_spatial_P=results.to_table()
table.write ('./table_spatial_P.tbl',format='votable',overwrite=True)
<Table masked=True length=138>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
KB.20171123.56727.fits ... 43.10178
KB.20171123.57000.fits ... 43.10297
KB.20171124.57171.fits ... 43.10244
KB.20171124.57380.fits ... 43.10244
KB.20171124.57601.fits ... 43.10244
KB.20171225.45861.fits ... 43.10192
KB.20171225.46079.fits ... 43.10192
KB.20171225.46485.fits ... 43.10192
KB.20171225.46542.fits ... 43.10192
KB.20171225.46600.fits ... 43.10192
                   ... ...      ...
KB.20191029.56006.fits ... 43.11114
KB.20191029.56158.fits ... 43.11114
KB.20191029.56352.fits ... 43.11114
KB.20191029.56555.fits ... 43.11114
KB.20191029.56753.fits ... 43.11114
KB.20191029.56946.fits ... 43.11114
KB.20191029.57200.fits ... 43.11114
KB.20191029.57395.fits ... 43.11114
KB.20191031.55873.fits ... 43.10281
KB.20191031.55947.fits ... 43.10281

Count number of records in date range

In [18]:
sql=("select count(*) as count from koa_kcwi where \
    date_obs=to_date('2019-05-09', 'YYYY-MM-DD')")

results = koa.run_async(sql)
print(results)
<Table masked=True length=1>
count
int32
-----
   96

Count total number of records in table

In [15]:
sql="select count(*) as total from koa_kcwi"

results = koa.run_async(sql)
print(results)
<Table masked=True length=1>
total
int32
-----
19744

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA