Using the nexsciTAP Server with the PyVO Client to Access NIRC2 Data At KOA In Asynchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, including cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform synchronous TAP-based queries for public raw science and calibation data acquired with the NIRC2 near-infrared imager ; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (July, 2021)

Set up

In [2]:
from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")
import time

import sys
import os
import time
import warnings
warnings.filterwarnings('ignore')

Query by date

In [8]:
sql = "select koaid, filehand from koa_nirc2 where koaid like '%20140906%'"

results= koa.run_async(sql)
print(results)


# write results to VOTable format
table=results.to_table()
table.write ('./table_ipacascii.vot',format='votable',overwrite=True)
<Table length=356>
        koaid                                 filehand                      
        object                                 object                       
---------------------- -----------------------------------------------------
N2.20140906.11394.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11394.fits
N2.20140906.11451.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11451.fits
N2.20140906.11508.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11508.fits
N2.20140906.11564.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11564.fits
N2.20140906.11621.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11621.fits
N2.20140906.11677.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11677.fits
N2.20140906.11734.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11734.fits
N2.20140906.11790.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11790.fits
N2.20140906.11847.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11847.fits
N2.20140906.11903.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11903.fits
                   ...                                                   ...
N2.20140906.36033.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36033.fits
N2.20140906.36059.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36059.fits
N2.20140906.36330.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36330.fits
N2.20140906.36402.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36402.fits
N2.20140906.36474.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36474.fits
N2.20140906.36556.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36556.fits
N2.20140906.36630.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36630.fits
N2.20140906.36799.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36799.fits
N2.20140906.36871.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36871.fits
N2.20140906.36944.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36944.fits

PyVO by default returns results in VOTable format. Convert them to IPAC ASCII and CSV format.

In [9]:
# write results to an IPAC ASCII table
sql = "select koaid, filehand from koa_nirc2 where koaid like '%20140906%' "
results = koa.run_async(sql)
table=results.to_table()
table.write ('./table_ipacascii.tbl',format='ascii.ipac',overwrite=True)
In [10]:
#  write results to CSV file
sql = "select koaid, filehand from koa_nirc2 where koaid like '%20140906%' "
results = koa.run_async(sql)
table=results.to_table()
table.write ('./table_csv.csv',format='csv', overwrite=True)

Select all keywords for data acquired on UT Date

In [13]:
sql = "select * from koa_nirc2 where koaid like '%20140906%' "
results = koa.run_async(sql)
print (results)

## write to an output file
table_selectall=results.to_table()
table_selectall.write ('./table_selectall.vot',format='votable',overwrite=True)
<Table length=356>
        koaid          propint airmass aoaoamed ...     y        z      spt_ind 
                        months                  ...                             
        object          int32  float64  int32   ...  float64  float64    int32  
---------------------- ------- ------- -------- ... --------- -------- ---------
N2.20140906.11394.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11451.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11508.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11564.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11621.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11677.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11734.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11790.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
N2.20140906.11847.fits      18 1.41291       17 ...  0.599661 0.707107 233211302
                   ...     ...     ...      ... ...       ...      ...       ...
N2.20140906.36033.fits      18 1.33107      243 ... -0.680536 0.641297 203030211
N2.20140906.36059.fits      18 1.33268      244 ... -0.680547 0.641282 203030211
N2.20140906.36330.fits      18 1.50117       26 ... -0.719692 0.645624 203032303
N2.20140906.36402.fits      18 1.50779       23 ... -0.719664 0.645646 203032303
N2.20140906.36474.fits      18 1.51454      132 ... -0.719682 0.645624 203032303
N2.20140906.36556.fits      18 1.52242       56 ... -0.719689 0.645626 203032303
N2.20140906.36630.fits      18 1.52953       66 ... -0.719667 0.645644 203032303
N2.20140906.36799.fits      18  1.5462      109 ... -0.719687 0.645628 203032303
N2.20140906.36871.fits      18 1.55344      126 ... -0.719669 0.645643 203032303
N2.20140906.36944.fits      18 1.56099      127 ... -0.719681 0.645628 203032303

Select columns acquired on UT date

In [14]:
sql = "select koaid, filehand  from koa_nirc2 where koaid like '%20140906%'"
results = koa.run_async(sql)
print (results)

table   = results.to_table()
table.write('./select_columns.tbl',format='ascii.ipac',overwrite=True)
<Table length=356>
        koaid                                 filehand                      
        object                                 object                       
---------------------- -----------------------------------------------------
N2.20140906.11394.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11394.fits
N2.20140906.11451.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11451.fits
N2.20140906.11508.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11508.fits
N2.20140906.11564.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11564.fits
N2.20140906.11621.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11621.fits
N2.20140906.11677.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11677.fits
N2.20140906.11734.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11734.fits
N2.20140906.11790.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11790.fits
N2.20140906.11847.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11847.fits
N2.20140906.11903.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11903.fits
                   ...                                                   ...
N2.20140906.36033.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36033.fits
N2.20140906.36059.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36059.fits
N2.20140906.36330.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36330.fits
N2.20140906.36402.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36402.fits
N2.20140906.36474.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36474.fits
N2.20140906.36556.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36556.fits
N2.20140906.36630.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36630.fits
N2.20140906.36799.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36799.fits
N2.20140906.36871.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36871.fits
N2.20140906.36944.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.36944.fits

Select a maximum number of records

In [17]:
sql="select koaid, filehand, frameno from koa_nirc2 \
    where koaid like '%20140906%'"
results = koa.run_async(sql, maxrec=20)
print (results)

table   = results.to_table()
table.write('./select_maxrec.tbl',format='ascii.ipac',overwrite=True)
<Table length=20>
        koaid          ... frameno
        object         ...  int32 
---------------------- ... -------
N2.20140906.11394.fits ...       1
N2.20140906.11451.fits ...       2
N2.20140906.11508.fits ...       3
N2.20140906.11564.fits ...       4
N2.20140906.11621.fits ...       5
N2.20140906.11677.fits ...       6
N2.20140906.11734.fits ...       7
N2.20140906.11790.fits ...       8
N2.20140906.11847.fits ...       9
N2.20140906.11903.fits ...      10
N2.20140906.12219.fits ...      11
N2.20140906.12254.fits ...      12
N2.20140906.12285.fits ...      13
N2.20140906.12317.fits ...      14
N2.20140906.12348.fits ...      15
N2.20140906.12380.fits ...      16
N2.20140906.12412.fits ...      17
N2.20140906.12443.fits ...      18
N2.20140906.12475.fits ...      19
N2.20140906.12507.fits ...      20

Select records by time range

In [18]:
sql="select koaid, filehand from koa_nirc2 \
     where (utdatetime >= to_date('2014-09-06 03:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2014-09-06 05:00:00', 'yyyy-mm-dd HH24:MI:SS'))"
results  = koa.run_async(sql)

print (results)

table_daterange=results.to_table()
table.write ('./table_daterange.vot',format='votable',overwrite=True)
<Table length=43>
        koaid                                 filehand                      
        object                                 object                       
---------------------- -----------------------------------------------------
N2.20140906.11394.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11394.fits
N2.20140906.11451.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11451.fits
N2.20140906.11508.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11508.fits
N2.20140906.11564.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11564.fits
N2.20140906.11621.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11621.fits
N2.20140906.11677.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11677.fits
N2.20140906.11734.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11734.fits
N2.20140906.11790.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11790.fits
N2.20140906.11847.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11847.fits
N2.20140906.11903.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.11903.fits
                   ...                                                   ...
N2.20140906.13132.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13132.fits
N2.20140906.13189.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13189.fits
N2.20140906.13249.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13249.fits
N2.20140906.13305.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13305.fits
N2.20140906.13362.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13362.fits
N2.20140906.13418.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13418.fits
N2.20140906.13475.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13475.fits
N2.20140906.13531.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13531.fits
N2.20140906.13588.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13588.fits
N2.20140906.13644.fits /koadata10/NIRC2/20140906/lev0/N2.20140906.13644.fits

Select records by program ID

In [21]:
sql= ("select koaid, filehand, progid  \
    from koa_nirc2 where (progid = 'N153N2') ")
results  = koa.run_async(sql)
print (results)

table_progid=results.to_table()
table.write ('./progid.vot',format='votable',overwrite=True)
<Table length=1923>
        koaid          ... progid
        object         ... object
---------------------- ... ------
N2.20081009.37236.fits ... N153N2
N2.20081009.37250.fits ... N153N2
N2.20081009.37264.fits ... N153N2
N2.20081009.37278.fits ... N153N2
N2.20081009.37292.fits ... N153N2
N2.20081009.37305.fits ... N153N2
N2.20081009.37319.fits ... N153N2
N2.20081009.37332.fits ... N153N2
N2.20081009.37346.fits ... N153N2
N2.20081009.37360.fits ... N153N2
                   ... ...    ...
N2.20140906.36033.fits ... N153N2
N2.20140906.36059.fits ... N153N2
N2.20140906.36330.fits ... N153N2
N2.20140906.36402.fits ... N153N2
N2.20140906.36474.fits ... N153N2
N2.20140906.36556.fits ... N153N2
N2.20140906.36630.fits ... N153N2
N2.20140906.36799.fits ... N153N2
N2.20140906.36871.fits ... N153N2
N2.20140906.36944.fits ... N153N2

Spatial cone search with column selection

In [22]:
sql=("select koaid, filehand, ra, dec  from koa_nirc2 where \
    contains(point('J2000', ra, dec), \
    circle('J2000', 238.0 27.6, 0.5)) = 1")


results  = koa.run_async(sql)
print(results)

table_spatial_1=results.to_table()
table.write ('./table_spatial_1.vot',format='votable',overwrite=True)
<Table length=130>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
N2.20140906.18417.fits ... 27.60311
N2.20140906.18437.fits ... 27.60297
N2.20140906.18450.fits ... 27.60324
N2.20140906.18462.fits ... 27.60297
N2.20140906.18476.fits ... 27.60324
N2.20140906.18593.fits ... 27.60311
N2.20140906.18603.fits ... 27.60311
N2.20140906.18614.fits ... 27.60311
N2.20140906.18624.fits ... 27.60311
N2.20140906.18635.fits ... 27.60311
                   ... ...      ...
N2.20140906.20063.fits ... 27.60304
N2.20140906.20074.fits ... 27.60304
N2.20140906.20084.fits ... 27.60304
N2.20140906.20095.fits ... 27.60304
N2.20140906.20106.fits ... 27.60304
N2.20140906.20127.fits ... 27.60318
N2.20140906.20138.fits ... 27.60318
N2.20140906.20149.fits ... 27.60318
N2.20140906.20159.fits ... 27.60318
N2.20140906.20170.fits ... 27.60318
In [23]:
sql=("select koaid, filehand, ra, dec from koa_nirc2 where \
     (contains(point('J2000',ra ,dec), box('J2000', 238.0 27.6, 0.5, 0.5)) = 1)")


results = koa.run_async(sql)
print (results)

table_spatial_box=results.to_table()
table.write('./table_spatial_box.vot',format='votable',overwrite=True)
<Table length=130>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
N2.20140906.18417.fits ... 27.60311
N2.20140906.18437.fits ... 27.60297
N2.20140906.18450.fits ... 27.60324
N2.20140906.18462.fits ... 27.60297
N2.20140906.18476.fits ... 27.60324
N2.20140906.18593.fits ... 27.60311
N2.20140906.18603.fits ... 27.60311
N2.20140906.18614.fits ... 27.60311
N2.20140906.18624.fits ... 27.60311
N2.20140906.18635.fits ... 27.60311
                   ... ...      ...
N2.20140906.20063.fits ... 27.60304
N2.20140906.20074.fits ... 27.60304
N2.20140906.20084.fits ... 27.60304
N2.20140906.20095.fits ... 27.60304
N2.20140906.20106.fits ... 27.60304
N2.20140906.20127.fits ... 27.60318
N2.20140906.20138.fits ... 27.60318
N2.20140906.20149.fits ... 27.60318
N2.20140906.20159.fits ... 27.60318
N2.20140906.20170.fits ... 27.60318
In [24]:
sql=("select koaid, filehand, ra, dec from koa_nirc2 where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',237.5 27.1 238.5 27.1 238.5 28.1 237.5 28.1)) = 1")

results = koa.run_async(sql)
print(results)

table_spatial_P=results.to_table()
table.write ('./table_spatial_P.vot',format='votable',overwrite=True)
<Table length=130>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
N2.20140906.18417.fits ... 27.60311
N2.20140906.18437.fits ... 27.60297
N2.20140906.18450.fits ... 27.60324
N2.20140906.18462.fits ... 27.60297
N2.20140906.18476.fits ... 27.60324
N2.20140906.18593.fits ... 27.60311
N2.20140906.18603.fits ... 27.60311
N2.20140906.18614.fits ... 27.60311
N2.20140906.18624.fits ... 27.60311
N2.20140906.18635.fits ... 27.60311
                   ... ...      ...
N2.20140906.20063.fits ... 27.60304
N2.20140906.20074.fits ... 27.60304
N2.20140906.20084.fits ... 27.60304
N2.20140906.20095.fits ... 27.60304
N2.20140906.20106.fits ... 27.60304
N2.20140906.20127.fits ... 27.60318
N2.20140906.20138.fits ... 27.60318
N2.20140906.20149.fits ... 27.60318
N2.20140906.20159.fits ... 27.60318
N2.20140906.20170.fits ... 27.60318

Count number of records in date range

In [6]:
sql=("select count(*) as count from koa_nirc2 where \
    date_obs=to_date('2014-09-06', 'YYYY-MM-DD')")

results = koa.run_async(sql)
print(results)
<Table length=1>
count
int32
-----
  356

Count total number of records in table

In [5]:
sql="select count(*) as total from koa_nirc2"

results = koa.run_async(sql)
print(results)
<Table length=1>
total 
int32 
------
923238

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA