Using the nexsciTAP Server with the TAP+ Client to Access NIRES Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the Near-Infrared Echellette Spectrometer (NIRES); these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.1 (July, 2021)

Set up

In [2]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [2]:
sql = "select koaid, filehand from koa_nires where koaid like '%20181218%' "
job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid                                 filehand                      
---------------------- -----------------------------------------------------
NI.20181218.00833.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00833.fits
NI.20181218.00884.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00884.fits
NI.20181218.00900.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00900.fits
NI.20181218.00920.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00920.fits
NI.20181218.01053.fits /koadata32/NIRES/20181218/lev0/NI.20181218.01053.fits
NI.20181218.06720.fits /koadata32/NIRES/20181218/lev0/NI.20181218.06720.fits
NI.20181218.38030.fits /koadata32/NIRES/20181218/lev0/NI.20181218.38030.fits
NI.20181218.38072.fits /koadata32/NIRES/20181218/lev0/NI.20181218.38072.fits
NI.20181218.38931.fits /koadata32/NIRES/20181218/lev0/NI.20181218.38931.fits
NI.20181218.41511.fits /koadata32/NIRES/20181218/lev0/NI.20181218.41511.fits
                   ...                                                   ...
NR.20181218.58036.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58036.fits
NR.20181218.58232.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58232.fits
NR.20181218.58315.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58315.fits
NR.20181218.58369.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58369.fits
NR.20181218.58431.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58431.fits
NR.20181218.58486.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58486.fits
NR.20181218.59684.fits /koadata32/NIRES/20181218/lev0/NR.20181218.59684.fits
NR.20181218.61034.fits /koadata32/NIRES/20181218/lev0/NR.20181218.61034.fits
NR.20181218.61377.fits /koadata32/NIRES/20181218/lev0/NR.20181218.61377.fits
NR.20181218.61713.fits /koadata32/NIRES/20181218/lev0/NR.20181218.61713.fits
NR.20181218.62049.fits /koadata32/NIRES/20181218/lev0/NR.20181218.62049.fits
Length = 142 rows

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [3]:
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date

In [4]:
sql = "select * from koa_nires where koaid like '%20181218%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
        koaid              airmass      ...          z           spt_ind 
---------------------- ---------------- ... ------------------- ---------
NI.20181218.00833.fits  2.5499223339225 ... -0.0018423685161384 132000000
NI.20181218.00884.fits 2.52727144745537 ... -0.0018423685161384 132000000
NI.20181218.00900.fits 2.51987090273206 ... -0.0018423685161384 132000000
NI.20181218.00920.fits 1.41291034446565 ...   0.707106781186547 233211302
NI.20181218.01053.fits 1.41291034446565 ...   0.707106781186547 233211302
NI.20181218.06720.fits 1.41291034446565 ...   0.707106781186547 233211302
NI.20181218.38030.fits 2.10447878934741 ...   0.030416396917203 220212302
NI.20181218.38072.fits 2.09380205617529 ...   0.030464894585008 220212302
NI.20181218.38931.fits 1.89565151628755 ...  0.0303583042928143 220212302
NI.20181218.41511.fits 1.50422806361724 ...  0.0304141290388975 220212302
                   ...              ... ...                 ...       ...
NR.20181218.58036.fits  1.0612797656373 ...   0.375120327424073 212332031
NR.20181218.58232.fits  1.0563770346548 ...   0.375140389030245 212332031
NR.20181218.58315.fits 1.05439626120835 ...   0.375131490758868 212332031
NR.20181218.58369.fits 1.05309873300075 ...   0.375122592452924 212332031
NR.20181218.58431.fits 1.05171401297284 ...    0.37511385590073 212332031
NR.20181218.58486.fits 1.05041742467484 ...   0.375104957526282 212332031
NR.20181218.59684.fits 13.3748135344195 ...  0.0010297440766911 230210000
NR.20181218.61034.fits 13.3748135344195 ...  0.0010297440766911 230210000
NR.20181218.61377.fits 13.3748135344195 ...  0.0010297440766911 230210000
NR.20181218.61713.fits 13.3748135344195 ...  0.0010297440766911 230210000
NR.20181218.62049.fits 13.3748135344195 ...  0.0010297440766911 230210000
Length = 142 rows

Select columns acquired on UT Date

In [5]:
sql = "select koaid, filehand from koa_nires \
    where koaid like '%20181218%' "

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='columns.tbl', \
                     output_format='ipac')
        koaid                                 filehand                      
---------------------- -----------------------------------------------------
NI.20181218.00833.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00833.fits
NI.20181218.00884.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00884.fits
NI.20181218.00900.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00900.fits
NI.20181218.00920.fits /koadata32/NIRES/20181218/lev0/NI.20181218.00920.fits
NI.20181218.01053.fits /koadata32/NIRES/20181218/lev0/NI.20181218.01053.fits
NI.20181218.06720.fits /koadata32/NIRES/20181218/lev0/NI.20181218.06720.fits
NI.20181218.38030.fits /koadata32/NIRES/20181218/lev0/NI.20181218.38030.fits
NI.20181218.38072.fits /koadata32/NIRES/20181218/lev0/NI.20181218.38072.fits
NI.20181218.38931.fits /koadata32/NIRES/20181218/lev0/NI.20181218.38931.fits
NI.20181218.41511.fits /koadata32/NIRES/20181218/lev0/NI.20181218.41511.fits
                   ...                                                   ...
NR.20181218.58036.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58036.fits
NR.20181218.58232.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58232.fits
NR.20181218.58315.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58315.fits
NR.20181218.58369.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58369.fits
NR.20181218.58431.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58431.fits
NR.20181218.58486.fits /koadata32/NIRES/20181218/lev0/NR.20181218.58486.fits
NR.20181218.59684.fits /koadata32/NIRES/20181218/lev0/NR.20181218.59684.fits
NR.20181218.61034.fits /koadata32/NIRES/20181218/lev0/NR.20181218.61034.fits
NR.20181218.61377.fits /koadata32/NIRES/20181218/lev0/NR.20181218.61377.fits
NR.20181218.61713.fits /koadata32/NIRES/20181218/lev0/NR.20181218.61713.fits
NR.20181218.62049.fits /koadata32/NIRES/20181218/lev0/NR.20181218.62049.fits
Length = 142 rows

Select a maximum number of records

In [6]:
sql = "select top 20 koaid, filehand, frameno from koa_nires where koaid like '%20181218%'"
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecors.tbl', output_format='ipac')

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='top20.tbl', \
                     output_format='ipac')
        koaid          ... frameno
---------------------- ... -------
NI.20181218.00884.fits ...       1
NI.20181218.00900.fits ...       2
NI.20181218.00920.fits ...       3
NI.20181218.01053.fits ...       4
NI.20181218.06720.fits ...       1
NR.20181218.44514.fits ...      33
NR.20181218.46365.fits ...      34
NR.20181218.46700.fits ...      35
NR.20181218.47034.fits ...      36
NR.20181218.47369.fits ...      37
NR.20181218.51515.fits ...      47
NR.20181218.51854.fits ...      48
NR.20181218.52189.fits ...      49
NR.20181218.52541.fits ...      50
NR.20181218.52875.fits ...      51
NR.20181218.53210.fits ...      52
NR.20181218.53544.fits ...      53
NR.20181218.54083.fits ...      54
NR.20181218.56264.fits ...      60
NR.20181218.56283.fits ...      61

Select columns and search by datetime range

In [6]:
sql="select koaid, filehand from koa_nires \
     where (utdatetime >= to_date('2018-12-28 03:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2018-12-28 03:30:00', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='datetime_range.tbl', \
                     output_format='ipac')
        koaid                                 filehand                      
---------------------- -----------------------------------------------------
NR.20181228.11306.fits /koadata32/NIRES/20181228/lev0/NR.20181228.11306.fits
NR.20181228.11423.fits /koadata32/NIRES/20181228/lev0/NR.20181228.11423.fits
NR.20181228.11537.fits /koadata32/NIRES/20181228/lev0/NR.20181228.11537.fits
NR.20181228.11653.fits /koadata32/NIRES/20181228/lev0/NR.20181228.11653.fits
NR.20181228.11769.fits /koadata32/NIRES/20181228/lev0/NR.20181228.11769.fits
NR.20181228.11887.fits /koadata32/NIRES/20181228/lev0/NR.20181228.11887.fits
NR.20181228.12002.fits /koadata32/NIRES/20181228/lev0/NR.20181228.12002.fits
NR.20181228.12120.fits /koadata32/NIRES/20181228/lev0/NR.20181228.12120.fits
NR.20181228.12235.fits /koadata32/NIRES/20181228/lev0/NR.20181228.12235.fits
NR.20181228.12353.fits /koadata32/NIRES/20181228/lev0/NR.20181228.12353.fits
NR.20181228.12469.fits /koadata32/NIRES/20181228/lev0/NR.20181228.12469.fits
NR.20181228.12586.fits /koadata32/NIRES/20181228/lev0/NR.20181228.12586.fits

Search by program ID

In [8]:
sql= ("select koaid, filehand, progid  \
    from koa_nires where (progid = 'Y025') ")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='progID.tbl', \
                     output_format='ipac')
        koaid          ... progid
---------------------- ... ------
NI.20181218.06720.fits ...   Y025
NI.20181218.38030.fits ...   Y025
NI.20181218.38072.fits ...   Y025
NI.20181218.38931.fits ...   Y025
NI.20181218.41511.fits ...   Y025
NI.20181218.41553.fits ...   Y025
NI.20181218.41699.fits ...   Y025
NI.20181218.41782.fits ...   Y025
NI.20181218.43651.fits ...   Y025
NI.20181218.45361.fits ...   Y025
                   ... ...    ...
NR.20181229.59273.fits ...   Y025
NR.20181229.59496.fits ...   Y025
NR.20181229.59633.fits ...   Y025
NR.20181229.59769.fits ...   Y025
NR.20181230.38256.fits ...   Y025
NR.20181230.38374.fits ...   Y025
NR.20181230.38492.fits ...   Y025
NR.20181230.38610.fits ...   Y025
NR.20181230.38727.fits ...   Y025
NR.20181230.38946.fits ...   Y025
NR.20181230.39082.fits ...   Y025
Length = 545 rows
In [9]:
sql=("select koaid, filehand,ra, dec from koa_nires where \
    contains(point('icrs', ra, dec), \
    circle('icrs', 194.25 22.03, 1.0)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
NI.20180131.55969.fits ... 22.03131
NI.20180131.55991.fits ... 22.03131
NI.20180131.56045.fits ... 22.03038
NI.20180131.56327.fits ... 22.03038
NI.20180131.56391.fits ...  22.0304
NI.20180131.56724.fits ... 22.03036
NI.20180131.57056.fits ... 22.03036
NI.20180131.57390.fits ...  22.0304
NI.20181218.56718.fits ... 22.03128
NI.20181218.56785.fits ... 22.03233
                   ... ...      ...
NR.20190519.17611.fits ... 22.03126
NR.20190519.17731.fits ... 22.03136
NR.20190519.17757.fits ... 22.03126
NR.20190519.17822.fits ... 22.03135
NR.20190519.17849.fits ... 22.03126
NR.20190519.17955.fits ... 22.03135
NR.20190519.17972.fits ... 22.03126
NR.20190519.31091.fits ... 22.03135
NR.20190519.31329.fits ... 22.03133
NR.20190519.31562.fits ... 22.03133
NR.20190519.31798.fits ... 22.03135
Length = 117 rows
In [10]:
sql=("select koaid, filehand, ra, dec from koa_nires where \
     contains(point('icrs', ra, dec), \
     box('icrs',194.25 22.03,1.0,120)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
NI.20180131.55969.fits ... 22.03131
NI.20180131.55991.fits ... 22.03131
NI.20180131.56045.fits ... 22.03038
NI.20180131.56327.fits ... 22.03038
NI.20180131.56391.fits ...  22.0304
NI.20180131.56724.fits ... 22.03036
NI.20180131.57056.fits ... 22.03036
NI.20180131.57390.fits ...  22.0304
NI.20181218.56718.fits ... 22.03128
NI.20181218.56785.fits ... 22.03233
                   ... ...      ...
NR.20190519.17611.fits ... 22.03126
NR.20190519.17731.fits ... 22.03136
NR.20190519.17757.fits ... 22.03126
NR.20190519.17822.fits ... 22.03135
NR.20190519.17849.fits ... 22.03126
NR.20190519.17955.fits ... 22.03135
NR.20190519.17972.fits ... 22.03126
NR.20190519.31091.fits ... 22.03135
NR.20190519.31329.fits ... 22.03133
NR.20190519.31562.fits ... 22.03133
NR.20190519.31798.fits ... 22.03135
Length = 117 rows

TAP+ returns the first 2,000 records in this query

In [12]:
sql=("select koaid, filehand, ra, dec from koa_nires  where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
NI.20180302.50965.fits ... 54.39806
NI.20180302.51018.fits ... 54.39711
NI.20180302.51208.fits ... 54.39895
NI.20180302.51624.fits ... 54.39801
NI.20180302.51921.fits ... 54.39895
NI.20180302.52284.fits ...  54.3999
NI.20180302.52673.fits ... 54.39895
NI.20180302.53019.fits ... 54.39801
NI.20180302.53396.fits ... 54.39895
NI.20180302.53991.fits ... 54.39982
                   ... ...      ...
NR.20180302.51623.fits ... 54.39801
NR.20180302.51921.fits ... 54.39895
NR.20180302.52284.fits ...  54.3999
NR.20180302.52673.fits ... 54.39895
NR.20180302.53019.fits ... 54.39801
NR.20180302.53396.fits ... 54.39895
NR.20180302.53991.fits ... 54.39982
NR.20180302.54384.fits ... 54.39793
NR.20180302.56342.fits ... 54.30151
NR.20180302.56693.fits ... 54.30223
NR.20180302.57162.fits ...  54.3008
Length = 35 rows

Count number of records on date

In [13]:
sql=("select count(*) from koa_nires where \
    date_obs=to_date('2018-12-18', 'YYYY-MM-DD')")

job = koa.launch_job(sql)
r = job.get_results()
print(r)
count(*)
--------
     142

Count total number of records in table

In [14]:
sql="select count(*) as total from koa_nires"

job = koa.launch_job(sql)
r = job.get_results()
print(r)
total
-----
 2000

Note: There are 200,000+ records that satisfy this query, but TAP+ only returns the first 2,000 in synchronous mode


Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA