Using the nexsciTAP Server with the TAP+ Client to Access NIRSPEC Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the NIRSPEC cross-dispersed echelle spectrograph; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (July, 2021)

Set up

In [21]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [2]:
sql = "select koaid, filehand from koa_nirspec where koaid like '%20170605%' "
job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid          ...
---------------------- ...
NC.20170605.45884.fits ...
NC.20170605.45978.fits ...
NC.20170605.46085.fits ...
NC.20170605.46186.fits ...
NC.20170605.46226.fits ...
NS.20170605.44190.fits ...
NS.20170605.44221.fits ...
NS.20170605.44231.fits ...
NS.20170605.44242.fits ...
NS.20170605.44252.fits ...
                   ... ...
NS.20170605.55442.fits ...
NS.20170605.55455.fits ...
NS.20170605.55467.fits ...
NS.20170605.55480.fits ...
NS.20170605.55491.fits ...
NS.20170605.55602.fits ...
NS.20170605.55628.fits ...
NS.20170605.55645.fits ...
NS.20170605.55754.fits ...
NS.20170605.55794.fits ...
NS.20170605.55811.fits ...
Length = 137 rows

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [3]:
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date

In [4]:
sql = "select * from koa_nirspec where koaid like '%20170605%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
        koaid          propint       date_obs      ...     z      spt_ind 
                        months                     ...                    
---------------------- ------- ------------------- ... --------- ---------
NC.20170605.45884.fits      18 2017-06-05 00:00:00 ... -0.405123 130120032
NC.20170605.45978.fits      18 2017-06-05 00:00:00 ... -0.405275 130120032
NC.20170605.46085.fits      18 2017-06-05 00:00:00 ... -0.405275 130120032
NC.20170605.46186.fits      18 2017-06-05 00:00:00 ... -0.405275 130120032
NC.20170605.46226.fits      18 2017-06-05 00:00:00 ... -0.405275 130120032
NS.20170605.44190.fits      18 2017-06-05 00:00:00 ... -0.499979 133300131
NS.20170605.44221.fits      18 2017-06-05 00:00:00 ... -0.500021 133300131
NS.20170605.44231.fits      18 2017-06-05 00:00:00 ... -0.500021 133300131
NS.20170605.44242.fits      18 2017-06-05 00:00:00 ...      -0.5 133300131
NS.20170605.44252.fits      18 2017-06-05 00:00:00 ...      -0.5 133300131
                   ...     ...                 ... ...       ...       ...
NS.20170605.55455.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55467.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55480.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55491.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55602.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55628.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55645.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55754.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55794.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
NS.20170605.55811.fits      18 2017-06-05 00:00:00 ...   0.00103 230210000
Length = 137 rows

Select columns acquired on UT Date

In [5]:
sql = "select koaid, filehand from koa_nirspec \
    where koaid like '%20170605%' "

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='columns.tbl', \
                     output_format='ipac')
        koaid          ...
---------------------- ...
NC.20170605.45884.fits ...
NC.20170605.45978.fits ...
NC.20170605.46085.fits ...
NC.20170605.46186.fits ...
NC.20170605.46226.fits ...
NS.20170605.44190.fits ...
NS.20170605.44221.fits ...
NS.20170605.44231.fits ...
NS.20170605.44242.fits ...
NS.20170605.44252.fits ...
                   ... ...
NS.20170605.55442.fits ...
NS.20170605.55455.fits ...
NS.20170605.55467.fits ...
NS.20170605.55480.fits ...
NS.20170605.55491.fits ...
NS.20170605.55602.fits ...
NS.20170605.55628.fits ...
NS.20170605.55645.fits ...
NS.20170605.55754.fits ...
NS.20170605.55794.fits ...
NS.20170605.55811.fits ...
Length = 137 rows

Select a maximum number of records

In [6]:
sql = "select top 20 koaid, filehand, frameno from koa_nirspec where koaid like '%20170605%'"
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecors.tbl', output_format='ipac')

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='top20.tbl', \
                     output_format='ipac')
        koaid          ... frameno
---------------------- ... -------
NS.20170605.44361.fits ...      15
NS.20170605.44598.fits ...      16
NS.20170605.44632.fits ...      17
NS.20170605.44642.fits ...      18
NS.20170605.44654.fits ...      19
NS.20170605.44665.fits ...      20
NS.20170605.44677.fits ...      21
NS.20170605.47903.fits ...      48
NS.20170605.48590.fits ...      54
NS.20170605.48636.fits ...      55
NS.20170605.48695.fits ...      56
NS.20170605.48954.fits ...      57
NS.20170605.54832.fits ...      96
NS.20170605.55148.fits ...     105
NS.20170605.55160.fits ...     106
NS.20170605.55172.fits ...     107
NS.20170605.55186.fits ...     108
NS.20170605.55201.fits ...     109
NS.20170605.55215.fits ...     110
NS.20170605.55288.fits ...     112

Select columns and search by datetime range

In [7]:
sql="select koaid, filehand from koa_nirspec \
     where (utdatetime >= to_date('2017-06-05 13:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2017-06-05 14:00:00', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='datetime_range.tbl', \
                     output_format='ipac')
        koaid          ...
---------------------- ...
NS.20170605.46805.fits ...
NS.20170605.46982.fits ...
NS.20170605.47079.fits ...
NS.20170605.47170.fits ...
NS.20170605.47245.fits ...
NS.20170605.47336.fits ...
NS.20170605.47719.fits ...
NS.20170605.47785.fits ...
NS.20170605.47837.fits ...
NS.20170605.47903.fits ...
                   ... ...
NS.20170605.48954.fits ...
NS.20170605.49047.fits ...
NS.20170605.49125.fits ...
NS.20170605.49219.fits ...
NS.20170605.49457.fits ...
NS.20170605.49551.fits ...
NS.20170605.49634.fits ...
NS.20170605.49728.fits ...
NS.20170605.50038.fits ...
NS.20170605.50188.fits ...
NS.20170605.50325.fits ...
Length = 29 rows

Search by program ID

In [9]:
sql= ("select koaid, filehand, progid  \
    from koa_nirspec where (progid = 'K236') ")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='progID.tbl', \
                     output_format='ipac')
        koaid          ... progid
---------------------- ... ------
NC.20170605.45884.fits ...   K236
NC.20170605.45978.fits ...   K236
NC.20170605.46085.fits ...   K236
NC.20170605.46186.fits ...   K236
NC.20170605.46226.fits ...   K236
NC.20170608.45997.fits ...   K236
NC.20170608.46091.fits ...   K236
NC.20170608.46147.fits ...   K236
NC.20170608.46243.fits ...   K236
NC.20170608.46278.fits ...   K236
                   ... ...    ...
NS.20170608.56034.fits ...   K236
NS.20170608.56048.fits ...   K236
NS.20170608.56060.fits ...   K236
NS.20170608.56072.fits ...   K236
NS.20170608.56086.fits ...   K236
NS.20170608.56157.fits ...   K236
NS.20170608.56189.fits ...   K236
NS.20170608.56208.fits ...   K236
NS.20170608.56281.fits ...   K236
NS.20170608.56309.fits ...   K236
NS.20170608.56326.fits ...   K236
Length = 282 rows
In [10]:
sql=("select koaid, filehand,ra, dec from koa_nirspec where \
    contains(point('icrs', ra, dec), \
    circle('icrs', 277.93 -9.985, 0.5)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
NC.20080621.43529.fits ...  -9.9875
NC.20080621.43543.fits ... -9.98611
NS.20080621.43867.fits ... -9.98775
NS.20080621.43983.fits ... -9.98755
NS.20080621.44085.fits ...  -9.9876
NS.20080621.44204.fits ... -9.98777
NS.20080621.44432.fits ... -9.98772
NS.20080621.44545.fits ... -9.98759
NS.20080621.44648.fits ... -9.98764
NS.20080621.44762.fits ... -9.98775
                   ... ...      ...
NS.20130726.38358.fits ... -9.98788
NS.20130726.38560.fits ... -9.98795
NS.20130726.38713.fits ...   -9.988
NS.20150808.27197.fits ... -9.98865
NS.20150808.27314.fits ... -9.98605
NS.20150808.27418.fits ...   -9.986
NS.20150808.27536.fits ... -9.98848
NS.20170605.50038.fits ... -9.98569
NS.20170605.50188.fits ... -9.98571
NS.20170605.50325.fits ... -9.98563
NS.20170605.50479.fits ...  -9.9856
Length = 28 rows
In [17]:
sql=("select koaid, filehand, ra, dec from koa_nirspec where \
     contains(point('J2000', ra, dec), \
     box('J2000', 277.93 -9.985, 0.5, 0.5))=1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
NC.20080621.43529.fits ...  -9.9875
NC.20080621.43543.fits ... -9.98611
NS.20080621.43867.fits ... -9.98775
NS.20080621.43983.fits ... -9.98755
NS.20080621.44085.fits ...  -9.9876
NS.20080621.44204.fits ... -9.98777
NS.20080621.44432.fits ... -9.98772
NS.20080621.44545.fits ... -9.98759
NS.20080621.44648.fits ... -9.98764
NS.20080621.44762.fits ... -9.98775
                   ... ...      ...
NS.20130726.38358.fits ... -9.98788
NS.20130726.38560.fits ... -9.98795
NS.20130726.38713.fits ...   -9.988
NS.20150808.27197.fits ... -9.98865
NS.20150808.27314.fits ... -9.98605
NS.20150808.27418.fits ...   -9.986
NS.20150808.27536.fits ... -9.98848
NS.20170605.50038.fits ... -9.98569
NS.20170605.50188.fits ... -9.98571
NS.20170605.50325.fits ... -9.98563
NS.20170605.50479.fits ...  -9.9856
Length = 28 rows

TAP+ returns the first 2,000 records in this query

In [18]:
sql=("select koaid, filehand, ra, dec from koa_nirspec  where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',277.43 -10.485 277.43 -9.485 278.43 -9.485 278.43 -10.485)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
NC.20080621.43529.fits ...  -9.9875
NC.20080621.43543.fits ... -9.98611
NS.20080621.43867.fits ... -9.98775
NS.20080621.43983.fits ... -9.98755
NS.20080621.44085.fits ...  -9.9876
NS.20080621.44204.fits ... -9.98777
NS.20080621.44432.fits ... -9.98772
NS.20080621.44545.fits ... -9.98759
NS.20080621.44648.fits ... -9.98764
NS.20080621.44762.fits ... -9.98775
                   ... ...      ...
NS.20130726.38358.fits ... -9.98788
NS.20130726.38560.fits ... -9.98795
NS.20130726.38713.fits ...   -9.988
NS.20150808.27197.fits ... -9.98865
NS.20150808.27314.fits ... -9.98605
NS.20150808.27418.fits ...   -9.986
NS.20150808.27536.fits ... -9.98848
NS.20170605.50038.fits ... -9.98569
NS.20170605.50188.fits ... -9.98571
NS.20170605.50325.fits ... -9.98563
NS.20170605.50479.fits ...  -9.9856
Length = 28 rows

Count number of records on date

In [19]:
sql=("select count(*) from koa_nirspec where \
    date_obs=to_date('2017-06-05', 'YYYY-MM-DD')")

job = koa.launch_job(sql)
r = job.get_results()
print(r)
count(*)
--------
     137

Count total number of records in table

In [20]:
sql="select count(*) as total from koa_nirspec"

job = koa.launch_job(sql)
r = job.get_results()
print(r)
total
-----
 2000

Note: There are 800,000+ records that satisfy this query, but TAP+ only returns the first 2,000 in synchronous mode


Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA