Using the nexsciTAP Server with the TAP+ Client to Access Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the Echellette Spectrograph and Imager (ESI); these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (March, 2021)

Set up

In [1]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [2]:
sql = "select koaid, filehand from koa_esi where koaid like '%20150902%' "
job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid                                filehand                     
---------------------- ---------------------------------------------------
ES.20150902.10478.fits /koadata36/ESI/20150902/lev0/ES.20150902.10478.fits
ES.20150902.10550.fits /koadata36/ESI/20150902/lev0/ES.20150902.10550.fits
ES.20150902.45958.fits /koadata36/ESI/20150902/lev0/ES.20150902.45958.fits
ES.20150902.46025.fits /koadata36/ESI/20150902/lev0/ES.20150902.46025.fits
ES.20150902.46101.fits /koadata36/ESI/20150902/lev0/ES.20150902.46101.fits
ES.20150902.46168.fits /koadata36/ESI/20150902/lev0/ES.20150902.46168.fits
ES.20150902.46410.fits /koadata36/ESI/20150902/lev0/ES.20150902.46410.fits
ES.20150902.85368.fits /koadata36/ESI/20150903/lev0/ES.20150902.85368.fits
ES.20150902.85427.fits /koadata36/ESI/20150903/lev0/ES.20150902.85427.fits
ES.20150902.85487.fits /koadata36/ESI/20150903/lev0/ES.20150902.85487.fits
ES.20150902.85546.fits /koadata36/ESI/20150903/lev0/ES.20150902.85546.fits
ES.20150902.85605.fits /koadata36/ESI/20150903/lev0/ES.20150902.85605.fits
ES.20150902.85664.fits /koadata36/ESI/20150903/lev0/ES.20150902.85664.fits

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [3]:
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date

In [4]:
sql = "select * from koa_esi where koaid like '%20150902%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
        koaid                ofname      filesize_mb ...    z      spt_ind 
---------------------- ----------------- ----------- ... -------- ---------
ES.20150902.10478.fits e150902_0001.fits          -- ... 0.707107 233211302
ES.20150902.10550.fits e150902_0002.fits          -- ... 0.707107 233211302
ES.20150902.45958.fits e150902_0003.fits          -- ... 0.707107 233211302
ES.20150902.46025.fits e150902_0004.fits          -- ... 0.707107 233211302
ES.20150902.46101.fits e150902_0005.fits          -- ... 0.707107 233211302
ES.20150902.46168.fits e150902_0006.fits          -- ... 0.707107 233211302
ES.20150902.46410.fits e150902_0007.fits          -- ... 0.707107 233211302
ES.20150902.85368.fits e150903_0001.fits          -- ... 0.710322 223311221
ES.20150902.85427.fits e150903_0002.fits          -- ... 0.710322 223311221
ES.20150902.85487.fits e150903_0003.fits          -- ... 0.710322 223311221
ES.20150902.85546.fits e150903_0004.fits          -- ... 0.710322 223311221
ES.20150902.85605.fits e150903_0005.fits          -- ... 0.710322 223311221
ES.20150902.85664.fits e150903_0006.fits          -- ... 0.710322 223311221

Select columns acquired on UT Date

In [5]:
sql = "select koaid, filehand from koa_esi \
    where koaid like '%20150902%' "

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='columns.tbl', \
                     output_format='ipac')
        koaid                                filehand                     
---------------------- ---------------------------------------------------
ES.20150902.10478.fits /koadata36/ESI/20150902/lev0/ES.20150902.10478.fits
ES.20150902.10550.fits /koadata36/ESI/20150902/lev0/ES.20150902.10550.fits
ES.20150902.45958.fits /koadata36/ESI/20150902/lev0/ES.20150902.45958.fits
ES.20150902.46025.fits /koadata36/ESI/20150902/lev0/ES.20150902.46025.fits
ES.20150902.46101.fits /koadata36/ESI/20150902/lev0/ES.20150902.46101.fits
ES.20150902.46168.fits /koadata36/ESI/20150902/lev0/ES.20150902.46168.fits
ES.20150902.46410.fits /koadata36/ESI/20150902/lev0/ES.20150902.46410.fits
ES.20150902.85368.fits /koadata36/ESI/20150903/lev0/ES.20150902.85368.fits
ES.20150902.85427.fits /koadata36/ESI/20150903/lev0/ES.20150902.85427.fits
ES.20150902.85487.fits /koadata36/ESI/20150903/lev0/ES.20150902.85487.fits
ES.20150902.85546.fits /koadata36/ESI/20150903/lev0/ES.20150902.85546.fits
ES.20150902.85605.fits /koadata36/ESI/20150903/lev0/ES.20150902.85605.fits
ES.20150902.85664.fits /koadata36/ESI/20150903/lev0/ES.20150902.85664.fits

Select a maximum number of records

In [6]:
sql = "select top 5 koaid, filehand, frameno from koa_esi where koaid like '%20150902%'"
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecors.tbl', output_format='ipac')

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='top5.tbl', \
                     output_format='ipac')
        koaid          ... frameno
---------------------- ... -------
ES.20150902.10478.fits ...       1
ES.20150902.10550.fits ...       2
ES.20150902.45958.fits ...       3
ES.20150902.46025.fits ...       4
ES.20150902.46101.fits ...       5

Select columns and search by date range

In [7]:
sql="select koaid, filehand from koa_esi \
     where (utdatetime >= to_date('2015-09-01 00:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2015-09-30 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='date_range.tbl', \
                     output_format='ipac')
        koaid                                filehand                     
---------------------- ---------------------------------------------------
ES.20150902.10478.fits /koadata36/ESI/20150902/lev0/ES.20150902.10478.fits
ES.20150902.10550.fits /koadata36/ESI/20150902/lev0/ES.20150902.10550.fits
ES.20150902.45958.fits /koadata36/ESI/20150902/lev0/ES.20150902.45958.fits
ES.20150902.46025.fits /koadata36/ESI/20150902/lev0/ES.20150902.46025.fits
ES.20150902.46101.fits /koadata36/ESI/20150902/lev0/ES.20150902.46101.fits
ES.20150902.46168.fits /koadata36/ESI/20150902/lev0/ES.20150902.46168.fits
ES.20150902.46410.fits /koadata36/ESI/20150902/lev0/ES.20150902.46410.fits
ES.20150902.85368.fits /koadata36/ESI/20150903/lev0/ES.20150902.85368.fits
ES.20150902.85427.fits /koadata36/ESI/20150903/lev0/ES.20150902.85427.fits
ES.20150902.85487.fits /koadata36/ESI/20150903/lev0/ES.20150902.85487.fits
                   ...                                                 ...
ES.20150911.24831.fits /koadata36/ESI/20150911/lev0/ES.20150911.24831.fits
ES.20150911.26693.fits /koadata36/ESI/20150911/lev0/ES.20150911.26693.fits
ES.20150911.35455.fits /koadata36/ESI/20150911/lev0/ES.20150911.35455.fits
ES.20150911.42470.fits /koadata36/ESI/20150911/lev0/ES.20150911.42470.fits
ES.20150911.44332.fits /koadata36/ESI/20150911/lev0/ES.20150911.44332.fits
ES.20150911.46194.fits /koadata36/ESI/20150911/lev0/ES.20150911.46194.fits
ES.20150911.48056.fits /koadata36/ESI/20150911/lev0/ES.20150911.48056.fits
ES.20150911.50078.fits /koadata36/ESI/20150911/lev0/ES.20150911.50078.fits
ES.20150911.51339.fits /koadata36/ESI/20150911/lev0/ES.20150911.51339.fits
ES.20150911.52604.fits /koadata36/ESI/20150911/lev0/ES.20150911.52604.fits
ES.20150911.53867.fits /koadata36/ESI/20150911/lev0/ES.20150911.53867.fits
Length = 225 rows

Search by program ID

In [8]:
sql= ("select koaid, filehand, progid, ha, az \
    from koa_esi where (progid = 'K197E') ")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='progID.tbl', \
                     output_format='ipac')
        koaid          ...      az     
---------------------- ... ------------
ES.20140724.08809.fits ...  57.99963545
ES.20140724.09553.fits ...  57.99963672
ES.20140724.10762.fits ...  57.99963736
ES.20140724.10811.fits ...    57.999638
ES.20140724.11148.fits ...    57.999638
ES.20140724.11286.fits ...  57.99963864
ES.20140724.11391.fits ...  57.99963864
ES.20140724.11496.fits ...  57.99963864
ES.20140724.11615.fits ...  57.99963928
ES.20140724.11721.fits ...  57.99963991
                   ... ...          ...
ES.20140724.33747.fits ... 200.97536896
ES.20140724.33833.fits ... 201.39384859
ES.20140724.33919.fits ... 201.81847091
ES.20140724.34019.fits ... 202.29891756
ES.20140724.34713.fits ... 205.58013977
ES.20140724.35396.fits ... 208.66363454
ES.20140724.36090.fits ... 211.63996849
ES.20140724.36465.fits ... 213.18497377
ES.20140724.36862.fits ... 214.76645206
ES.20140724.36948.fits ... 215.10402524
ES.20140724.37034.fits ... 215.44095486
Length = 105 rows
In [9]:
sql=("select koaid, filehand,ra, dec from koa_esi where \
    contains(point('icrs', ra, dec), \
    circle('icrs',255.03913 -11.82214 1.0)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
        koaid          ...    dec   
---------------------- ... ---------
ES.20140724.19862.fits ... -11.82214
ES.20140724.19954.fits ... -11.82214
ES.20140724.20187.fits ... -11.82214
ES.20140724.20277.fits ... -11.82214
ES.20140724.20549.fits ... -11.82214
ES.20140724.20649.fits ... -11.82214
ES.20140724.20925.fits ... -11.82214
ES.20140724.21025.fits ... -11.82214
In [10]:
sql=("select koaid, filehand, ra, dec from koa_esi where \
     contains(point('icrs', ra, dec), \
     box('icrs',23.48, 30.60,1.0,120)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
ES.19990831.33124.fits ... 89.72169
ES.19990831.33223.fits ... 89.72169
ES.19990831.33424.fits ... 89.72169
ES.19990831.33559.fits ... 89.72169
ES.19990831.33822.fits ... 89.72169
ES.19990831.34017.fits ... 89.72169
ES.19990831.34113.fits ... 89.72169
ES.19990831.34210.fits ... 89.72169
ES.19990831.34307.fits ... 89.72169
ES.19990831.35315.fits ... 89.72169
                   ... ...      ...
ES.20190506.77803.fits ...    90.01
ES.20190506.77915.fits ...    90.01
ES.20190506.78101.fits ...    90.01
ES.20190506.78161.fits ...    90.01
ES.20190506.78213.fits ...    90.01
ES.20190506.78268.fits ...    90.01
ES.20190506.78320.fits ...    90.01
ES.20190506.78374.fits ...    90.01
ES.20190506.78428.fits ...    90.01
ES.20190506.78482.fits ...    90.01
ES.20190506.78535.fits ...    90.01
Length = 1355 rows
In [11]:
sql=("select koaid, filehand, ra, dec from koa_esi  where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
ES.20120315.48168.fits ... 54.41872
ES.20120315.49127.fits ... 54.41872

Count number of records in date range

In [12]:
sql="select koaid, filehand from koa_esi \
   where (utdatetime >= to_date('2015-09-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') \
   and utdatetime <= to_date('2015-09-30 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"


job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid                                filehand                     
---------------------- ---------------------------------------------------
ES.20150902.10478.fits /koadata36/ESI/20150902/lev0/ES.20150902.10478.fits
ES.20150902.10550.fits /koadata36/ESI/20150902/lev0/ES.20150902.10550.fits
ES.20150902.45958.fits /koadata36/ESI/20150902/lev0/ES.20150902.45958.fits
ES.20150902.46025.fits /koadata36/ESI/20150902/lev0/ES.20150902.46025.fits
ES.20150902.46101.fits /koadata36/ESI/20150902/lev0/ES.20150902.46101.fits
ES.20150902.46168.fits /koadata36/ESI/20150902/lev0/ES.20150902.46168.fits
ES.20150902.46410.fits /koadata36/ESI/20150902/lev0/ES.20150902.46410.fits
ES.20150902.85368.fits /koadata36/ESI/20150903/lev0/ES.20150902.85368.fits
ES.20150902.85427.fits /koadata36/ESI/20150903/lev0/ES.20150902.85427.fits
ES.20150902.85487.fits /koadata36/ESI/20150903/lev0/ES.20150902.85487.fits
                   ...                                                 ...
ES.20150911.24831.fits /koadata36/ESI/20150911/lev0/ES.20150911.24831.fits
ES.20150911.26693.fits /koadata36/ESI/20150911/lev0/ES.20150911.26693.fits
ES.20150911.35455.fits /koadata36/ESI/20150911/lev0/ES.20150911.35455.fits
ES.20150911.42470.fits /koadata36/ESI/20150911/lev0/ES.20150911.42470.fits
ES.20150911.44332.fits /koadata36/ESI/20150911/lev0/ES.20150911.44332.fits
ES.20150911.46194.fits /koadata36/ESI/20150911/lev0/ES.20150911.46194.fits
ES.20150911.48056.fits /koadata36/ESI/20150911/lev0/ES.20150911.48056.fits
ES.20150911.50078.fits /koadata36/ESI/20150911/lev0/ES.20150911.50078.fits
ES.20150911.51339.fits /koadata36/ESI/20150911/lev0/ES.20150911.51339.fits
ES.20150911.52604.fits /koadata36/ESI/20150911/lev0/ES.20150911.52604.fits
ES.20150911.53867.fits /koadata36/ESI/20150911/lev0/ES.20150911.53867.fits
Length = 225 rows

Count total number of records in table

In [18]:
sql="select count(*) as total from koa_esi"

job = koa.launch_job(sql)
r = job.get_results()
print(r)
total
-----
 2000

Note: There are 70,000 records that satisfy this query, but TAP+ in synchronous mode only returns the first 2,000


Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA