Using the nexsciTAP Server with the TAP+ Client to Access Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the KCWI integral field spectrograph; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (May, 2021)

Set up

In [1]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [2]:
sql = "select koaid, filehand from koa_kcwi where koaid like '%20190509%' "
job = koa.launch_job_async(sql)
r = job.get_results()
print(r)
INFO: Query finished. [astroquery.utils.tap.core]
        koaid                                filehand                      
---------------------- ----------------------------------------------------
KB.20190509.07554.fits /koadata28/KCWI/20190509/lev0/KB.20190509.07554.fits
KB.20190509.08692.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08692.fits
KB.20190509.08718.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08718.fits
KB.20190509.08743.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08743.fits
KB.20190509.08770.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08770.fits
KB.20190509.08795.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08795.fits
KB.20190509.08821.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08821.fits
KB.20190509.08846.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08846.fits
KB.20190509.08872.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08872.fits
KB.20190509.08897.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08897.fits
                   ...                                                  ...
KB.20190509.49694.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49694.fits
KB.20190509.49752.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49752.fits
KB.20190509.49811.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49811.fits
KB.20190509.49876.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49876.fits
KB.20190509.49934.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49934.fits
KF.20190509.07636.fits /koadata28/KCWI/20190509/lev0/KF.20190509.07636.fits
KF.20190509.18675.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18675.fits
KF.20190509.18849.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18849.fits
KF.20190509.18929.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18929.fits
KF.20190509.25327.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25327.fits
KF.20190509.25416.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25416.fits
Length = 96 rows

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [3]:
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date

In [4]:
sql = "select * from koa_kcwi where koaid like '%20190509%'"
job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          propint filesize_mb ...         z          spt_ind 
---------------------- ------- ----------- ... ----------------- ---------
KB.20190509.07554.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08692.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08718.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08743.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08770.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08795.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08821.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08846.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08872.fits      18          -- ... 0.707106781186547 233211302
KB.20190509.08897.fits      18          -- ... 0.707106781186547 233211302
                   ...     ...         ... ...               ...       ...
KB.20190509.49694.fits      18          -- ... 0.900349381924348 211211221
KB.20190509.49752.fits      18          -- ... 0.900349381924348 211211221
KB.20190509.49811.fits      18          -- ... 0.900349381924348 211211221
KB.20190509.49876.fits      18          -- ... 0.900349381924348 211211221
KB.20190509.49934.fits      18          -- ... 0.900349381924348 211211221
KF.20190509.07636.fits      18          -- ... 0.707106781186547 233211302
KF.20190509.18675.fits      18          -- ... 0.685910818175545 223233033
KF.20190509.18849.fits      18          -- ... 0.685910818175545 223233033
KF.20190509.18929.fits      18          -- ... 0.685910818175545 223233033
KF.20190509.25327.fits      18          -- ... 0.272960330782114 220322212
KF.20190509.25416.fits      18          -- ... 0.272960330782114 220322212
Length = 96 rows

Select columns acquired on UT Date

In [5]:
sql = "select koaid, filehand from koa_kcwi \
    where koaid like '%20190509%' "

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='columns.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid                                filehand                      
---------------------- ----------------------------------------------------
KB.20190509.07554.fits /koadata28/KCWI/20190509/lev0/KB.20190509.07554.fits
KB.20190509.08692.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08692.fits
KB.20190509.08718.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08718.fits
KB.20190509.08743.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08743.fits
KB.20190509.08770.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08770.fits
KB.20190509.08795.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08795.fits
KB.20190509.08821.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08821.fits
KB.20190509.08846.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08846.fits
KB.20190509.08872.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08872.fits
KB.20190509.08897.fits /koadata28/KCWI/20190509/lev0/KB.20190509.08897.fits
                   ...                                                  ...
KB.20190509.49694.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49694.fits
KB.20190509.49752.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49752.fits
KB.20190509.49811.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49811.fits
KB.20190509.49876.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49876.fits
KB.20190509.49934.fits /koadata28/KCWI/20190509/lev0/KB.20190509.49934.fits
KF.20190509.07636.fits /koadata28/KCWI/20190509/lev0/KF.20190509.07636.fits
KF.20190509.18675.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18675.fits
KF.20190509.18849.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18849.fits
KF.20190509.18929.fits /koadata28/KCWI/20190509/lev0/KF.20190509.18929.fits
KF.20190509.25327.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25327.fits
KF.20190509.25416.fits /koadata28/KCWI/20190509/lev0/KF.20190509.25416.fits
Length = 96 rows

Select a maximum number of records

In [6]:
sql = "select top 20 koaid, filehand, frameno from koa_kcwi where koaid like '%20190509%'"
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecors.tbl', output_format='ipac')

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='top20.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ... frameno
---------------------- ... -------
KB.20190509.11305.fits ...      35
KB.20190509.11364.fits ...      36
KB.20190509.11422.fits ...      37
KB.20190509.11481.fits ...      38
KB.20190509.11652.fits ...      39
KB.20190509.11724.fits ...      40
KB.20190509.11796.fits ...      41
KB.20190509.12317.fits ...      42
KB.20190509.12392.fits ...      43
KB.20190509.12460.fits ...      44
KB.20190509.12493.fits ...      45
KB.20190509.12555.fits ...      46
KB.20190509.12617.fits ...      47
KB.20190509.12680.fits ...      48
KB.20190509.12742.fits ...      49
KB.20190509.12804.fits ...      50
KB.20190509.13110.fits ...      51
KB.20190509.13318.fits ...      52
KB.20190509.13526.fits ...      53
KB.20190509.19959.fits ...      57

Select columns and search by datetime range

In [7]:
sql="select koaid, filehand from koa_kcwi \
     where (utdatetime >= to_date('2019-05-09 03:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2019-05-09 03:30:00', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='datetime_range.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid                                filehand                      
---------------------- ----------------------------------------------------
KB.20190509.11052.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11052.fits
KB.20190509.11110.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11110.fits
KB.20190509.11158.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11158.fits
KB.20190509.11187.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11187.fits
KB.20190509.11246.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11246.fits
KB.20190509.11305.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11305.fits
KB.20190509.11364.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11364.fits
KB.20190509.11422.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11422.fits
KB.20190509.11481.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11481.fits
KB.20190509.11652.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11652.fits
KB.20190509.11724.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11724.fits
KB.20190509.11796.fits /koadata28/KCWI/20190509/lev0/KB.20190509.11796.fits
KB.20190509.12317.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12317.fits
KB.20190509.12392.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12392.fits
KB.20190509.12460.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12460.fits
KB.20190509.12493.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12493.fits
KB.20190509.12555.fits /koadata28/KCWI/20190509/lev0/KB.20190509.12555.fits

Search by program ID

In [8]:
sql= ("select koaid, filehand, progid, ha, az \
    from koa_kcwi where (progid = 'U216') ")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='progID.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...     az   
---------------------- ... ---------
KB.20191029.02565.fits ...      58.0
KB.20191029.02592.fits ...      58.0
KB.20191029.02621.fits ...      58.0
KB.20191029.02648.fits ...      58.0
KB.20191029.02676.fits ...      58.0
KB.20191029.02704.fits ...      58.0
KB.20191029.02732.fits ...      58.0
KB.20191029.02761.fits ...      58.0
KB.20191029.02789.fits ...      58.0
KB.20191029.04150.fits ...      58.0
                   ... ...       ...
KB.20191101.46716.fits ...    -43.35
KB.20191101.47975.fits ...    -46.56
KB.20191101.49236.fits ...    -48.94
KB.20191101.50496.fits ...    -50.66
KB.20191101.51753.fits ...    -51.85
KB.20191101.53021.fits ...    -52.59
KF.20191029.01516.fits ... 175.42999
KF.20191029.16281.fits ...     52.36
KF.20191029.16402.fits ...      52.3
KF.20191029.79371.fits ... 143.17999
KF.20191030.83125.fits ...    -18.27
Length = 357 rows
In [9]:
sql=("select koaid, filehand,ra, dec from koa_kcwi where \
    contains(point('icrs', ra, dec), \
    circle('icrs',23.48 ,30.6,1.0)) = 1")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...   dec   
---------------------- ... --------
KB.20191007.19853.fits ... 30.70153
KB.20191007.20637.fits ... 30.69856
KB.20191007.21303.fits ... 30.73003
KB.20191007.22362.fits ... 30.69642
KB.20191007.23112.fits ... 30.69642
KB.20191007.23772.fits ... 30.73003
KB.20191007.24613.fits ... 30.69108
KB.20191007.25504.fits ... 30.69117
KB.20191007.26159.fits ... 30.73003
KB.20191007.26883.fits ... 30.69333
                   ... ...      ...
KB.20191007.46353.fits ... 30.69339
KB.20191007.47014.fits ... 30.73294
KB.20191007.47783.fits ... 30.68761
KB.20191007.48477.fits ... 30.68761
KB.20191007.49134.fits ... 30.73294
KB.20191007.49836.fits ... 30.74714
KB.20191007.50501.fits ... 30.74267
KB.20191007.51157.fits ...   30.962
KB.20191007.51815.fits ... 30.74267
KB.20191007.52473.fits ... 30.74769
KB.20191007.53013.fits ... 30.74475
Length = 46 rows
In [10]:
sql=("select koaid, filehand, ra, dec from koa_kcwi where \
     contains(point('icrs', ra, dec), \
     box('icrs',23.48 ,30.60,1.0,120)) = 1")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...   dec   
---------------------- ... --------
KB.20170919.17047.fits ... 90.01667
KB.20170919.17126.fits ... 90.01667
KB.20170919.17211.fits ... 90.01667
KB.20170919.17300.fits ... 90.01667
KB.20170919.17411.fits ... 90.01667
KB.20170919.17639.fits ... 90.01667
KB.20170919.17865.fits ... 90.01667
KB.20170919.18233.fits ... 90.01667
KB.20170919.55567.fits ...     90.0
KB.20170919.55942.fits ...     90.0
                   ... ...      ...
KF.20190101.84250.fits ... 90.01719
KF.20190103.04308.fits ... 90.01719
KF.20190202.22921.fits ... 33.38131
KF.20190202.23172.fits ... 33.33467
KF.20190202.23233.fits ... 33.33467
KF.20190204.04009.fits ... 90.01719
KF.20190205.04782.fits ... 90.01719
KF.20190331.02925.fits ... 90.01719
KF.20190426.70623.fits ...     89.9
KF.20190427.74839.fits ... 90.01719
KF.20190628.04306.fits ...     90.0
Length = 910 rows

TAP+ returns the first 2,000 records in this query

In [11]:
sql=("select koaid, filehand, ra, dec from koa_kcwi  where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")


job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...   dec   
---------------------- ... --------
KB.20180210.38010.fits ... 54.24056
KB.20180210.38183.fits ... 54.24056
KB.20180210.38358.fits ... 54.24056
KB.20180210.38612.fits ... 54.24056
KB.20180210.39152.fits ... 54.24056
KB.20180210.39761.fits ... 54.24056
KB.20180210.40235.fits ... 54.24056
KB.20180210.40723.fits ... 54.24056
KB.20180210.41197.fits ... 54.24056
KB.20180210.41672.fits ... 54.24056
                   ... ...      ...
KB.20180210.52991.fits ... 54.39689
KB.20180210.53359.fits ... 54.39689
KB.20180210.53713.fits ... 54.39689
KB.20180210.54067.fits ... 54.39689
KB.20180210.54422.fits ... 54.39689
KB.20180210.54776.fits ... 54.39689
KB.20180210.55130.fits ... 54.39689
KB.20180210.55503.fits ... 54.39689
KB.20180210.55857.fits ... 54.39689
KB.20180210.56211.fits ... 54.39689
KB.20180210.56578.fits ... 54.39689
Length = 42 rows

Count number of records on date

In [12]:
sql=("select count(*) from koa_kcwi where \
    date_obs=to_date('2019-05-09', 'YYYY-MM-DD')")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)
INFO: Query finished. [astroquery.utils.tap.core]
count(*)
--------
      96

Count total number of records in table

In [13]:
sql="select count(*) as total from koa_kcwi"

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)
INFO: Query finished. [astroquery.utils.tap.core]
total
-----
19744

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA