Using the nexsciTAP Server with the TAP+ Client to Access Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the OSIRIS integral field spectrograph; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (March, 2021)

Set up

In [1]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [2]:
sql = "select koaid, filehand from koa_osiris where koaid like '%20150529%' "
job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid                                 filehand                       
---------------------- ------------------------------------------------------
OI.20150529.41901.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.41901.fits
OI.20150529.42096.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42096.fits
OI.20150529.42327.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42327.fits
OI.20150529.46047.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.46047.fits
OS.20150529.01240.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01240.fits
OS.20150529.01400.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01400.fits
OS.20150529.01554.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01554.fits
OS.20150529.01606.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01606.fits
OS.20150529.01737.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01737.fits
OS.20150529.01820.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01820.fits
                   ...                                                    ...
OS.20150529.54818.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.54818.fits
OS.20150529.54981.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.54981.fits
OS.20150529.55131.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55131.fits
OS.20150529.55209.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55209.fits
OS.20150529.55541.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55541.fits
OS.20150529.55592.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55592.fits
OS.20150529.55667.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55667.fits
OS.20150529.55735.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55735.fits
OS.20150529.83374.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83374.fits
OS.20150529.83590.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83590.fits
OS.20150529.83772.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83772.fits
Length = 123 rows

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [3]:
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date

In [4]:
sql = "select * from koa_osiris where koaid like '%20150529%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
        koaid          propint        ofname        ...     z      spt_ind 
---------------------- ------- -------------------- ... --------- ---------
OI.20150529.41901.fits      18 i150529_a024016.fits ...  0.026658 202000111
OI.20150529.42096.fits      18 i150529_a024017.fits ...  0.026658 202000111
OI.20150529.42327.fits      18 i150529_a024018.fits ...  0.026749 202000111
OI.20150529.46047.fits      18 i150529_a026019.fits ...  0.026722 202000111
OS.20150529.01240.fits      18 s150529_a001002.fits ... -0.001497 132000000
OS.20150529.01400.fits      18 s150529_a001003.fits ... -0.001497 132000000
OS.20150529.01554.fits      18 s150529_a001004.fits ... -0.001497 132000000
OS.20150529.01606.fits      18 s150529_a001005.fits ... -0.001497 132000000
OS.20150529.01737.fits      18 s150529_a001006.fits ... -0.001497 132000000
OS.20150529.01820.fits      18 s150529_a001007.fits ... -0.001497 132000000
                   ...     ...                  ... ...       ...       ...
OS.20150529.54818.fits      18 s150529_s032002.fits ...   0.07067 202001131
OS.20150529.54981.fits      18 s150529_s033001.fits ...  0.070675 202001131
OS.20150529.55131.fits      18 s150529_s034001.fits ...  0.070675 202001131
OS.20150529.55209.fits      18 s150529_s034002.fits ...  0.070665 202001131
OS.20150529.55541.fits      18 s150529_s035001.fits ... -0.102996 130033020
OS.20150529.55592.fits      18 s150529_s035002.fits ... -0.102986 130033020
OS.20150529.55667.fits      18 s150529_s036001.fits ... -0.103003 130033020
OS.20150529.55735.fits      18 s150529_s036002.fits ... -0.102989 130033020
OS.20150529.83374.fits      30 s150529_a001001.fits ...       0.0 200020010
OS.20150529.83590.fits      30 s150529_a001002.fits ...       0.0 200020010
OS.20150529.83772.fits      30 s150529_a001003.fits ...       0.0 200020010
Length = 123 rows

Select columns acquired on UT Date

In [5]:
sql = "select koaid, filehand from koa_osiris \
    where koaid like '%20150529%' "

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='columns.tbl', \
                     output_format='ipac')
        koaid                                 filehand                       
---------------------- ------------------------------------------------------
OI.20150529.41901.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.41901.fits
OI.20150529.42096.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42096.fits
OI.20150529.42327.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42327.fits
OI.20150529.46047.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.46047.fits
OS.20150529.01240.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01240.fits
OS.20150529.01400.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01400.fits
OS.20150529.01554.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01554.fits
OS.20150529.01606.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01606.fits
OS.20150529.01737.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01737.fits
OS.20150529.01820.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01820.fits
                   ...                                                    ...
OS.20150529.54818.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.54818.fits
OS.20150529.54981.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.54981.fits
OS.20150529.55131.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55131.fits
OS.20150529.55209.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55209.fits
OS.20150529.55541.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55541.fits
OS.20150529.55592.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55592.fits
OS.20150529.55667.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55667.fits
OS.20150529.55735.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.55735.fits
OS.20150529.83374.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83374.fits
OS.20150529.83590.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83590.fits
OS.20150529.83772.fits /koadata23/OSIRIS/20150530/lev0/OS.20150529.83772.fits
Length = 123 rows

Select a maximum number of records

In [6]:
sql = "select top 20 koaid, filehand, frameno from koa_osiris where koaid like '%20150529%'"
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecors.tbl', output_format='ipac')

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='top20.tbl', \
                     output_format='ipac')
        koaid          ... frameno
---------------------- ... -------
OI.20150529.41901.fits ...      16
OI.20150529.42096.fits ...      17
OI.20150529.42327.fits ...      18
OI.20150529.46047.fits ...      19
OS.20150529.01240.fits ...       2
OS.20150529.23625.fits ...       1
OS.20150529.23777.fits ...       1
OS.20150529.45616.fits ...       1
OS.20150529.45783.fits ...       2
OS.20150529.46315.fits ...       1
OS.20150529.46963.fits ...       2
OS.20150529.47668.fits ...       1
OS.20150529.48307.fits ...       2
OS.20150529.48950.fits ...       3
OS.20150529.49599.fits ...       4
OS.20150529.50306.fits ...       1
OS.20150529.50959.fits ...       2
OS.20150529.51607.fits ...       3
OS.20150529.52975.fits ...       2
OS.20150529.53621.fits ...       3

Select columns and search by date range

In [7]:
sql="select koaid, filehand from koa_osiris \
     where (utdatetime >= to_date('2015-05-29 00:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2015-05-31 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='date_range.tbl', \
                     output_format='ipac')
        koaid                                 filehand                       
---------------------- ------------------------------------------------------
OI.20150529.41901.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.41901.fits
OI.20150529.42096.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42096.fits
OI.20150529.42327.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.42327.fits
OI.20150529.46047.fits /koadata23/OSIRIS/20150529/lev0/OI.20150529.46047.fits
OS.20150529.01240.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01240.fits
OS.20150529.01400.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01400.fits
OS.20150529.01554.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01554.fits
OS.20150529.01606.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01606.fits
OS.20150529.01737.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01737.fits
OS.20150529.01820.fits /koadata23/OSIRIS/20150529/lev0/OS.20150529.01820.fits
                   ...                                                    ...
OS.20150531.59900.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.59900.fits
OS.20150531.59990.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.59990.fits
OS.20150531.60047.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60047.fits
OS.20150531.60105.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60105.fits
OS.20150531.60162.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60162.fits
OS.20150531.60220.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60220.fits
OS.20150531.60277.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60277.fits
OS.20150531.60325.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60325.fits
OS.20150531.60373.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60373.fits
OS.20150531.60422.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60422.fits
OS.20150531.60469.fits /koadata23/OSIRIS/20150531/lev0/OS.20150531.60469.fits
Length = 438 rows

Search by program ID

In [8]:
sql= ("select koaid, filehand, progid, ha, az \
    from koa_osiris where (progid = 'N021') ")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='progID.tbl', \
                     output_format='ipac')
        koaid          ...     az   
---------------------- ... ---------
OI.20190529.09103.fits ... 229.99966
OI.20190529.09810.fits ... 229.99967
OI.20190529.10083.fits ... 229.99967
OI.20190529.10304.fits ... 229.99967
OI.20190529.10373.fits ... 229.99967
OI.20190529.10484.fits ... 229.99967
OI.20190529.10526.fits ... 229.99967
OI.20190529.10568.fits ... 229.99967
OI.20190529.10609.fits ... 229.99968
OI.20190529.10650.fits ... 229.99968
                   ... ...       ...
OI.20190531.51540.fits ... 218.27779
OI.20190531.51610.fits ... 218.49028
OI.20190531.51700.fits ... 218.75805
OI.20190531.51772.fits ... 218.97391
OI.20190531.51842.fits ... 219.18109
OI.20190531.52132.fits ... 218.77133
OI.20190531.52204.fits ... 219.00335
OI.20190531.52276.fits ... 219.23572
OI.20190531.52361.fits ...  219.5109
OI.20190531.52434.fits ...  219.7435
OI.20190531.52514.fits ...  219.9956
Length = 549 rows
In [9]:
sql=("select koaid, filehand,ra, dec from koa_osiris where \
    contains(point('icrs', ra, dec), \
    circle('icrs',270.4 -27.2 1.0)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
        koaid          ...    dec   
---------------------- ... ---------
OI.20190421.48934.fits ...   -27.547
OI.20190421.49094.fits ...   -27.547
OI.20190421.49287.fits ...   -27.547
OI.20190421.49344.fits ...   -27.547
OI.20190421.49394.fits ...   -27.547
OI.20190421.49442.fits ...   -27.547
OI.20190421.49493.fits ...   -27.547
OI.20190421.49560.fits ... -27.54716
OI.20190421.49610.fits ... -27.54716
OI.20190421.49659.fits ... -27.54716
                   ... ...       ...
OI.20190513.43152.fits ... -27.54727
OI.20190513.43202.fits ... -27.54727
OI.20190513.43252.fits ... -27.54727
OI.20190513.43301.fits ... -27.54727
OI.20190513.43350.fits ... -27.54727
OS.20160813.24913.fits ... -26.52847
OS.20160813.24983.fits ... -26.52819
OS.20160813.25147.fits ... -26.52847
OS.20160813.25218.fits ... -26.52819
OS.20160813.25335.fits ... -26.52847
OS.20160813.25407.fits ... -26.52819
Length = 122 rows
In [10]:
sql=("select koaid, filehand, ra, dec from koa_osiris where \
     contains(point('icrs', ra, dec), \
     box('icrs',23.4, 30.60,1.0,120)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
        koaid          ...    dec   
---------------------- ... ---------
OI.20051012.12061.fits ...  89.96783
OI.20051012.12087.fits ...  89.96783
OI.20051012.12111.fits ...  89.96783
OI.20051012.12135.fits ...  89.96783
OI.20051012.12160.fits ...  89.96783
OI.20051012.12957.fits ...  89.96783
OI.20051012.13651.fits ...  89.96783
OI.20051012.13676.fits ...  89.96783
OI.20051012.13692.fits ...  89.96783
OI.20051012.13717.fits ...  89.96783
                   ... ...       ...
OS.20210114.24130.fits ... -13.08002
OS.20210114.24190.fits ... -13.08002
OS.20210114.24245.fits ... -13.08002
OS.20210114.24299.fits ... -13.08002
OS.20210114.24357.fits ... -13.08002
OS.20210114.24411.fits ... -13.08002
OS.20210114.24464.fits ... -13.08002
OS.20210114.24523.fits ... -13.08002
OS.20210114.24578.fits ... -13.08002
OS.20210114.24632.fits ... -13.08002
OS.20210114.24690.fits ... -13.08002
Length = 2000 rows

TAP+ returns the first 2,000 records in this query

In [11]:
sql=("select koaid, filehand, ra, dec from koa_osiris  where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
OS.20160416.49503.fits ... 54.41594
OS.20160416.49605.fits ... 54.41594
OS.20160416.49631.fits ... 54.41594
OS.20160416.49658.fits ... 54.41594
OS.20160416.49683.fits ... 54.41594
OS.20160416.49709.fits ... 54.41594
OS.20160416.49744.fits ... 54.41594
OS.20160416.49769.fits ... 54.41594
OS.20160416.49796.fits ... 54.41594
OS.20160416.49826.fits ... 54.41594
                   ... ...      ...
OS.20170509.43779.fits ... 54.23825
OS.20170509.43806.fits ... 54.23825
OS.20170509.43836.fits ... 54.23825
OS.20170509.43864.fits ... 54.23825
OS.20170509.43890.fits ... 54.23784
OS.20170509.43920.fits ... 54.23784
OS.20170509.43948.fits ... 54.23825
OS.20170509.43974.fits ... 54.23825
OS.20170509.44001.fits ... 54.23825
OS.20170509.44027.fits ... 54.23825
OS.20170509.44083.fits ... 54.23825
Length = 215 rows

Count number of records in date range

In [12]:
sql=("select count(*) from koa_osiris where \
    (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2009-01-31 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"  )

job = koa.launch_job(sql)
r = job.get_results()
print(r)
count(*)
--------
     306

Count total number of records in table

In [13]:
sql="select count(*) as total from koa_osiris"

job = koa.launch_job(sql)
r = job.get_results()
print(r)
total
-----
 2000

Note: There are 200,000+ records that satisfy this query, but TAP+ only returns the first 2,000 in synchronous mode


Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA