Using the nexsciTAP Server with the TAP+ Client to Access Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibation data acquired with the HIRES instrument; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (March, 2021)

Set up

In [1]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [2]:
sql = "select koaid, filehand from koa_hires where koaid like '%20040824%' "
job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid                                filehand                      
---------------------- ----------------------------------------------------
HI.20040824.08204.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08204.fits
HI.20040824.08544.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08544.fits
HI.20040824.08591.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08591.fits
HI.20040824.08638.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08638.fits
HI.20040824.08685.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08685.fits
HI.20040824.08732.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08732.fits
HI.20040824.08779.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08779.fits
HI.20040824.08826.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08826.fits
HI.20040824.08873.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08873.fits
HI.20040824.08919.fits /koadata1/HIRES/20040824/lev0/HI.20040824.08919.fits
                   ...                                                  ...
HI.20040824.54756.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54756.fits
HI.20040824.54803.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54803.fits
HI.20040824.54849.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54849.fits
HI.20040824.54896.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54896.fits
HI.20040824.54944.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54944.fits
HI.20040824.54990.fits /koadata1/HIRES/20040824/lev0/HI.20040824.54990.fits
HI.20040824.55037.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55037.fits
HI.20040824.55084.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55084.fits
HI.20040824.55131.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55131.fits
HI.20040824.55177.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55177.fits
HI.20040824.55224.fits /koadata1/HIRES/20040824/lev0/HI.20040824.55224.fits
Length = 105 rows

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [3]:
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date 20040824

In [4]:
sql = "select * from koa_hires where koaid like '%20040824%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
 el  elaptime equinox eramode ...      z        spt_ind      ofname    
---- -------- ------- ------- ... ------------ --------- --------------
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2300.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2301.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2302.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2303.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2304.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2305.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2306.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2307.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2308.fits
45.0        1     0.0 B, G, R ... 0.7068576891 203233131 hires2309.fits
 ...      ...     ...     ... ...          ...       ...            ...
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2394.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2395.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2396.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2397.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2398.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2399.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2400.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2401.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2402.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2403.fits
0.01        1  2000.0 B, G, R ... 0.0001813223 200021201 hires2404.fits
Length = 105 rows

Select columns acquired on UT Date 20040824 and order by signal-to-noise (sig2nois)

In [5]:
sql = "select koaid, filehand, sig2nois from koa_hires \
    where koaid like '%20040824%'order by sig2nois"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='orderbysig2nois.tbl', \
                     output_format='ipac')
        koaid          ... sig2nois
---------------------- ... --------
HI.20040824.09677.fits ...      2.0
HI.20040824.09724.fits ...      2.0
HI.20040824.54001.fits ...      2.0
HI.20040824.53947.fits ...      2.0
HI.20040824.45942.fits ...      2.0
HI.20040824.44695.fits ...      2.0
HI.20040824.44395.fits ...      2.0
HI.20040824.44343.fits ...      2.0
HI.20040824.43054.fits ...      2.0
HI.20040824.41806.fits ...      2.0
                   ... ...      ...
HI.20040824.09058.fits ...     11.0
HI.20040824.09011.fits ...     11.0
HI.20040824.08965.fits ...     11.0
HI.20040824.08873.fits ...     11.0
HI.20040824.09105.fits ...     11.0
HI.20040824.08779.fits ...     11.0
HI.20040824.08732.fits ...     11.0
HI.20040824.08826.fits ...     11.0
HI.20040824.08591.fits ...     11.0
HI.20040824.08204.fits ...     11.0
HI.20040824.08685.fits ...     15.0
Length = 105 rows

Select a maximum number of records

In [6]:
sql = "select top 20 koaid, filehand, frameno from koa_hires \
       where koaid like '%20040824%'"
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecors.tbl', output_format='ipac')

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='top20.tbl', \
                     output_format='ipac')
        koaid          ... frameno
---------------------- ... -------
HI.20040824.08732.fits ...    2305
HI.20040824.08965.fits ...    2310
HI.20040824.09910.fits ...    2321
HI.20040824.10096.fits ...    2325
HI.20040824.10189.fits ...    2327
HI.20040824.16108.fits ...    2334
HI.20040824.16433.fits ...    2339
HI.20040824.16480.fits ...    2340
HI.20040824.16526.fits ...    2341
HI.20040824.17057.fits ...    2350
HI.20040824.18573.fits ...    2352
HI.20040824.29673.fits ...    2362
HI.20040824.30924.fits ...    2363
HI.20040824.36514.fits ...    2369
HI.20040824.44695.fits ...    2379
HI.20040824.54207.fits ...    2385
HI.20040824.54301.fits ...    2387
HI.20040824.54347.fits ...    2388
HI.20040824.54488.fits ...    2391
HI.20040824.54896.fits ...    2397

Select columns and search by date range

In [7]:
sql="select koaid, filehand from koa_hires \
     where (utdatetime >= to_date('2009-01-01 00:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2009-01-05 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='date_range.tbl', \
                     output_format='ipac')
        koaid                                filehand                      
---------------------- ----------------------------------------------------
HI.20090101.09133.fits /koadata4/HIRES/20090101/lev0/HI.20090101.09133.fits
HI.20090101.09658.fits /koadata4/HIRES/20090101/lev0/HI.20090101.09658.fits
HI.20090101.09784.fits /koadata4/HIRES/20090101/lev0/HI.20090101.09784.fits
HI.20090104.02004.fits /koadata4/HIRES/20090104/lev0/HI.20090104.02004.fits
HI.20090104.02222.fits /koadata4/HIRES/20090104/lev0/HI.20090104.02222.fits
HI.20090104.02477.fits /koadata4/HIRES/20090104/lev0/HI.20090104.02477.fits
HI.20090105.04559.fits /koadata4/HIRES/20090105/lev0/HI.20090105.04559.fits
HI.20090105.05148.fits /koadata4/HIRES/20090105/lev0/HI.20090105.05148.fits
HI.20090105.06188.fits /koadata4/HIRES/20090105/lev0/HI.20090105.06188.fits
HI.20090105.08445.fits /koadata4/HIRES/20090105/lev0/HI.20090105.08445.fits
                   ...                                                  ...
HI.20090105.10541.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10541.fits
HI.20090105.10587.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10587.fits
HI.20090105.10632.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10632.fits
HI.20090105.10678.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10678.fits
HI.20090105.10724.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10724.fits
HI.20090105.10770.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10770.fits
HI.20090105.10816.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10816.fits
HI.20090105.10862.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10862.fits
HI.20090105.10908.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10908.fits
HI.20090105.10954.fits /koadata4/HIRES/20090105/lev0/HI.20090105.10954.fits
HI.20090105.11000.fits /koadata4/HIRES/20090105/lev0/HI.20090105.11000.fits
Length = 61 rows

Search by program ID

In [9]:
sql="select koaid, filehand, progid, iodin, koaimtyp from \
      koa_hires where progid = 'C307'"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='progID.txt', \
                     output_format='ipac')
        koaid          ... koaimtyp
---------------------- ... --------
HI.20180331.02525.fits ...    focus
HI.20180331.03342.fits ...    focus
HI.20180331.03598.fits ...    focus
HI.20180331.03786.fits ...    focus
HI.20180331.03943.fits ...    focus
HI.20180331.04060.fits ...    focus
HI.20180331.04218.fits ...    focus
HI.20180331.04338.fits ...    focus
HI.20180331.04502.fits ...  arclamp
HI.20180331.04578.fits ...  arclamp
                   ... ...      ...
HI.20180801.44904.fits ...   object
HI.20180801.45058.fits ...   object
HI.20180801.45222.fits ...   object
HI.20180801.45356.fits ...   object
HI.20180801.45644.fits ...   object
HI.20180801.45867.fits ...   object
HI.20180801.45945.fits ...   object
HI.20180801.46157.fits ... flatlamp
HI.20180801.46246.fits ... flatlamp
HI.20180801.46399.fits ...  arclamp
HI.20180801.46463.fits ...  arclamp
Length = 538 rows
In [10]:
sql=("select koaid, filehand, ra, dec from koa_hires where \
    contains(point('icrs', ra, dec), \
    circle('icrs',262.0,17.0,1.0)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
HI.19980718.34975.fits ... 16.82711
HI.19980912.26279.fits ... 16.82489
HI.19980913.27523.fits ... 16.82269
HI.19980914.23928.fits ... 16.82186
HI.19980914.24204.fits ... 16.82186
HI.19980914.29108.fits ... 16.82222
HI.19980915.26653.fits ...  16.8225
HI.19980916.25837.fits ... 16.82369
HI.19980918.24196.fits ... 16.82519
HI.19990421.49245.fits ...  17.5081
                   ... ...      ...
HI.20040824.18573.fits ... 17.50994
HI.20040824.18925.fits ... 17.50994
HI.20040825.18800.fits ... 17.51019
HI.20041002.18948.fits ... 17.50994
HI.20041002.19291.fits ... 17.50994
HI.20041002.19935.fits ... 17.50994
HI.20041002.20579.fits ... 17.50994
HI.20050301.49856.fits ... 16.82503
HI.20160126.56113.fits ...  16.8245
HI.20160714.38961.fits ... 16.82319
HI.20180702.37596.fits ... 16.82131
Length = 42 rows
In [11]:
sql=("select koaid, filehand, ra, dec from koa_hires where \
     contains(point('icrs', ra, dec), \
     box('icrs',262.0,17.0,2.0,120)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
        koaid          ...    dec   
---------------------- ... ---------
HI.19941214.78458.fits ...  29.92281
HI.19950716.29399.fits ... -37.29384
HI.19950716.29658.fits ... -37.29384
HI.19950716.29859.fits ... -37.29384
HI.19950716.30097.fits ... -37.29384
HI.19951129.15228.fits ...      25.0
HI.19951129.15327.fits ...      25.0
HI.19960523.16106.fits ...   5.33211
HI.19960523.16363.fits ...   5.33211
HI.19960523.16460.fits ...   5.33211
                   ... ...       ...
HI.20190614.36586.fits ...  34.27092
HI.20190614.36658.fits ...  34.27092
HI.20190616.31037.fits ...  67.30686
HI.20190715.29454.fits ...  60.05008
HI.20190715.29515.fits ...  60.05008
HI.20190715.29575.fits ...  60.05008
HI.20190715.29654.fits ...  67.30667
HI.20190715.29719.fits ...  67.30667
HI.20190715.29786.fits ...  67.30667
HI.20190715.29853.fits ...  67.30667
HI.20190715.29924.fits ...  67.30667
Length = 1031 rows
In [12]:
sql=("select koaid, filehand, ra, dec from koa_hires where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
HI.20040114.51275.fits ... 54.35381
HI.20040114.53552.fits ... 54.35381
HI.20040114.55829.fits ... 54.35381
HI.20040114.58182.fits ... 54.35383
HI.20060420.33559.fits ... 54.35292
HI.20060420.35412.fits ... 54.35292
HI.20060420.37265.fits ... 54.35292
HI.20060420.39117.fits ... 54.35292
HI.20060420.41232.fits ... 54.31703
HI.20060420.43085.fits ... 54.31703
                   ... ...      ...
HI.20110903.19681.fits ... 54.27442
HI.20110905.19585.fits ... 54.27675
HI.20110909.17581.fits ... 54.27289
HI.20110909.18064.fits ... 54.27289
HI.20110909.18415.fits ... 54.27289
HI.20110909.19067.fits ... 54.27289
HI.20110909.19719.fits ... 54.27289
HI.20110920.17641.fits ... 54.27347
HI.20110923.18047.fits ... 54.27414
HI.20110923.18126.fits ... 54.27414
HI.20110923.18478.fits ... 54.27414
Length = 28 rows

Count number of records in date range

In [13]:
sql=("select count(*) from koa_hires where \
    (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2009-01-05 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"  )

job = koa.launch_job(sql)
r = job.get_results()
print(r)
count(*)
--------
      61

Count total number of records in table

In [14]:
sql="select count(*) as total from koa_hires"

job = koa.launch_job(sql)
r = job.get_results()
print(r)
total
-----
 2000

Note: There are >400,000 records that satisfy this query, but TAP+ only returns the first 2,000


Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA