Using the nexsciTAP Server with the TAP+ Client to Access Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the MOSFIRE near-infrared multi-object spectrograph; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (June, 2021)

Set up

In [20]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [6]:
sql = "select koaid, filehand from koa_mosfire where koaid like '%20121128%' "
job = koa.launch_job_async(sql)
r = job.get_results()
print(r)
INFO: Query finished. [astroquery.utils.tap.core]
        koaid                                  filehand                       
---------------------- -------------------------------------------------------
MF.20121128.07841.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07841.fits
MF.20121128.07866.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07866.fits
MF.20121128.07892.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07892.fits
MF.20121128.07917.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07917.fits
MF.20121128.07942.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07942.fits
MF.20121128.07966.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07966.fits
MF.20121128.07996.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07996.fits
MF.20121128.08031.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08031.fits
MF.20121128.08051.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08051.fits
MF.20121128.08077.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08077.fits
                   ...                                                     ...
MF.20121128.58884.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58884.fits
MF.20121128.58916.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58916.fits
MF.20121128.58982.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58982.fits
MF.20121128.59002.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59002.fits
MF.20121128.59026.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59026.fits
MF.20121128.59074.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59074.fits
MF.20121128.59104.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59104.fits
MF.20121128.59135.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59135.fits
MF.20121128.60043.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.60043.fits
MF.20121128.73782.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73782.fits
MF.20121128.73795.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73795.fits
Length = 556 rows

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [7]:
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date

In [8]:
sql = "select * from koa_mosfire where koaid like '%20121128%'"
job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid                ofname      coadds ...    z      spt_ind   cntr
---------------------- ----------------- ------ ... -------- --------- -----
MF.20121128.07841.fits m121128_0001.fits      1 ... 0.707107 233000300 20359
MF.20121128.07866.fits m121128_0002.fits      1 ... 0.707107 233000300 20360
MF.20121128.07892.fits m121128_0003.fits      1 ... 0.707107 233000300 20361
MF.20121128.07917.fits m121128_0004.fits      1 ... 0.707107 233000300 20362
MF.20121128.07942.fits m121128_0005.fits      1 ... 0.707107 233000300 20363
MF.20121128.07966.fits m121128_0006.fits      1 ... 0.707107 233000300 20364
MF.20121128.07996.fits m121128_0007.fits      1 ... 0.707107 233000300 20365
MF.20121128.08031.fits m121128_0008.fits      1 ... 0.707107 233000300 20366
MF.20121128.08051.fits m121128_0009.fits      1 ... 0.707107 233000300 20367
MF.20121128.08077.fits m121128_0010.fits      1 ... 0.707107 233000300 20368
                   ...               ...    ... ...      ...       ...   ...
MF.20121128.58884.fits m121128_0548.fits      1 ... 0.707107 233000300 20904
MF.20121128.58916.fits m121128_0549.fits      1 ... 0.707107 233000300 20905
MF.20121128.58982.fits m121128_0550.fits      1 ... 0.707107 233000300 20906
MF.20121128.59002.fits m121128_0551.fits      1 ... 0.707107 233000300 20907
MF.20121128.59026.fits m121128_0552.fits      1 ... 0.707107 233000300 20908
MF.20121128.59074.fits m121128_0553.fits      1 ... 0.707107 233000300 20909
MF.20121128.59104.fits m121128_0554.fits      1 ... 0.707107 233000300 20910
MF.20121128.59135.fits m121128_0555.fits      1 ... 0.707107 233000300 20911
MF.20121128.60043.fits m121128_0556.fits      1 ... 0.707107 233000300 20912
MF.20121128.73782.fits m121129_0001.fits      1 ... 0.707107 233000300 13514
MF.20121128.73795.fits m121129_0002.fits      1 ... 0.707107 233000300 13515
Length = 556 rows

Select columns acquired on UT Date

In [9]:
sql = "select koaid, filehand from koa_mosfire \
    where koaid like '%20121128%' "

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='columns.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid                                  filehand                       
---------------------- -------------------------------------------------------
MF.20121128.07841.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07841.fits
MF.20121128.07866.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07866.fits
MF.20121128.07892.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07892.fits
MF.20121128.07917.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07917.fits
MF.20121128.07942.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07942.fits
MF.20121128.07966.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07966.fits
MF.20121128.07996.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07996.fits
MF.20121128.08031.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08031.fits
MF.20121128.08051.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08051.fits
MF.20121128.08077.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08077.fits
                   ...                                                     ...
MF.20121128.58884.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58884.fits
MF.20121128.58916.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58916.fits
MF.20121128.58982.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58982.fits
MF.20121128.59002.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59002.fits
MF.20121128.59026.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59026.fits
MF.20121128.59074.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59074.fits
MF.20121128.59104.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59104.fits
MF.20121128.59135.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59135.fits
MF.20121128.60043.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.60043.fits
MF.20121128.73782.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73782.fits
MF.20121128.73795.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73795.fits
Length = 556 rows

Select a maximum number of records

In [10]:
sql = "select top 20 koaid, filehand, frameno from koa_mosfire where koaid like '%20121128%'"
job = koa.launch_job_async(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecords.tbl', output_format='ipac')

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='top20.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ... frameno
---------------------- ... -------
MF.20121128.12463.fits ...     112
MF.20121128.12485.fits ...     113
MF.20121128.12510.fits ...     114
MF.20121128.12535.fits ...     115
MF.20121128.12586.fits ...     117
MF.20121128.12613.fits ...     118
MF.20121128.12683.fits ...     119
MF.20121128.12701.fits ...     120
MF.20121128.12781.fits ...     121
MF.20121128.13125.fits ...     133
MF.20121128.13150.fits ...     134
MF.20121128.13193.fits ...     135
MF.20121128.13312.fits ...     139
MF.20121128.21629.fits ...     197
MF.20121128.21906.fits ...     198
MF.20121128.21999.fits ...     199
MF.20121128.22090.fits ...     200
MF.20121128.22184.fits ...     201
MF.20121128.73782.fits ...       1
MF.20121128.73795.fits ...       2

Select columns and search by datetime range

In [11]:
sql="select koaid, filehand from koa_mosfire \
     where (utdatetime >= to_date('2012-11-28 03:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2012-11-28 04:00:00', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='datetime_range.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid                                  filehand                       
---------------------- -------------------------------------------------------
MF.20121128.10814.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10814.fits
MF.20121128.10839.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10839.fits
MF.20121128.10865.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10865.fits
MF.20121128.10886.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10886.fits
MF.20121128.10921.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10921.fits
MF.20121128.10952.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10952.fits
MF.20121128.10974.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10974.fits
MF.20121128.10999.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.10999.fits
MF.20121128.11024.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.11024.fits
MF.20121128.11049.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.11049.fits
                   ...                                                     ...
MF.20121128.13341.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13341.fits
MF.20121128.13370.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13370.fits
MF.20121128.13801.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13801.fits
MF.20121128.13826.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13826.fits
MF.20121128.13851.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13851.fits
MF.20121128.13875.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13875.fits
MF.20121128.13905.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13905.fits
MF.20121128.13925.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13925.fits
MF.20121128.13951.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13951.fits
MF.20121128.13986.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.13986.fits
MF.20121128.14011.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.14011.fits
Length = 80 rows

Search by program ID

In [12]:
sql= ("select koaid, filehand, progid, ha, az \
    from koa_mosfire where (progid = 'N071') ")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='progID.tbl', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...      az     
---------------------- ... ------------
MF.20191012.37804.fits ...  48.10982738
MF.20191012.38102.fits ...  45.75980785
MF.20191012.38126.fits ...  45.55830003
MF.20191012.38184.fits ...  45.06581566
MF.20191012.38204.fits ...  44.88882713
MF.20191012.38390.fits ...  43.21210814
MF.20191012.38412.fits ...  43.00555516
MF.20191012.38470.fits ...  42.45414012
MF.20191012.38489.fits ...  42.26586424
MF.20191012.38837.fits ...  60.06885765
                   ... ...          ...
MF.20191014.56524.fits ... 229.99992868
MF.20191014.56538.fits ... 229.99992697
MF.20191014.56552.fits ... 229.99992795
MF.20191014.56566.fits ... 229.99992527
MF.20191014.56580.fits ... 229.99992917
MF.20191014.56594.fits ... 229.99992575
MF.20191014.56607.fits ...  229.9999282
MF.20191014.56620.fits ... 229.99992722
MF.20191014.56690.fits ...  229.9999282
MF.20191014.56701.fits ... 229.99992722
MF.20191014.56710.fits ...  229.9999282
Length = 474 rows
In [16]:
sql=("select koaid, filehand,ra, dec from koa_mosfire where \
    contains(point('icrs', ra, dec), \
    circle('icrs', 57.63 29.74, 0.5)) = 1")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...   dec   
---------------------- ... --------
MF.20121127.37568.fits ... 29.74463
MF.20121127.37610.fits ... 29.74483
MF.20121127.37795.fits ... 29.74524
MF.20121127.37820.fits ... 29.74441
MF.20121127.37899.fits ... 29.74524
MF.20121127.37926.fits ... 29.74441
MF.20121127.37946.fits ... 29.74441
MF.20121127.37969.fits ... 29.74524
MF.20121127.38012.fits ... 29.74524
MF.20121127.38034.fits ... 29.74441
                   ... ...      ...
MF.20191014.42587.fits ... 29.77373
MF.20191014.42603.fits ...  29.7819
MF.20191014.51892.fits ... 29.74492
MF.20191014.52137.fits ... 29.74492
MF.20191014.52197.fits ... 29.74479
MF.20191014.52333.fits ... 29.71124
MF.20191014.52356.fits ... 29.70292
MF.20191014.52417.fits ... 29.71124
MF.20191014.52440.fits ... 29.70292
MF.20191014.52464.fits ... 29.71124
MF.20191014.52487.fits ... 29.70292
Length = 530 rows
In [15]:
sql=("select koaid, filehand, ra, dec from koa_mosfire where \
     contains(point('icrs', ra, dec), \
     box('icrs', 57.63 29.74, 0.5, 0.5)) = 1")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...   dec   
---------------------- ... --------
MF.20121127.37568.fits ... 29.74463
MF.20121127.37610.fits ... 29.74483
MF.20121127.37795.fits ... 29.74524
MF.20121127.37820.fits ... 29.74441
MF.20121127.37899.fits ... 29.74524
MF.20121127.37926.fits ... 29.74441
MF.20121127.37946.fits ... 29.74441
MF.20121127.37969.fits ... 29.74524
MF.20121127.38012.fits ... 29.74524
MF.20121127.38034.fits ... 29.74441
                   ... ...      ...
MF.20191014.42587.fits ... 29.77373
MF.20191014.42603.fits ...  29.7819
MF.20191014.51892.fits ... 29.74492
MF.20191014.52137.fits ... 29.74492
MF.20191014.52197.fits ... 29.74479
MF.20191014.52333.fits ... 29.71124
MF.20191014.52356.fits ... 29.70292
MF.20191014.52417.fits ... 29.71124
MF.20191014.52440.fits ... 29.70292
MF.20191014.52464.fits ... 29.71124
MF.20191014.52487.fits ... 29.70292
Length = 530 rows
In [17]:
sql=("select koaid, filehand, ra, dec from koa_mosfire  where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',57.6, 29.7, 57.6, 29.8, 57.7, 29.8, 57.7, 29.7)) = 1")


job = koa.launch_job_async(sql)
r = job.get_results()
print(r)

job = koa.launch_job_async(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
INFO: Query finished. [astroquery.utils.tap.core]
        koaid          ...   dec   
---------------------- ... --------
MF.20121127.37568.fits ... 29.74463
MF.20121127.37610.fits ... 29.74483
MF.20121127.37795.fits ... 29.74524
MF.20121127.37820.fits ... 29.74441
MF.20121127.37899.fits ... 29.74524
MF.20121127.37926.fits ... 29.74441
MF.20121127.37946.fits ... 29.74441
MF.20121127.37969.fits ... 29.74524
MF.20121127.38012.fits ... 29.74524
MF.20121127.38034.fits ... 29.74441
                   ... ...      ...
MF.20191014.42587.fits ... 29.77373
MF.20191014.42603.fits ...  29.7819
MF.20191014.51892.fits ... 29.74492
MF.20191014.52137.fits ... 29.74492
MF.20191014.52197.fits ... 29.74479
MF.20191014.52333.fits ... 29.71124
MF.20191014.52356.fits ... 29.70292
MF.20191014.52417.fits ... 29.71124
MF.20191014.52440.fits ... 29.70292
MF.20191014.52464.fits ... 29.71124
MF.20191014.52487.fits ... 29.70292
Length = 526 rows

Count number of records on date

In [18]:
sql=("select count(*) from koa_mosfire where \
    date_obs=to_date('2012-11-28', 'YYYY-MM-DD')")

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)
INFO: Query finished. [astroquery.utils.tap.core]
count(*)
--------
     556

Count total number of records in table

In [19]:
sql="select count(*) as total from koa_mosfire"

job = koa.launch_job_async(sql)
r = job.get_results()
print(r)
INFO: Query finished. [astroquery.utils.tap.core]
total 
------
293473

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA