Using the nexsciTAP Server with the TAP+ Client to Access Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the DEep Imaging Multi-Object Spectrograph (DEIMOS) these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (March, 2021)

Set up

In [1]:
import numpy as np
from astroquery.utils.tap.core import TapPlus

koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: koa.ipac.caltech.edu
	Use HTTPS: True
	Port: 443
	SSL Port: 443

Search by date

In [2]:
sql = "select koaid, filehand from koa_deimos where koaid like '%20150901%' "
job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid                                 filehand                       
---------------------- ------------------------------------------------------
DE.20150901.07424.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.07424.fits
DE.20150901.08740.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08740.fits
DE.20150901.08811.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08811.fits
DE.20150901.08879.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08879.fits
DE.20150901.08950.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08950.fits
DE.20150901.09052.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09052.fits
DE.20150901.09522.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09522.fits

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [3]:
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate1.tbl', \
                     output_format='ipac')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate2.vot', \
                     output_format='votable')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate3.csv', \
                     output_format='csv')
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapDate4.tsv', \
                     output_format='tsv')

Select all keywords for data acquired on UT date

In [4]:
sql = "select * from koa_deimos where koaid like '%20150901%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='table_ascii.tbl', \
                     output_format='ipac')
        koaid               ofname     filesize_mb ...  spt_ind   cntr 
---------------------- --------------- ----------- ... --------- ------
DE.20150901.07424.fits d0827_0037.fits          -- ... 132000000 150853
DE.20150901.08740.fits d0827_0038.fits          -- ... 132000000 150854
DE.20150901.08811.fits d0827_0039.fits          -- ... 132000000 150855
DE.20150901.08879.fits d0827_0040.fits          -- ... 132000000 150856
DE.20150901.08950.fits d0827_0041.fits          -- ... 132000000 150857
DE.20150901.09052.fits d0827_0042.fits          -- ... 132000000 150858
DE.20150901.09522.fits d0827_0043.fits          -- ... 132000000 150859

Select columns acquired on UT Date

In [5]:
sql = "select koaid, filehand from koa_deimos \
    where koaid like '%20150901%' "

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='columns.tbl', \
                     output_format='ipac')
        koaid                                 filehand                       
---------------------- ------------------------------------------------------
DE.20150901.07424.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.07424.fits
DE.20150901.08740.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08740.fits
DE.20150901.08811.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08811.fits
DE.20150901.08879.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08879.fits
DE.20150901.08950.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08950.fits
DE.20150901.09052.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09052.fits
DE.20150901.09522.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09522.fits

Select a maximum number of records

In [7]:
sql = "select top 5 koaid, filehand, frameno from koa_deimos where koaid like '%20150901%'"
job = koa.launch_job(sql, dump_to_file=True, output_file='KoaTapTestFiniteRecors.tbl', output_format='ipac')

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='top20.tbl', \
                     output_format='ipac')
        koaid          ... frameno
---------------------- ... -------
DE.20150901.08740.fits ...      38
DE.20150901.08811.fits ...      39
DE.20150901.08879.fits ...      40
DE.20150901.08950.fits ...      41
DE.20150901.09522.fits ...      43

Select columns and search by date range

In [8]:
sql="select koaid, filehand from koa_deimos \
     where (utdatetime >= to_date('2015-09-01 00:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2015-09-30 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='date_range.tbl', \
                     output_format='ipac')
        koaid                                 filehand                       
---------------------- ------------------------------------------------------
DE.20150901.07424.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.07424.fits
DE.20150901.08740.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08740.fits
DE.20150901.08811.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08811.fits
DE.20150901.08879.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08879.fits
DE.20150901.08950.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.08950.fits
DE.20150901.09052.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09052.fits
DE.20150901.09522.fits /koadata37/DEIMOS/20150901/lev0/DE.20150901.09522.fits
DE.20150902.68203.fits /koadata37/DEIMOS/20150902/lev0/DE.20150902.68203.fits
DE.20150902.68248.fits /koadata37/DEIMOS/20150902/lev0/DE.20150902.68248.fits
DE.20150902.68293.fits /koadata37/DEIMOS/20150902/lev0/DE.20150902.68293.fits
                   ...                                                    ...
DE.20150928.74389.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.74389.fits
DE.20150928.75494.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.75494.fits
DE.20150928.77735.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.77735.fits
DE.20150928.77781.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.77781.fits
DE.20150928.77829.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.77829.fits
DE.20150928.77877.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.77877.fits
DE.20150928.77922.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.77922.fits
DE.20150928.77970.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.77970.fits
DE.20150928.78015.fits /koadata37/DEIMOS/20150929/lev0/DE.20150928.78015.fits
DE.20150929.00207.fits /koadata37/DEIMOS/20150929/lev0/DE.20150929.00207.fits
DE.20150929.00555.fits /koadata37/DEIMOS/20150929/lev0/DE.20150929.00555.fits
Length = 1055 rows

Search by program ID

In [9]:
sql= ("select koaid, filehand, progid, ha, az, detector, gratenam \
    from koa_deimos where (progid = 'U024') ")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='progID.tbl', \
                     output_format='ipac')
        koaid          ... gratenam
---------------------- ... --------
DE.20190605.06053.fits ...    600ZD
DE.20190605.10617.fits ...    600ZD
DE.20190605.10691.fits ...    600ZD
DE.20190605.12860.fits ...    600ZD
DE.20190605.12938.fits ...    600ZD
DE.20190605.13014.fits ...    600ZD
DE.20190605.13089.fits ...    600ZD
DE.20190605.13197.fits ...    600ZD
DE.20190605.13275.fits ...    600ZD
DE.20190605.13351.fits ...    600ZD
                   ... ...      ...
DF.20190606.09520.fits ...     830G
DF.20190606.09732.fits ...     830G
DF.20190705.10814.fits ...    600ZD
DF.20190705.10919.fits ...    600ZD
DF.20190705.11248.fits ...    600ZD
DF.20190705.11506.fits ...     830G
DF.20190706.06286.fits ...    600ZD
DF.20190706.06705.fits ...    600ZD
DF.20190706.06915.fits ...    600ZD
DF.20190706.07219.fits ...     830G
DF.20190706.07433.fits ...     830G
Length = 222 rows
In [11]:
sql=("select koaid, filehand, ra, dec from koa_deimos where \
    contains(point('icrs', ra, dec), \
    circle('icrs', 23.4, 30.60,1.0)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='Spatial_cone.txt', \
                     output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
DE.20031018.19913.fits ... 30.95914
DE.20031018.20064.fits ... 30.95914
DE.20031018.20249.fits ... 30.95881
DE.20031018.20392.fits ... 30.95872
DE.20031018.20535.fits ... 30.95878
DE.20031018.21134.fits ... 30.95325
DE.20031018.21289.fits ... 30.95297
DE.20031018.21429.fits ...   30.953
DE.20031018.21616.fits ...   30.953
DE.20031018.22378.fits ... 30.95336
                   ... ...      ...
DE.20181204.26084.fits ... 30.73564
DE.20181204.27263.fits ... 30.73564
DE.20181204.28434.fits ... 30.73564
DE.20181204.29603.fits ... 30.73564
DE.20181204.31202.fits ... 30.60358
DE.20181204.31296.fits ... 30.60375
DE.20181204.31462.fits ... 30.60375
DE.20181204.32522.fits ... 30.60375
DE.20181204.33582.fits ... 30.60375
DE.20181204.34642.fits ... 30.60375
DE.20181204.35702.fits ... 30.60375
Length = 401 rows
In [12]:
sql=("select koaid, filehand, ra, dec from koa_deimos where \
     contains(point('icrs', ra, dec), \
     box('icrs',23.4, 30.60,1.0,120)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatBox.txt', output_format='ipac')
        koaid          ...    dec   
---------------------- ... ---------
DE.20020827.49257.fits ...  60.41256
DE.20020827.49632.fits ...  60.41256
DE.20020827.49790.fits ...  60.41256
DE.20020917.59390.fits ...      90.0
DE.20030108.16887.fits ...   -27.814
DE.20030108.16970.fits ...   -27.814
DE.20030109.17248.fits ... -27.49983
DE.20030109.17330.fits ... -27.49983
DE.20030819.84802.fits ...      90.0
DE.20030819.85813.fits ...      90.0
                   ... ...       ...
DE.20190825.76600.fits ...     90.01
DE.20190825.77507.fits ...     90.01
DE.20190825.77609.fits ...     90.01
DE.20190825.80669.fits ...     90.01
DE.20190825.82643.fits ...     90.01
DE.20190825.82739.fits ...     90.01
DE.20190825.84498.fits ...     90.01
DE.20190825.85012.fits ...     90.01
DE.20190825.85257.fits ...     90.01
DE.20190825.85455.fits ...     90.01
DE.20190825.85569.fits ...     90.01
Length = 2000 rows

TAP+ returns the first 2,000 records in this query

In [14]:
sql=("select koaid, filehand, ra, dec from koa_deimos  where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file='SpatPoly.txt', \
                       output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
DE.20100313.56134.fits ... 54.34917
DE.20100313.56233.fits ... 54.34917
DE.20100313.56356.fits ... 54.34917
DE.20100313.56466.fits ... 54.34917
DE.20100313.56584.fits ... 54.34917
DE.20100313.56780.fits ... 54.34917
DE.20100313.56898.fits ... 54.34917
DE.20100313.57009.fits ... 54.34917
DE.20120226.55210.fits ... 54.26819
DE.20120226.55594.fits ... 54.26869
                   ... ...      ...
DE.20160428.47679.fits ... 54.39894
DE.20160428.48021.fits ...  54.4045
DE.20160704.30504.fits ... 54.45833
DE.20160704.30975.fits ... 54.45833
DE.20160704.31540.fits ... 54.45833
DE.20160704.31910.fits ... 54.45833
DE.20160704.32336.fits ... 54.45833
DE.20160704.32707.fits ... 54.45833
DE.20180705.24528.fits ... 54.43911
DE.20180705.26500.fits ... 54.43772
DE.20180705.26576.fits ... 54.43772
Length = 37 rows

Count number of records in date range

In [15]:
sql=("select count(*) from koa_deimos where \
    (utdatetime >= to_date('2009-01-01 00:00:00', 'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2009-01-31 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"  )

job = koa.launch_job(sql)
r = job.get_results()
print(r)
count(*)
--------
     404

Count total number of records in table

In [16]:
sql="select count(*) as total from koa_deimos"

job = koa.launch_job(sql)
r = job.get_results()
print(r)
total
-----
 2000

Note: There are 170,000+ records that satisfy this query, but TAP+ only returns the first 2,000


Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA