Using the nexsciTAP Server with the TAP+ Client to Access LRIS Data From KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the TAP+ client to perform synchronous TAP-based queries for public raw science and calibration data acquired with the Low Resolution Imaging Spectrograph (LRIS) data, hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

TAP+ was developed at the ESAC Space Data Centre (ESDC: http://www.cosmos.esa.int/web/esdc/) and can be installed from PyPI as part of the astroquery package:

$ pip install --upgrade astroquery

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily. In synchronous mode, the TAP+ client returns the first 2,000 records returned.

Version 1.0 (September, 2021)

Set up

In [1]:
import os
import numpy as np
from astroquery.utils.tap.core import TapPlus
koa = TapPlus(url="https://koa.ipac.caltech.edu/TAP")
Created TAP+ (v1.2.1) - Connection:
	Host: vmkoatest.ipac.caltech.edu
	Use HTTPS: False
	Port: 8000
	SSL Port: 443

Define search parameters

In [2]:
instrument = 'LRIS'

instr = 'lris'
koatable = 'koa_lris'    
outdir = './outputLR_tapplus_sync/'
koaid_date = '20150123'

startdate = '2015-01-01 00:00:00'
enddate   = '2015-03-01 23:59:59'
    
date='20150123'

program = 'C305'


circle_pos = '174.27 29.8 1.0'
box_pos = '174.27 29.8 1 1'
polygon_pos = '173.27 28.8 173.27 30.8 176.27 29.8 '

Create output directory

In [3]:
try:
    os.mkdir(outdir)
except:
    print(instrument + " Directory exists already", flush=True)

Search by date

In [4]:
sql = "select koaid, filehand from " + koatable + " where koaid like '%" + koaid_date + "%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)
        koaid                                filehand                      
---------------------- ----------------------------------------------------
LR.20150123.00166.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00166.fits
LR.20150123.00300.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00300.fits
LR.20150123.00538.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00538.fits
LR.20150123.00792.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00792.fits
LR.20150123.01035.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01035.fits
LR.20150123.01289.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01289.fits
LR.20150123.01603.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01603.fits

Write the output to the IPAC ASCII, VOtable CSV and TSV formats

In [5]:
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_KoaTapDate1.tbl", output_format='ipac')
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_KoaTapDate2.vot", output_format='votable')
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_KoaTapDate3.csv", output_format='csv')
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_KoaTapDate4.tsv", output_format='tsv')

Select all keywords for data acquired on UT date

In [6]:
sql = "select * from " + koatable + " where koaid like '%" + koaid_date + "%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)

job = koa.launch_job(sql, dump_to_file=True, output_file=outdir + instr + "_table_ascii.tbl", \
                     output_format='ipac')
        koaid          propint instrume ...    y      z   spt_ind 
                                        ...                       
---------------------- ------- -------- ... -------- --- ---------
LR.20150123.00166.fits      18     LRIS ... -0.92293 0.0 210210000
LR.20150123.00300.fits      18     LRIS ... -0.92293 0.0 210210000
LR.20150123.00538.fits      18     LRIS ... -0.92293 0.0 210210000
LR.20150123.00792.fits      18     LRIS ... -0.92293 0.0 210210000
LR.20150123.01035.fits      18     LRIS ... -0.92293 0.0 210210000
LR.20150123.01289.fits      18     LRIS ... -0.92293 0.0 210210000
LR.20150123.01603.fits      18     LRIS ... -0.92293 0.0 210210000

Select columns acquired on UT Date

In [7]:
sql = "select koaid, filehand from " + koatable + " where koaid like '%" + koaid_date + "%'"
job = koa.launch_job(sql)
r = job.get_results()
print(r)
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_columns.tbl", output_format='ipac')
        koaid                                filehand                      
---------------------- ----------------------------------------------------
LR.20150123.00166.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00166.fits
LR.20150123.00300.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00300.fits
LR.20150123.00538.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00538.fits
LR.20150123.00792.fits /koadata25/LRIS/20150123/lev0/LR.20150123.00792.fits
LR.20150123.01035.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01035.fits
LR.20150123.01289.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01289.fits
LR.20150123.01603.fits /koadata25/LRIS/20150123/lev0/LR.20150123.01603.fits

Select a maximum number of records

In [8]:
sql = "select top 20 koaid, filehand, frameno from " + koatable + " where koaid like '%" + koaid_date + "%'"
job = koa.launch_job(sql, dump_to_file=True, output_file=  outdir + instr + "_KoaTapTestFiniteRecors.tbl", output_format='ipac')
job = koa.launch_job(sql)
r = job.get_results()
print(r)
job = koa.launch_job(sql, dump_to_file=True, output_file=  outdir + instr + "_top20.tbl", output_format='ipac')
        koaid          ... frameno
---------------------- ... -------
LR.20150123.00166.fits ...       8
LR.20150123.00300.fits ...       9
LR.20150123.00538.fits ...      10
LR.20150123.00792.fits ...      11
LR.20150123.01035.fits ...      12
LR.20150123.01289.fits ...      13
LR.20150123.01603.fits ...      14

Select columns and search by datetime range

In [9]:
sql = "select koaid, filehand from " + koatable + " where (utdatetime >= to_date('" + startdate + "', 'yyyy-mm-dd HH24:MI:SS') and utdatetime <= to_date('" + enddate + "', 'yyyy-mm-dd HH24:MI:SS'))"
job = koa.launch_job(sql)
r = job.get_results()
print(r)
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_datetime_range.tbl", output_format='ipac')
        koaid                                filehand                      
---------------------- ----------------------------------------------------
LB.20150119.72427.fits /koadata25/LRIS/20150120/lev0/LB.20150119.72427.fits
LB.20150119.72771.fits /koadata25/LRIS/20150120/lev0/LB.20150119.72771.fits
LB.20150119.78800.fits /koadata25/LRIS/20150120/lev0/LB.20150119.78800.fits
LB.20150120.02709.fits /koadata25/LRIS/20150120/lev0/LB.20150120.02709.fits
LB.20150120.03172.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03172.fits
LB.20150120.03492.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03492.fits
LB.20150120.03563.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03563.fits
LB.20150120.03635.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03635.fits
LB.20150120.03704.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03704.fits
LB.20150120.03774.fits /koadata25/LRIS/20150120/lev0/LB.20150120.03774.fits
                   ...                                                  ...
LR.20150222.61716.fits /koadata25/LRIS/20150222/lev0/LR.20150222.61716.fits
LR.20150222.61846.fits /koadata25/LRIS/20150222/lev0/LR.20150222.61846.fits
LR.20150222.62009.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62009.fits
LR.20150222.62140.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62140.fits
LR.20150222.62272.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62272.fits
LR.20150222.62402.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62402.fits
LR.20150222.62533.fits /koadata25/LRIS/20150222/lev0/LR.20150222.62533.fits
LR.20150224.75522.fits /koadata25/LRIS/20150225/lev0/LR.20150224.75522.fits
LR.20150224.76599.fits /koadata25/LRIS/20150225/lev0/LR.20150224.76599.fits
LR.20150224.76732.fits /koadata25/LRIS/20150225/lev0/LR.20150224.76732.fits
LR.20150224.77046.fits /koadata25/LRIS/20150225/lev0/LR.20150224.77046.fits
Length = 2000 rows

Search by program ID

In [10]:
sql= ("select koaid, filehand, progid from " + koatable + " where (progid = '" + program  + "')")
job = koa.launch_job(sql)
r = job.get_results()
print(r)
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_progID.tbl", output_format='ipac')
        koaid          ... progid
---------------------- ... ------
LB.20180512.01660.fits ...   C305
LB.20180512.03116.fits ...   C305
LB.20180512.03186.fits ...   C305
LB.20180512.03390.fits ...   C305
LB.20180512.03473.fits ...   C305
LB.20180512.04976.fits ...   C305
LB.20180512.05041.fits ...   C305
LB.20180512.05101.fits ...   C305
LB.20180512.05161.fits ...   C305
LB.20180512.05221.fits ...   C305
                   ... ...    ...
LR.20180512.38834.fits ...   C305
LR.20180512.40161.fits ...   C305
LR.20180512.41491.fits ...   C305
LR.20180512.42818.fits ...   C305
LR.20180512.44145.fits ...   C305
LR.20180512.45472.fits ...   C305
LR.20180512.46800.fits ...   C305
LR.20180512.48127.fits ...   C305
LR.20180512.49454.fits ...   C305
LR.20180512.50781.fits ...   C305
LR.20180512.52109.fits ...   C305
Length = 140 rows
In [11]:
sql=("select koaid, filehand,ra, dec from " + koatable + " where contains(point('icrs', ra, dec), circle('icrs', " + circle_pos + ")) = 1")
job = koa.launch_job(sql)
r = job.get_results()
print(r)
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_Spatial_cone.txt", output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
LB.20010529.20873.fits ... 30.00314
LB.20010529.21063.fits ... 30.00314
LB.20010529.21331.fits ... 29.99631
LB.20010529.21738.fits ...   29.996
LB.20010529.21996.fits ... 29.99589
LB.20010529.22312.fits ... 29.99603
LB.20010529.23664.fits ... 29.99608
LB.20010529.25022.fits ...   29.996
LB.20011211.57154.fits ... 29.79737
LB.20020109.59153.fits ... 29.80264
                   ... ...      ...
LR.20100121.57297.fits ... 29.79419
LR.20100121.57605.fits ... 29.78861
LR.20120428.22909.fits ... 29.79953
LR.20120428.23105.fits ... 29.79953
LR.20180512.20205.fits ... 29.79947
LR.20180512.20631.fits ... 29.79947
LR.20180512.20857.fits ... 29.79947
LR.20180512.21080.fits ... 29.79947
LR.20181229.56431.fits ... 29.79942
LR.20181229.56539.fits ... 29.79942
LR.20190304.33130.fits ... 29.94239
Length = 101 rows
In [12]:
sql=("select koaid, filehand,ra, dec from " + koatable + " where contains(point('J2000', ra, dec), box('J2000', " + box_pos + ")) = 1")
job = koa.launch_job(sql)
r = job.get_results()
print(r)
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_Spatial_box.txt", output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
LB.20010529.20873.fits ... 30.00314
LB.20010529.21063.fits ... 30.00314
LB.20010529.21331.fits ... 29.99631
LB.20010529.21738.fits ...   29.996
LB.20010529.21996.fits ... 29.99589
LB.20010529.22312.fits ... 29.99603
LB.20010529.23664.fits ... 29.99608
LB.20010529.25022.fits ...   29.996
LB.20011211.57154.fits ... 29.79737
LB.20020109.59153.fits ... 29.80264
                   ... ...      ...
LR.20100121.57297.fits ... 29.79419
LR.20100121.57605.fits ... 29.78861
LR.20120428.22909.fits ... 29.79953
LR.20120428.23105.fits ... 29.79953
LR.20180512.20205.fits ... 29.79947
LR.20180512.20631.fits ... 29.79947
LR.20180512.20857.fits ... 29.79947
LR.20180512.21080.fits ... 29.79947
LR.20181229.56431.fits ... 29.79942
LR.20181229.56539.fits ... 29.79942
LR.20190304.33130.fits ... 29.94239
Length = 95 rows
In [13]:
sql=("select koaid, filehand,ra, dec from " + koatable + " where contains(point('J2000', ra, dec), polygon('J2000', " + polygon_pos + ")) = 1")
job = koa.launch_job(sql)
r = job.get_results()
print(r)
job = koa.launch_job(sql, dump_to_file=True, output_file= outdir + instr + "_Spatial_polygon.txt", output_format='ipac')
        koaid          ...   dec   
---------------------- ... --------
LB.20010529.20873.fits ... 30.00314
LB.20010529.21063.fits ... 30.00314
LB.20010529.21331.fits ... 29.99631
LB.20010529.21738.fits ...   29.996
LB.20010529.21996.fits ... 29.99589
LB.20010529.22312.fits ... 29.99603
LB.20010529.23664.fits ... 29.99608
LB.20010529.25022.fits ...   29.996
LB.20011211.57154.fits ... 29.79737
LB.20020109.59153.fits ... 29.80264
                   ... ...      ...
LR.20100121.57297.fits ... 29.79419
LR.20100121.57605.fits ... 29.78861
LR.20120428.22909.fits ... 29.79953
LR.20120428.23105.fits ... 29.79953
LR.20180512.20205.fits ... 29.79947
LR.20180512.20631.fits ... 29.79947
LR.20180512.20857.fits ... 29.79947
LR.20180512.21080.fits ... 29.79947
LR.20181229.56431.fits ... 29.79942
LR.20181229.56539.fits ... 29.79942
LR.20190304.33130.fits ... 29.94239
Length = 101 rows

Count number of records on date

In [14]:
sql=("select count(*) from " + koatable + " where date_obs=to_date('" + date + "', 'YYYY-MM-DD')")
job = koa.launch_job(sql)
r = job.get_results()
print(r)
count(*)
--------
       7

Count total number of records in table

In [15]:
sql="select count(*) as total from " + koatable
job = koa.launch_job(sql)
r = job.get_results()
print(r)
total
-----
 2000

Note: There are more than 500,000 records that satisfy this query, but TAP+ only returns the first 2,000 in synchronous mode


Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA