Using the nexsciTAP Server with the PyVO Client to Access MOSFIRE Data At KOA In Synchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, including cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform synchronous TAP-based queries for public raw science and calibation data acquired with the MOSFIRE near-infrared multi-object spectrograph; these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or above), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (June, 2021)

Set up

In [1]:
from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")

import time

import sys
import os
import time

Query by date

In [2]:
sql = "select koaid, filehand from koa_mosfire where koaid like '%20121128%'"
results= koa.run_sync(sql)
print(results)


# write results to VOTable format
table=results.to_table()
table.write ('./table_ipacascii.vot',format='ascii.ipac',overwrite=True)
<Table masked=True length=556>
        koaid                                  filehand                       
        object                                  object                        
---------------------- -------------------------------------------------------
MF.20121128.07841.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07841.fits
MF.20121128.07866.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07866.fits
MF.20121128.07892.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07892.fits
MF.20121128.07917.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07917.fits
MF.20121128.07942.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07942.fits
MF.20121128.07966.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07966.fits
MF.20121128.07996.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07996.fits
MF.20121128.08031.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08031.fits
MF.20121128.08051.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08051.fits
MF.20121128.08077.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08077.fits
                   ...                                                     ...
MF.20121128.58916.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58916.fits
MF.20121128.58982.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58982.fits
MF.20121128.59002.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59002.fits
MF.20121128.59026.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59026.fits
MF.20121128.59074.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59074.fits
MF.20121128.59104.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59104.fits
MF.20121128.59135.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59135.fits
MF.20121128.60043.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.60043.fits
MF.20121128.73782.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73782.fits
MF.20121128.73795.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73795.fits

PyVO by default returns results in VOTable format. Convert them to IPAC ASCII and CSV format.

In [3]:
# write results to an IPAC ASCII table
sql = "select koaid, filehand from koa_mosfire where koaid like '%20121128%' "
results = koa.run_sync(sql)
table=results.to_table()
table.write ('./table_ipacascii.tbl',format='ascii.ipac',overwrite=True)
In [4]:
#  write results to CSV file
sql = "select koaid, filehand from koa_mosfire where koaid like '%20121128%' "
results = koa.run_sync(sql)
table=results.to_table()
table.write ('./table_csv.csv',format='csv', overwrite=True)

Select all keywords for data acquired on a UT Date

In [5]:
sql = "select * from koa_mosfire where koaid like '%20121128%' "
results = koa.run_sync(sql)
print (results)

## write to an output file
table_selectall=results.to_table()
table_selectall.write ('./table_selectall.vot',format='votable',overwrite=True)
<Table masked=True length=556>
        koaid                ofname      coadds ...  spt_ind   cntr
        object               object      int32  ...   int32   int32
---------------------- ----------------- ------ ... --------- -----
MF.20121128.07841.fits m121128_0001.fits      1 ... 233000300 20359
MF.20121128.07866.fits m121128_0002.fits      1 ... 233000300 20360
MF.20121128.07892.fits m121128_0003.fits      1 ... 233000300 20361
MF.20121128.07917.fits m121128_0004.fits      1 ... 233000300 20362
MF.20121128.07942.fits m121128_0005.fits      1 ... 233000300 20363
MF.20121128.07966.fits m121128_0006.fits      1 ... 233000300 20364
MF.20121128.07996.fits m121128_0007.fits      1 ... 233000300 20365
MF.20121128.08031.fits m121128_0008.fits      1 ... 233000300 20366
MF.20121128.08051.fits m121128_0009.fits      1 ... 233000300 20367
MF.20121128.08077.fits m121128_0010.fits      1 ... 233000300 20368
                   ...               ...    ... ...       ...   ...
MF.20121128.58916.fits m121128_0549.fits      1 ... 233000300 20905
MF.20121128.58982.fits m121128_0550.fits      1 ... 233000300 20906
MF.20121128.59002.fits m121128_0551.fits      1 ... 233000300 20907
MF.20121128.59026.fits m121128_0552.fits      1 ... 233000300 20908
MF.20121128.59074.fits m121128_0553.fits      1 ... 233000300 20909
MF.20121128.59104.fits m121128_0554.fits      1 ... 233000300 20910
MF.20121128.59135.fits m121128_0555.fits      1 ... 233000300 20911
MF.20121128.60043.fits m121128_0556.fits      1 ... 233000300 20912
MF.20121128.73782.fits m121129_0001.fits      1 ... 233000300 13514
MF.20121128.73795.fits m121129_0002.fits      1 ... 233000300 13515

Select columns acquired on UT date

In [6]:
sql = "select koaid, filehand  from koa_mosfire where koaid like '%20121128%'"
results = koa.run_sync(sql)
print (results)

table   = results.to_table()
table.write('./select_orderdoesnot.vot',format='ascii.ipac',overwrite=True)
<Table masked=True length=556>
        koaid                                  filehand                       
        object                                  object                        
---------------------- -------------------------------------------------------
MF.20121128.07841.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07841.fits
MF.20121128.07866.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07866.fits
MF.20121128.07892.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07892.fits
MF.20121128.07917.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07917.fits
MF.20121128.07942.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07942.fits
MF.20121128.07966.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07966.fits
MF.20121128.07996.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07996.fits
MF.20121128.08031.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08031.fits
MF.20121128.08051.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08051.fits
MF.20121128.08077.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08077.fits
                   ...                                                     ...
MF.20121128.58916.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58916.fits
MF.20121128.58982.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.58982.fits
MF.20121128.59002.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59002.fits
MF.20121128.59026.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59026.fits
MF.20121128.59074.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59074.fits
MF.20121128.59104.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59104.fits
MF.20121128.59135.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.59135.fits
MF.20121128.60043.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.60043.fits
MF.20121128.73782.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73782.fits
MF.20121128.73795.fits /koadata34/MOSFIRE/20121129/lev0/MF.20121128.73795.fits

Select a maximum number of records

In [7]:
sql="select koaid, filehand, frameno from koa_mosfire \
    where koaid like '%20121128%'"
results = koa.run_sync(sql, maxrec=5)
print (results)

table   = results.to_table()
table.write('./select_maxrec.tbl',format='ascii.ipac',overwrite=True)
<Table masked=True length=5>
        koaid          ... frameno
        object         ...  int32 
---------------------- ... -------
MF.20121128.07841.fits ...       1
MF.20121128.07866.fits ...       2
MF.20121128.07892.fits ...       3
MF.20121128.07917.fits ...       4
MF.20121128.07942.fits ...       5

Select records by time range

In [8]:
sql="select koaid, filehand from koa_mosfire \
     where (utdatetime >= to_date('2012-11-28 00:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2012-12-01 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"
results  = koa.run_sync(sql)

print (results)

table_daterange=results.to_table()
table.write ('./table_daterange.vot',format='votable',overwrite=True)
<Table masked=True length=579>
        koaid                                  filehand                       
        object                                  object                        
---------------------- -------------------------------------------------------
MF.20121128.07841.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07841.fits
MF.20121128.07866.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07866.fits
MF.20121128.07892.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07892.fits
MF.20121128.07917.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07917.fits
MF.20121128.07942.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07942.fits
MF.20121128.07966.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07966.fits
MF.20121128.07996.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.07996.fits
MF.20121128.08031.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08031.fits
MF.20121128.08051.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08051.fits
MF.20121128.08077.fits /koadata34/MOSFIRE/20121128/lev0/MF.20121128.08077.fits
                   ...                                                     ...
MF.20121201.02631.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.02631.fits
MF.20121201.02723.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.02723.fits
MF.20121201.02766.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.02766.fits
MF.20121201.02824.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.02824.fits
MF.20121201.02908.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.02908.fits
MF.20121201.03103.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.03103.fits
MF.20121201.03352.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.03352.fits
MF.20121201.04038.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.04038.fits
MF.20121201.04110.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.04110.fits
MF.20121201.04500.fits /koadata34/MOSFIRE/20121201/lev0/MF.20121201.04500.fits

Select records by program ID

In [9]:
sql= ("select koaid, filehand, progid  \
    from koa_mosfire where (progid = 'N071') ")
results  = koa.run_sync(sql)
print (results)

table_progid=results.to_table()
table.write ('./progid.vot',format='votable',overwrite=True)
<Table masked=True length=474>
        koaid          ... progid
        object         ... object
---------------------- ... ------
MF.20191012.37804.fits ...   N071
MF.20191012.38102.fits ...   N071
MF.20191012.38126.fits ...   N071
MF.20191012.38184.fits ...   N071
MF.20191012.38204.fits ...   N071
MF.20191012.38390.fits ...   N071
MF.20191012.38412.fits ...   N071
MF.20191012.38470.fits ...   N071
MF.20191012.38489.fits ...   N071
MF.20191012.38837.fits ...   N071
                   ... ...    ...
MF.20191014.56538.fits ...   N071
MF.20191014.56552.fits ...   N071
MF.20191014.56566.fits ...   N071
MF.20191014.56580.fits ...   N071
MF.20191014.56594.fits ...   N071
MF.20191014.56607.fits ...   N071
MF.20191014.56620.fits ...   N071
MF.20191014.56690.fits ...   N071
MF.20191014.56701.fits ...   N071
MF.20191014.56710.fits ...   N071

Spatial cone search with column selection

In [10]:
sql=("select koaid, filehand, ra, dec  from koa_mosfire where \
    contains(point('icrs', ra, dec), \
    circle('icrs', 57.63, 29.74, 0.5)) = 1")

results  = koa.run_sync(sql)
print(results)

table_spatial_1=results.to_table()
table.write ('./table_spatial_1.vot',format='votable',overwrite=True)
<Table masked=True length=530>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
MF.20121127.37568.fits ... 29.74463
MF.20121127.37610.fits ... 29.74483
MF.20121127.37795.fits ... 29.74524
MF.20121127.37820.fits ... 29.74441
MF.20121127.37899.fits ... 29.74524
MF.20121127.37926.fits ... 29.74441
MF.20121127.37946.fits ... 29.74441
MF.20121127.37969.fits ... 29.74524
MF.20121127.38012.fits ... 29.74524
MF.20121127.38034.fits ... 29.74441
                   ... ...      ...
MF.20191014.42603.fits ...  29.7819
MF.20191014.51892.fits ... 29.74492
MF.20191014.52137.fits ... 29.74492
MF.20191014.52197.fits ... 29.74479
MF.20191014.52333.fits ... 29.71124
MF.20191014.52356.fits ... 29.70292
MF.20191014.52417.fits ... 29.71124
MF.20191014.52440.fits ... 29.70292
MF.20191014.52464.fits ... 29.71124
MF.20191014.52487.fits ... 29.70292
In [11]:
sql=("select koaid, filehand, ra, dec from koa_mosfire where \
     contains(point('icrs', ra, dec), \
     box('icrs',57.63, 29.74,1.0,120)) = 1")       

results = koa.run_sync(sql)
print (results)

table_spatial_box=results.to_table()
table.write('./table_spatial_box.vot',format='votable',overwrite=True)
<Table masked=True length=1384>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
MF.20121127.37568.fits ... 29.74463
MF.20121127.37610.fits ... 29.74483
MF.20121127.37795.fits ... 29.74524
MF.20121127.37820.fits ... 29.74441
MF.20121127.37899.fits ... 29.74524
MF.20121127.37926.fits ... 29.74441
MF.20121127.37946.fits ... 29.74441
MF.20121127.37969.fits ... 29.74524
MF.20121127.38012.fits ... 29.74524
MF.20121127.38034.fits ... 29.74441
                   ... ...      ...
MF.20191014.42603.fits ...  29.7819
MF.20191014.51892.fits ... 29.74492
MF.20191014.52137.fits ... 29.74492
MF.20191014.52197.fits ... 29.74479
MF.20191014.52333.fits ... 29.71124
MF.20191014.52356.fits ... 29.70292
MF.20191014.52417.fits ... 29.71124
MF.20191014.52440.fits ... 29.70292
MF.20191014.52464.fits ... 29.71124
MF.20191014.52487.fits ... 29.70292
In [12]:
sql=("select koaid, filehand, ra, dec from koa_mosfire where \
    contains(point('icrs', ra, dec), \
    polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1")

results = koa.run_sync(sql)
print(results)

table_spatial_P=results.to_table()
table.write ('./table_spatial_P.tbl',format='votable',overwrite=True)
<Table masked=True length=625>
        koaid          ...   dec   
        object         ... float64 
---------------------- ... --------
MF.20140607.39374.fits ... 54.38761
MF.20140607.39442.fits ... 54.38761
MF.20140607.39516.fits ... 54.38769
MF.20140607.39877.fits ... 54.38809
MF.20140607.40092.fits ... 54.38727
MF.20140607.40307.fits ... 54.38809
MF.20140607.40523.fits ... 54.38727
MF.20140607.40737.fits ... 54.38809
MF.20140607.40952.fits ... 54.38727
MF.20140607.41165.fits ... 54.38809
                   ... ...      ...
MF.20190713.25344.fits ... 54.29987
MF.20190713.25495.fits ... 54.30065
MF.20190713.25645.fits ... 54.30065
MF.20190713.25796.fits ... 54.29987
MF.20190713.25947.fits ... 54.29987
MF.20190713.26099.fits ... 54.30065
MF.20190713.26249.fits ... 54.30065
MF.20190713.26400.fits ... 54.29987
MF.20190713.26550.fits ... 54.29987
MF.20190713.26702.fits ... 54.30065

Count number of records in date range

In [13]:
sql=("select count(*) as total from koa_mosfire where \
    (utdatetime >= to_date('2012-11-28 00:00:00', 'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2012-12-01 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"  )

results = koa.run_sync(sql)
print(results)
<Table masked=True length=1>
total
int32
-----
  579

Count total number of records in table

In [14]:
sql="select count(*) as total from koa_mosfire"

results = koa.run_sync(sql)
print(results)
<Table masked=True length=1>
total 
int32 
------
293473

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA