Access Public KPF Data From KOA with the PyVO Client In Asynchronous Mode

The nexsciTAP server

The NASA Exoplanet Science Institute (NExScI) (visit https://nexsci.caltech.edu) has developed a Python-based server to implement an API that complies with the Virtual Observatory (VO) Table Access Protocol(TAP) version 1.1 (September 2019) (http://ivoa.net/documents/TAP/20190927/index.html), a standard recommended by the International Virtual Observatory alliance (IVOA) (http://ivoa.net).

The TAP API enables a rich variety of searches against tabular data, includung cone, box or all-sky searches, temporal searches, combinations of spatial searches and temporal searches, searches against instrumental attributes and program attributes.

This tutorial supports demonstrates how to use the PyVO client to perform asynchronous TAP-based queries for public raw science and calibation data acquired with the Keck Planet Finder (KPF) these data are hosted at the Keck Observatory Archive (KOA; https://koa.ipac.caltech.edu).

KPF entered shared-risk usage at the W. M. Keck Observatory in February 2023. Calibration data are immediately made public, and are accessible anonymously through PyKOA. Science data are protected and accessible only by PIs until August 202

Only calibration data are public as May 2023, and the queries shown here will return only calibration files until science data are released no earlier than November 2023 (approximately).

The number of records returned by each query may differ from the number returned in this Notebook because new data are made public daily

Requirements

This tutorial uses PyVO version 1.1.1, and can be installed from PyPI:

$ pip install --upgrade PyVO

The tutorial requires Python 3.6 (or later), plus the table read and write functions from Astropy. We have tested with Astropy 4.0.1, but any version should work. We recommend using the Ananconda Python distribution.

A note on output records

The number of records returned here may differ from those returned here because new data are released daily.

Version 1.0 (June 2023)

Set up

from pyvo.dal import tap 
koa = tap.TAPService("https://koa.ipac.caltech.edu/TAP")

import sys
import os
import time

Query by date

sql = "select koaid, filehand, imtype from koa_kpf where koaid like '%20230203%' "


job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results_date = job.fetch_result()
print(results_date)

# write output file
table=results_date.to_table()
table.write ('./table_date.vot',format='votable',overwrite=True)

url:     https://koa.ipac.caltech.edu//TAP/async/tap__8bc0659
job_id:  tap__8bc0659
phase:   PENDING
phase:   COMPLETED
<Table length=176>
          koaid           ...  imtype 
          object          ...  object 
------------------------- ... --------
KP.20230203.04645.81.fits ...     Bias
KP.20230203.04707.10.fits ...     Bias
KP.20230203.04768.58.fits ...     Bias
KP.20230203.04829.85.fits ...     Bias
KP.20230203.04891.27.fits ...     Bias
KP.20230203.04952.81.fits ...     Bias
KP.20230203.05013.97.fits ...     Bias
KP.20230203.05075.24.fits ...     Bias
KP.20230203.05136.75.fits ...     Bias
KP.20230203.05198.15.fits ...     Bias
                      ... ...      ...
KP.20230203.71589.50.fits ...  Arclamp
KP.20230203.71687.88.fits ...  Arclamp
KP.20230203.71950.07.fits ...  Arclamp
KP.20230203.72212.26.fits ...  Arclamp
KP.20230203.74273.23.fits ... Flatlamp
KP.20230203.74342.62.fits ... Flatlamp
KP.20230203.74412.34.fits ... Flatlamp
KP.20230203.74611.03.fits ...  Arclamp
KP.20230203.74681.56.fits ...  Arclamp
KP.20230203.74752.55.fits ...  Arclamp

PyVO by default returns results in VOTable format. You can also write to IPAC ASCII and CSV format for storage.

# IPAC ASCII
table_date=results_date.to_table()
table.write ('./table_date.tbl',format='ascii.ipac', \
             overwrite=True)

# CSV
table_csv=results_date.to_table()
table.write ('./table_csv.csv',format='csv',overwrite=True)

Select all keywords for data acquired on UT Date

sql = "select * from koa_kpf where koaid like '%20230203%' "
job = koa.submit_job(sql)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results_key= job.fetch_result()

print(results_key)

# write output file in VTtable format
table_selectall=results_key.to_table()
table_selectall.write ('./table_select_key.vot', \
                        format='votable',overwrite=True)

url:     https://koa.ipac.caltech.edu//TAP/async/tap_cejy964x
job_id:  tap_cejy964x
phase:   PENDING
phase:   COMPLETED
<Table length=176>
          koaid           propint koaimtyp ...     y        z      spt_ind 
                                           ...                             
          object           int32   object  ...  float64  float64    int32  
------------------------- ------- -------- ... --------- -------- ---------
KP.20230203.04645.81.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.04707.10.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.04768.58.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.04829.85.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.04891.27.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.04952.81.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.05013.97.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.05075.24.fits       0     bias ... -0.358368 0.000675 200020012
KP.20230203.05136.75.fits       0     bias ... -0.358368 0.000675 200020012
                      ...     ...      ... ...       ...      ...       ...
KP.20230203.71589.50.fits       0  arclamp ... -0.358368 0.000675 200020012
KP.20230203.71687.88.fits       0  arclamp ... -0.358368 0.000675 200020012
KP.20230203.71950.07.fits       0  arclamp ... -0.358368 0.000675 200020012
KP.20230203.72212.26.fits       0  arclamp ... -0.358368 0.000675 200020012
KP.20230203.74273.23.fits       0 flatlamp ... -0.358368 0.000675 200020012
KP.20230203.74342.62.fits       0 flatlamp ... -0.358368 0.000675 200020012
KP.20230203.74412.34.fits       0 flatlamp ... -0.358368 0.000675 200020012
KP.20230203.74611.03.fits       0  arclamp ... -0.358368 0.000675 200020012
KP.20230203.74681.56.fits       0  arclamp ... -0.358368 0.000675 200020012
KP.20230203.74752.55.fits       0  arclamp ... -0.358368 0.000675 200020012

WARNING: W50: ?:?:?: W50: Invalid unit string 'pixels' [astropy.io.votable.tree]
WARNING: W50: ?:?:?: W50: Invalid unit string 'arcseconds' [astropy.io.votable.tree]
WARNING: W50: ?:?:?: W50: Invalid unit string 'degrees' [astropy.io.votable.tree]
WARNING: W50: ?:?:?: W50: Invalid unit string 'e/Dn' [astropy.io.votable.tree]
WARNING: W50: ?:?:?: W50: Invalid unit string 'arcseconds/s' [astropy.io.votable.tree]
WARNING: W50: ?:?:?: W50: Invalid unit string 'Celsius' [astropy.io.votable.tree]

Select columns acquired on UT Date

sql = "select koaid, filehand, imtype from koa_kpf where koaid like '%20230203%'"
job = koa.submit_job(sql)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results = job.fetch_result( )

print(results)

url:     https://koa.ipac.caltech.edu//TAP/async/tap_y_sfojwo
job_id:  tap_y_sfojwo
phase:   PENDING
phase:   COMPLETED
<Table length=176>
          koaid           ...  imtype 
          object          ...  object 
------------------------- ... --------
KP.20230203.04645.81.fits ...     Bias
KP.20230203.04707.10.fits ...     Bias
KP.20230203.04768.58.fits ...     Bias
KP.20230203.04829.85.fits ...     Bias
KP.20230203.04891.27.fits ...     Bias
KP.20230203.04952.81.fits ...     Bias
KP.20230203.05013.97.fits ...     Bias
KP.20230203.05075.24.fits ...     Bias
KP.20230203.05136.75.fits ...     Bias
KP.20230203.05198.15.fits ...     Bias
                      ... ...      ...
KP.20230203.71589.50.fits ...  Arclamp
KP.20230203.71687.88.fits ...  Arclamp
KP.20230203.71950.07.fits ...  Arclamp
KP.20230203.72212.26.fits ...  Arclamp
KP.20230203.74273.23.fits ... Flatlamp
KP.20230203.74342.62.fits ... Flatlamp
KP.20230203.74412.34.fits ... Flatlamp
KP.20230203.74611.03.fits ...  Arclamp
KP.20230203.74681.56.fits ...  Arclamp
KP.20230203.74752.55.fits ...  Arclamp

Select maximum number of records to return from a query

The return message from PyVO can be disregarded

sql = "select koaid, filehand, frameno from koa_kpf where koaid like '%20230203%'"
job = koa.submit_job(sql,maxrec=5)

print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results = job.fetch_result( )

print(results)

url:     https://koa.ipac.caltech.edu//TAP/async/tap_00_e2fbu
job_id:  tap_00_e2fbu
phase:   PENDING
phase:   COMPLETED
<Table length=5>
          koaid           ... frameno
          object          ...  int32 
------------------------- ... -------
KP.20230203.04645.81.fits ...   29404
KP.20230203.04707.10.fits ...   29405
KP.20230203.04768.58.fits ...   29406
KP.20230203.04829.85.fits ...   29407
KP.20230203.04891.27.fits ...   29408

Select records by date range

sql="select koaid, filehand, imtype from koa_kpf  \
     where (utdatetime >= to_date('2023-02-03 00:00:00', \
    'yyyy-mm-dd HH24:MI:SS') and \
     utdatetime <= to_date('2023-02-04 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))"


job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsTIME= job.fetch_result( )

print(resultsTIME)

## write to output file
table_progID_TIME=resultsTIME.to_table()
table.write ('./table_progID_ENG.vot',format='votable',overwrite=True)

url:     https://koa.ipac.caltech.edu//TAP/async/tap_ej3kfmnv
job_id:  tap_ej3kfmnv
phase:   PENDING
phase:   COMPLETED
<Table length=355>
          koaid           ...  imtype 
          object          ...  object 
------------------------- ... --------
KP.20230203.04645.81.fits ...     Bias
KP.20230203.04707.10.fits ...     Bias
KP.20230203.04768.58.fits ...     Bias
KP.20230203.04829.85.fits ...     Bias
KP.20230203.04891.27.fits ...     Bias
KP.20230203.04952.81.fits ...     Bias
KP.20230203.05013.97.fits ...     Bias
KP.20230203.05075.24.fits ...     Bias
KP.20230203.05136.75.fits ...     Bias
KP.20230203.05198.15.fits ...     Bias
                      ... ...      ...
KP.20230204.71583.28.fits ...  Arclamp
KP.20230204.71680.83.fits ...  Arclamp
KP.20230204.71942.91.fits ...  Arclamp
KP.20230204.72205.05.fits ...  Arclamp
KP.20230204.74268.25.fits ... Flatlamp
KP.20230204.74338.59.fits ... Flatlamp
KP.20230204.74408.86.fits ...  Arclamp
KP.20230204.74551.83.fits ...  Arclamp
KP.20230204.74621.89.fits ...  Arclamp
KP.20230204.74690.44.fits ...  Arclamp

Query by program ID

sql=("select koaid, filehand, progid, imtype, ha, az from \
      koa_kpf where (progid = 'ENG')")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

resultsPROGID= job.fetch_result( )

print(resultsPROGID)

## write to output file
table_progID_C307=resultsPROGID.to_table()
table.write ('./table_progID_ENG.vot',format='votable',overwrite=True)

url:     https://koa.ipac.caltech.edu//TAP/async/tap_b3l3cgxi
job_id:  tap_b3l3cgxi
phase:   PENDING
phase:   COMPLETED
<Table length=36741>
          koaid           ...    az  
                          ... degrees
          object          ... float64
------------------------- ... -------
KP.20230111.77177.87.fits ...  -74.22
KP.20230111.77910.84.fits ... -124.35
KP.20230111.76444.02.fits ...   -20.0
KP.20230111.78033.13.fits ... -124.35
KP.20230111.77544.02.fits ... -124.35
KP.20230111.77666.50.fits ... -124.35
KP.20230111.77788.75.fits ... -124.35
KP.20230111.78155.23.fits ... -124.35
KP.20230111.77300.07.fits ... -119.51
                      ... ...     ...
KP.20230619.63712.69.fits ... -112.66
KP.20230619.63784.66.fits ... -112.66
KP.20230619.63928.29.fits ... -112.66
KP.20230618.84870.52.fits ...   233.0
KP.20230618.81560.93.fits ...   233.0
KP.20230618.10511.01.fits ...   233.0
KP.20230619.02607.83.fits ...   233.0
KP.20230618.09865.08.fits ...   233.0
KP.20230618.06010.73.fits ...   233.0
KP.20230618.85422.24.fits ...   233.0

Count number of records in date range

sql=("select count(*) from koa_kpf where \
     (utdatetime >= to_date('2023-02-03 00:00:00', 'yyyy-mm-dd HH24:MI:SS') and  \
     utdatetime <= to_date('2023-02-04 23:59:59', 'yyyy-mm-dd HH24:MI:SS'))")

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
    time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)

url:     https://koa.ipac.caltech.edu//TAP/async/tap_3gvvsoyw
job_id:  tap_3gvvsoyw
phase:   PENDING
phase:   COMPLETED
<Table length=1>
count(*)
 int32  
--------
     355

Count total number of records

sql="select count(*) as total from koa_kpf"

job = koa.submit_job(sql)
print('url:    ', job.url)
print('job_id: ', job.job_id)
print('phase:  ', job.phase)

job.run()

while job.phase == 'EXECUTING':
      time.sleep(2)
    
print('phase:  ', job.phase)

results_count= job.fetch_result( )

print(results_count)

url:     https://koa.ipac.caltech.edu//TAP/async/tap_c1zqwvai
job_id:  tap_c1zqwvai
phase:   PENDING
phase:   COMPLETED
<Table length=1>
total
int32
-----
36766

---

Visit KOA at https://koa.ipac.caltech.edu.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk at https://koa.ipac.caltech.edu/cgi-bin/Helpdesk/nph-genTicketForm?projname=KOA