Keck Observatory Archive (KOA) Python Client - Tutorial: Access to KPF Observations

May 2023 - pykoa v1.4.5

PyKOA offers access to public raw science and calibration files acquired at the W. M. Keck Observatory Archive, and for Keck Observatory PIs, secure access to their protected data with the KOA credentials assigned to them. PyKOA also supports queries to the nexsciTAP Table Access Protocol (TAP) server https://github.com/Caltech-IPAC/nexsciTAP. The PyKOA client thus enables a rich variety of searches, including cone, box, polygon, or all-sky spatial searches; temporal searches; searches on program information; and complex searches on multiple attributes.

This Jupyter Notebook provides examples of how to discover and access data acquired with the Keck Planet Finder (KPF) instrument with the methods supported by PyKOA, and examples of how Keck PIs may access their protected data. KPF calibration data are made public on acquisition.

Installation

PyKOA can be installed from PyPI:

$ pip install --upgrade pykoa

Requirements

Requires Python 3.6 (or above), plus table read and write functions from Astropy. We have tested with Astropy 4.0.1. We recommend using the Anaconda Python distribution.

Overview of this KPF Tutorial

PyKOA supports methods for discovering and downloading public and private data archived at KOA. A dedicated method enables downloads of data discovered through queries.

KPF entered shared-risk usage at the W. M. Keck Observatory in February 2023. Calibration data are immediately made public, and are accessible anonymously through PyKOA. Science data are protected and accessible only by PIs until August 2024. This tutorial consists of two parts: it demonstrates methods for anonymously accessing calibration data, and methods of accessing protected data by PIs with their KOA credentials.

Only calibration data are public as May 2023, and the queries shown here will return only calibration files until science data are released no earlier than November 2023 (approximately).

The number of records returned by each query may differ from the number returned in this Notebook because new data are made public daily

PyKOA supports several output table formats, which are selected with the 'format' field. These formats are: IPAC (column delimited ASCII; default), VOTable, CSV, and TSV. All the examples below deliver output in IPAC format.

import sys
import io
import os
from pykoa.koa import Koa 
from astropy.table import Table,Column

---

View the help file

help(Koa)

Help on Archive in module pykoa.koa.core object:

class Archive(builtins.object)
 |  Archive(**kwargs)
 |  
 |  The 'Archive' class provides functions for accessing data stored in the 
 |  Keck Observatory Archive (KOA). Queries are performed via the nexsciTAP
 |  server.
 |  
 |  Keck PIs can use the KOA credentials assigned to them when data were 
 |  acquired (given at login) to search for their proprietary data.
 |  
 |  Example:
 |  --------
 |  
 |  import os
 |  import sys 
 |  
 |  from pykoa.koa import Koa 
 |  
 |  Koa.query_datetime ('hires',         '2018-03-16 00:00:00/2018-03-18 00:00:00',         outpath= './meta.xml',         format='ipac')
 |  
 |  Methods defined here:
 |  
 |  __init__(self, **kwargs)
 |      'init' method initializes the class with optional debugfile flag.
 |      
 |      Optional inputs:
 |      ----------------
 |      debugfile: a file path for the debug output
 |  
 |  download(self, metapath, format, outdir, **kwargs)
 |      'download' method allows download of FITS files shown in the 
 |      metadata file. The same method has the option of download the
 |      associated calibration files and level 1 files. 
 |      
 |      *** Requirement: 
 |          
 |          To download FITS files, the following two columns: 
 |          instrume and filehand must be included in the metadata file.
 |      
 |          To download associated calibrated files, the following three 
 |          columns: instrume, koaid, and filehand must be included in the 
 |          metadata file.
 |      
 |      
 |      Required input:
 |      -----
 |      metapath (string): a full path metadata table obtained from running
 |                query methods    
 |      
 |      format (string):   metadata table's format: ipac, votable, csv, or tsv.
 |      
 |      outdir (string):   the directory for depositing the returned files      
 |      
 |      At least one of the following three optional parameters:
 |      lev1file, lev1file, and calibfile should be set to 1.:
 |      
 |      
 |      Optional input:
 |      ----------------
 |      cookiepath (string): cookie file path for downloading the proprietary 
 |                           KOA data;
 |      
 |      start_row (integer): default is start_row = 0;
 |      
 |      end_row (integer): default is end_row = nrows - 1 where nrows is the 
 |                         number of rows in the metadata file;
 |      
 |      lev0file (integer): 1/0; 
 |          1: download level0 files (i.e. raw data) in 'lev0' directory;
 |          0: do not download level0 files.
 |          default is 1.
 |      
 |      calibfile (integer): 1/0;
 |          1: download calibration files;
 |          0: do not download calibration files.
 |          default is 0.
 |       
 |          Note: The calibration files are saved in the directory determined 
 |          by paramter 'calibdir' -- either in the 'calib' sub-directory, or
 |          in the same directory as the raw data files.
 |      
 |      lev1file (integer): 1/0;
 |          1: download level1 files in 'lev1' directory;
 |          0: do not download level1 files.
 |          default is 0.
 |      
 |      calibdir (integer): 1/0; 
 |          1: put calibration files in their own directory named 'calib'; 
 |          0: put the calibration files in the 'lev0' directory with other 
 |              raw, science files.
 |          default is 1.
 |  
 |  login(self, cookiepath, **kwargs)
 |      'login' method validates a user has a valid KOA account; it takes two
 |      'keyword' arguments: userid and password. If the inputs are not 
 |      provided in the keyword, the login method prompts for inputs.
 |      
 |      Required input:
 |      ---------------     
 |      cookiepath (string): a file path provided by the user to save 
 |               returned cookie (in login method) or to serve
 |               as input parameter for the subsequent koa 
 |               query and download methods.
 |      
 |      Keyword input:
 |      ---------------     
 |      userid     (string): a valid user id assigned by KOA;
 |      
 |      password   (string): a valid password in the KOA's user table; 
 |      
 |      
 |      Calling synopsis: 
 |      
 |      koa.login (cookiepath, userid='xxxx', password='xxxxxx'), or
 |      
 |      koa.login (cookiepath): and the program will prompt for 
 |                               userid and password
 |  
 |  print_data(self)
 |  
 |  query_adql(self, query, outpath, **kwargs)
 |      'query_adql' method receives a qualified ADQL query string from
 |      the user input.
 |      
 |      Required Inputs:
 |      ---------------    
 |          query (string):  an ADQL query
 |      
 |          outpath (string): the output filename of the returned metadata table
 |      
 |      Optional inputs:
 |      ----------------
 |          cookiepath (string): cookie file path for query the proprietary 
 |                               KOA data.
 |      
 |          format (string):  output format: votable, ipac, csv, or tsv 
 |                   (default: ipac)
 |      
 |          maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_criteria(self, param, outpath, **kwargs)
 |      'query_criteria' method allows searches of KOA by multiple
 |      parameters specified in a python dictionary (param).
 |      
 |      param: a dictionary containing a list of acceptable parameters:
 |      
 |          instrument (required): HIRES
 |      
 |          datetime (string): a datetime range string in the format of 
 |              datetime1/datetime2, '/' being the separator between first
 |              and second datetime valaues where datetime format is 
 |              'yyyy-mm-dd hh:mm:ss'
 |          
 |          date (string): a date range string in the format of 
 |              date1/date2, '/' being the separator between first
 |              and second date valaues where date format is 'yyyy-mm-dd'
 |          
 |          pos (string): a position string in the format of 
 |      
 |              1.  circle ra dec radius;
 |      
 |              2.  polygon ra1 dec1 ra2 dec2 ra3 dec3 ra4 dec4;
 |      
 |              3.  box ra dec width height;
 |      
 |              all RA Dec in decimal degree J2000 coordinate.
 |           
 |          target (string): target name used in the project, this will be 
 |              searched against the database -- not SIMBAD or NED.
 |      
 |      outpath (string): file path for the returned metadata table 
 |      
 |      Optional parameters:
 |      --------------------
 |      cookiepath (string): cookie file path obtained via login method, only
 |                           required for querying the proprietary KOA data.
 |      
 |      format (string): output table format -- votable, ipac, csv, or tsv;
 |          default: ipac
 |          
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_date(self, instrument, date, outpath, **kwargs)
 |      'query_date' method searches KOA by 'date_obs' range
 |      
 |      Required Inputs:
 |      ---------------    
 |      instrument (string): HIRES
 |      
 |      date (string): a date_obs string in the format of 
 |          date1/date2 where '/' separates the two date values` 
 |          of format 'yyyy-mm-dd'
 |      
 |          the following inputs are acceptable:
 |      
 |          date1/: will search data with date later than (>=) 
 |                      date1
 |          
 |          /date2: will search data with date earlier than (<=)
 |                      date2
 |      
 |          date1: will search data with date equal to (=) date1
 |      
 |      outpath (string): a full output filepath of the returned metadata 
 |          table
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          date = '2018-03-16/2018-03-18' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          date = '2018-03-16/' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          date = '/2018-03-18' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          date = '2018-03-16' 
 |      
 |      Optional inputs:
 |      ----------------
 |      cookiepath (string): cookie file path for querying the proprietary 
 |                           KOA data
 |      
 |      format (string):  Output format: votable, ipac, csv, or tsv 
 |                        (default: ipac)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_datetime(self, instrument, datetime, outpath, **kwargs)
 |      'query_datetime' method searches KOA by 'datetime' range
 |      
 |      Required Inputs:
 |      ---------------    
 |      instrument (string): HIRES
 |      
 |      datetime (string): a datetime string in the format of 
 |          datetime1/datetime2 where '/' separates the two datetime values
 |          of format 'yyyy-mm-dd hh:mm:ss'
 |      
 |          the following inputs are acceptable:
 |      
 |          datetime1/: will search data with datetime later than (>=) 
 |                      datetime1
 |          
 |          /datetime2: will search data with datetime earlier than (<=)
 |                      datetime2
 |      
 |          datetime1: will search data with datetime equal to (=) datetime1
 |      
 |      outpath (string): a full output filepath of the returned metadata 
 |          table
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16 06:10:55/2018-03-18 00:00:00' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16 06:10:55/' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '/2018-03-18 00:00:00' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16 06:10:55' 
 |      
 |      Optional inputs:
 |      ----------------
 |      cookiepath (string): cookie file path for query the proprietary 
 |                           KOA data
 |      
 |      format (string):  Output format: votable, ipac, csv, or tsv 
 |                        (default: ipac)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_moving_object(self, **kwargs)
 |      'query_moving_object' method searches KOA for moving object. 
 |      Due to the high number of input parameters, all inputs are made 
 |      to be keyword parameters.
 |      
 |      Required Inputs:
 |      ---------------    
 |      instrument (string): one of KOA instruments e.g. nirc2, nirspec, etc..;
 |      
 |      object (string): moving object to search e.g. pluto, jupiter, etc.. 
 |      
 |      startdate (string): a date string in the format of 'yyyy-mm-dd'
 |      
 |      outdir (string): a output directory for the returned output files: 
 |      
 |      outfile (string): output result list file name (a json file),
 |      
 |      Output files contains the following:
 |      
 |          --  a json formated result lists that contains the information of 
 |              the moving objects found including:
 |          
 |              input parameters,
 |              numbers of moving objects found,
 |              metadata,
 |              URL link to the file that is the same metadata in IPAC format,
 |              URL link to the PNG files showing the intersections of the KOA 
 |              observation with the moving objects' trajectory.
 |          
 |          -- the metadata files associated the moving objects.
 |      
 |      
 |      Optional inputs:
 |      ----------------
 |      enddate (string): a date string in the format of 'yyyy-mm'dd';
 |          if missing the default is the current date,
 |          
 |      naifid (integer): JPL's NAIF object ID number to more precisely 
 |          identify the moving object; this is needed when the object name
 |          such as 'pluto' which results in multiple targets.
 |      
 |      cookiepath (string): a cookie file path obtained via login method, only
 |                           required for querying the proprietary KOA data.
 |      
 |      datatype (string): "image", "spec", or "both" -- to search for either
 |          the image data, the spectra data, or both; default is "both",
 |      
 |      graphoption (integer): 0 or 1 -- indicates whether to create graphic 
 |          png files of moving object tracjectories; default is 1,
 |      
 |      orbitalinput (int): 0 or 1 indicates whether the precise orbital 
 |          input parameters are provided; default is 0,
 |      
 |      The orbital input contains the following parameters:
 |      
 |          epoch (mjd),  
 |          ec (eccentricity),
 |          om (deg): longitude of ascending node, 
 |          w (deg) : argument of perihelion,
 |          inc (deg): inclination,
 |        
 |          qr (au) (perihelion distance -- for Comet),
 |          tp (JD) (perihelion Julian date -- for Comet),
 |      
 |          or 
 |      
 |          a (au):   semi-major axis -- for Asteroid,
 |          m0 (deg):  mean anomaly -- for Asteroid
 |      
 |      
 |      where the notations are consistent with the inputs to the JPL's url
 |      for obtaining bsp file which constains the object's SPK ID:
 |      
 |          -EPOCH: epoch, 
 |          -EC:    eccentricity,
 |          -OM:    longitude of ascending node,
 |          -W:     argument of perihelion,     
 |          -INC:    inclination,
 |          -QR:    perihelion distance,
 |          -TP:    perihelion Julian date,
 |      
 |      
 |      Calling synopsis:
 |      ----------------
 |      e.g. 
 |          Koa.query_moving_object (                 instrument = 'nirspec',                 object = 'pluto',                 startdate = '1995-01-01',                 enddate = '2020-07-08',                 outdir = './nirspec_pluto',                 outfile = 'nirspec_pluto.json',                 naifid = '999',                 datatype = 'both',                 graphoption = 1)
 |  
 |  query_object(self, instrument, object, outpath, **kwargs)
 |      'query_object' method searches KOA by 'object name' 
 |      
 |      Required Inputs:
 |      ---------------    
 |      
 |      instrument: HIRES
 |      
 |      object (string): an object name resolvable by SIMBAD, NED, and
 |          ExoPlanet's name_resolve; 
 |      
 |      This method resolves the object name into coordiates to be used as the
 |      center of the circle position search with default radius of 0.5 deg.
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          object = 'WD 1145+017'
 |      
 |      Optional Input:
 |      ---------------    
 |      cookiepath (string): cookie file path for query the proprietary 
 |                           KOA data.
 |      
 |      format (string):  Output format: votable, ipac, csv, tsv (default: ipac)
 |      
 |      radius (float): search radius in deg (default = 0.5 deg)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_position(self, instrument, pos, outpath, **kwargs)
 |      'query_position' method searches KOA by 'position' 
 |      
 |      Required Inputs:
 |      ---------------    
 |      
 |      instrument (string): HIRES
 |      
 |      pos (string): a position string in the format of 
 |      
 |      1.  circle ra dec radius;
 |      
 |      2.  polygon ra1 dec1 ra2 dec2 ra3 dec3 ra4 dec4;
 |      
 |      3.  box ra dec width height;
 |      
 |      All RA Dec in decimal degree J2000 coordinate.
 |           
 |      e.g. 
 |          instrument = 'hires',
 |          pos = 'circle 230.0 45.0 0.5'
 |      
 |      outpath (string): a full filepath for the returned metadata table
 |      
 |      Optional Input:
 |      ---------------    
 |      cookiepath (string): cookie file path for querying the proprietary 
 |                           KOA data.
 |      
 |      format (string): votable, ipac, csv, tsv  (default: ipac)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)
 |  
 |  ----------------------------------------------------------------------
 |  Data and other attributes defined here:
 |  
 |  astropytbl = None
 |  
 |  content_type = ''
 |  
 |  debug = 0
 |  
 |  debugfname = './koa.debug'
 |  
 |  format = 'ipac'
 |  
 |  maxrec = -1
 |  
 |  msg = ''
 |  
 |  ncaliblist = 0
 |  
 |  ndnloaded = 0
 |  
 |  ndnloaded_calib = 0
 |  
 |  ndnloaded_lev1 = 0
 |  
 |  nlev1list = 0
 |  
 |  outdir = ''
 |  
 |  outpath = ''
 |  
 |  parampath = ''
 |  
 |  propflag = 1
 |  
 |  query = ''
 |  
 |  status = ''
 |  
 |  tap = None

Create output directory

try:
    os.mkdir('./output')
except:
    print(" Directory exists already", flush=True)

 Directory exists already

Anonymous access to public data

Query by date range

Koa.query_datetime ('kpf', \
   '2023-02-03 00:00:00/2023-02-04 23:59:59', \
   './output/datetime.kpf.tbl', \
    format='ipac')

rec = Table.read ('./output/datetime.kpf.tbl',format='ipac')
print (rec)

submitting request...
Result downloaded to file [./output/datetime.kpf.tbl]
          koaid                     ofname          ...   semid   propint
------------------------- ------------------------- ... --------- -------
KP.20230203.04645.81.fits KP.20230203.04645.81.fits ... 2023a_eng       0
KP.20230203.04707.10.fits KP.20230203.04707.10.fits ... 2023a_eng       0
KP.20230203.04768.58.fits KP.20230203.04768.58.fits ... 2023a_eng       0
KP.20230203.04829.85.fits KP.20230203.04829.85.fits ... 2023a_eng       0
KP.20230203.04891.27.fits KP.20230203.04891.27.fits ... 2023a_eng       0
KP.20230203.04952.81.fits KP.20230203.04952.81.fits ... 2023a_eng       0
KP.20230203.05013.97.fits KP.20230203.05013.97.fits ... 2023a_eng       0
KP.20230203.05075.24.fits KP.20230203.05075.24.fits ... 2023a_eng       0
KP.20230203.05136.75.fits KP.20230203.05136.75.fits ... 2023a_eng       0
KP.20230203.05198.15.fits KP.20230203.05198.15.fits ... 2023a_eng       0
                      ...                       ... ...       ...     ...
KP.20230204.71510.86.fits KP.20230204.71510.86.fits ... 2023a_eng       0
KP.20230204.71583.28.fits KP.20230204.71583.28.fits ... 2023a_eng       0
KP.20230204.71680.83.fits KP.20230204.71680.83.fits ... 2023a_eng       0
KP.20230204.71942.91.fits KP.20230204.71942.91.fits ... 2023a_eng       0
KP.20230204.72205.05.fits KP.20230204.72205.05.fits ... 2023a_eng       0
KP.20230204.74268.25.fits KP.20230204.74268.25.fits ... 2023a_eng       0
KP.20230204.74338.59.fits KP.20230204.74338.59.fits ... 2023a_eng       0
KP.20230204.74408.86.fits KP.20230204.74408.86.fits ... 2023a_eng       0
KP.20230204.74551.83.fits KP.20230204.74551.83.fits ... 2023a_eng       0
KP.20230204.74621.89.fits KP.20230204.74621.89.fits ... 2023a_eng       0
KP.20230204.74690.44.fits KP.20230204.74690.44.fits ... 2023a_eng       0
Length = 355 rows

Query by date

Koa.query_date('kpf', \
   '2023-02-03', \
   './output/date.kpf.tbl', \
    format='ipac')

rec = Table.read ('./output/date.kpf.tbl',format='ipac')
print (rec)

submitting request...
Result downloaded to file [./output/date.kpf.tbl]
          koaid                     ofname          ...   semid   propint
------------------------- ------------------------- ... --------- -------
KP.20230203.04645.81.fits KP.20230203.04645.81.fits ... 2023a_eng       0
KP.20230203.04707.10.fits KP.20230203.04707.10.fits ... 2023a_eng       0
KP.20230203.04768.58.fits KP.20230203.04768.58.fits ... 2023a_eng       0
KP.20230203.04829.85.fits KP.20230203.04829.85.fits ... 2023a_eng       0
KP.20230203.04891.27.fits KP.20230203.04891.27.fits ... 2023a_eng       0
KP.20230203.04952.81.fits KP.20230203.04952.81.fits ... 2023a_eng       0
KP.20230203.05013.97.fits KP.20230203.05013.97.fits ... 2023a_eng       0
KP.20230203.05075.24.fits KP.20230203.05075.24.fits ... 2023a_eng       0
KP.20230203.05136.75.fits KP.20230203.05136.75.fits ... 2023a_eng       0
KP.20230203.05198.15.fits KP.20230203.05198.15.fits ... 2023a_eng       0
                      ...                       ... ...       ...     ...
KP.20230203.71517.44.fits KP.20230203.71517.44.fits ... 2023a_eng       0
KP.20230203.71589.50.fits KP.20230203.71589.50.fits ... 2023a_eng       0
KP.20230203.71687.88.fits KP.20230203.71687.88.fits ... 2023a_eng       0
KP.20230203.71950.07.fits KP.20230203.71950.07.fits ... 2023a_eng       0
KP.20230203.72212.26.fits KP.20230203.72212.26.fits ... 2023a_eng       0
KP.20230203.74273.23.fits KP.20230203.74273.23.fits ... 2023a_eng       0
KP.20230203.74342.62.fits KP.20230203.74342.62.fits ... 2023a_eng       0
KP.20230203.74412.34.fits KP.20230203.74412.34.fits ... 2023a_eng       0
KP.20230203.74611.03.fits KP.20230203.74611.03.fits ... 2023a_eng       0
KP.20230203.74681.56.fits KP.20230203.74681.56.fits ... 2023a_eng       0
KP.20230203.74752.55.fits KP.20230203.74752.55.fits ... 2023a_eng       0
Length = 176 rows

Query by KPF calibration data types.

The first two examples query for all 'autocal-bias' and all 'DOME FLAT' calibration files, recorded in the 'Target' field:

param = dict()
param['instrument'] = 'kpf'
param['target'] = 'autocal-bias'

Koa.query_criteria (param, \
    './output/autocal.kpf.tbl', \
    format='ipac')

rec = Table.read ('./output/autocal.kpf.tbl', format='ascii.ipac')
print (rec)

submitting request...
Result downloaded to file [./output/autocal.kpf.tbl]
          koaid                     ofname          ...   semid   propint
------------------------- ------------------------- ... --------- -------
KP.20221202.71029.33.fits KP.20221202.71029.33.fits ... 2022b_eng       0
KP.20221202.71090.77.fits KP.20221202.71090.77.fits ... 2022b_eng       0
KP.20221202.71152.13.fits KP.20221202.71152.13.fits ... 2022b_eng       0
KP.20221202.71213.48.fits KP.20221202.71213.48.fits ... 2022b_eng       0
KP.20221202.71274.79.fits KP.20221202.71274.79.fits ... 2022b_eng       0
KP.20221202.71336.14.fits KP.20221202.71336.14.fits ... 2022b_eng       0
KP.20221202.71397.35.fits KP.20221202.71397.35.fits ... 2022b_eng       0
KP.20221202.71458.62.fits KP.20221202.71458.62.fits ... 2022b_eng       0
KP.20221202.71519.75.fits KP.20221202.71519.75.fits ... 2022b_eng       0
KP.20221202.71581.13.fits KP.20221202.71581.13.fits ... 2022b_eng       0
                      ...                       ... ...       ...     ...
KP.20230510.84334.03.fits KP.20230510.84334.03.fits ... 2023a_eng       0
KP.20230510.84395.30.fits KP.20230510.84395.30.fits ... 2023a_eng       0
KP.20230510.84456.60.fits KP.20230510.84456.60.fits ... 2023a_eng       0
KP.20230510.84517.84.fits KP.20230510.84517.84.fits ... 2023a_eng       0
KP.20230510.84579.27.fits KP.20230510.84579.27.fits ... 2023a_eng       0
KP.20230510.84640.47.fits KP.20230510.84640.47.fits ... 2023a_eng       0
KP.20230510.84701.73.fits KP.20230510.84701.73.fits ... 2023a_eng       0
KP.20230510.84763.00.fits KP.20230510.84763.00.fits ... 2023a_eng       0
KP.20230510.84824.25.fits KP.20230510.84824.25.fits ... 2023a_eng       0
KP.20230510.84885.52.fits KP.20230510.84885.52.fits ... 2023a_eng       0
KP.20230510.84946.79.fits KP.20230510.84946.79.fits ... 2023a_eng       0
Length = 3014 rows

param = dict()
param['instrument'] = 'kpf'
param['target'] = 'DOME FLATS'

Koa.query_criteria (param, \
    './output/dome.kpf.tbl', \
    format='ipac')

rec = Table.read ('./output/dome.kpf.tbl', format='ascii.ipac')
print (rec)

submitting request...
Result downloaded to file [./output/dome.kpf.tbl]
          koaid                     ofname          ...   semid   propint
------------------------- ------------------------- ... --------- -------
KP.20220920.67983.24.fits KP.20220920.67983.24.fits ... 2022b_eng       0
KP.20220920.69908.92.fits KP.20220920.69908.92.fits ... 2022b_eng       0
KP.20220920.70155.26.fits KP.20220920.70155.26.fits ... 2022b_eng       0
KP.20220920.77545.14.fits KP.20220920.77545.14.fits ... 2022b_eng       0
KP.20221022.58425.52.fits KP.20221022.58425.52.fits ... 2022b_eng       0
KP.20221022.58576.24.fits KP.20221022.58576.24.fits ... 2022b_eng       0
KP.20221022.58726.92.fits KP.20221022.58726.92.fits ... 2022b_eng       0
KP.20221022.58877.69.fits KP.20221022.58877.69.fits ... 2022b_eng       0
KP.20221022.59028.51.fits KP.20221022.59028.51.fits ... 2022b_eng       0
KP.20221022.59179.33.fits KP.20221022.59179.33.fits ... 2022b_eng       0
                      ...                       ... ...       ...     ...
KP.20230509.69309.75.fits KP.20230509.69309.75.fits ... 2023a_eng       0
KP.20230509.69372.02.fits KP.20230509.69372.02.fits ... 2023a_eng       0
KP.20230509.69434.29.fits KP.20230509.69434.29.fits ... 2023a_eng       0
KP.20230509.69528.96.fits KP.20230509.69528.96.fits ... 2023a_eng       0
KP.20230509.69591.25.fits KP.20230509.69591.25.fits ... 2023a_eng       0
KP.20230509.69653.76.fits KP.20230509.69653.76.fits ... 2023a_eng       0
KP.20230509.69716.03.fits KP.20230509.69716.03.fits ... 2023a_eng       0
KP.20230509.69778.38.fits KP.20230509.69778.38.fits ... 2023a_eng       0
KP.20230509.69871.52.fits KP.20230509.69871.52.fits ... 2023a_eng       0
KP.20230509.69993.61.fits KP.20230509.69993.61.fits ... 2023a_eng       0
KP.20230509.70115.83.fits KP.20230509.70115.83.fits ... 2023a_eng       0
Length = 1909 rows

Calibration types 'bias',''dark','arclamp', 'flatlamp' are recorded in the 'Image Type' field ('imtype'), and may be discovered as in the following examples:

Find all KPF bias files

param = dict()
param['instrument'] = 'kpf'
param['imtype'] = 'bias'

Koa.query_criteria (param, \
    './output/bias.kpf.tbl', \
    format='ipac')

rec = Table.read ('./output/bias.kpf.tbl', format='ascii.ipac')
print (rec)

submitting request...
Result downloaded to file [./output/bias.kpf.tbl]
          koaid                     ofname          ...   semid   propint
------------------------- ------------------------- ... --------- -------
KP.20221104.00386.61.fits KP.20221104.00386.61.fits ... 2022b_eng       0
KP.20221104.02613.41.fits KP.20221104.02613.41.fits ... 2022b_eng       0
KP.20221108.60905.84.fits KP.20221108.60905.84.fits ... 2022b_eng       0
KP.20221108.61020.90.fits KP.20221108.61020.90.fits ... 2022b_eng       0
KP.20221108.61137.39.fits KP.20221108.61137.39.fits ... 2022b_eng       0
KP.20221108.61268.32.fits KP.20221108.61268.32.fits ... 2022b_eng       0
KP.20221108.61400.97.fits KP.20221108.61400.97.fits ... 2022b_eng       0
KP.20221108.61591.27.fits KP.20221108.61591.27.fits ... 2022b_eng       0
KP.20221108.61769.66.fits KP.20221108.61769.66.fits ... 2022b_eng       0
KP.20221108.61972.44.fits KP.20221108.61972.44.fits ... 2022b_eng       0
                      ...                       ... ...       ...     ...
KP.20230510.84334.03.fits KP.20230510.84334.03.fits ... 2023a_eng       0
KP.20230510.84395.30.fits KP.20230510.84395.30.fits ... 2023a_eng       0
KP.20230510.84456.60.fits KP.20230510.84456.60.fits ... 2023a_eng       0
KP.20230510.84517.84.fits KP.20230510.84517.84.fits ... 2023a_eng       0
KP.20230510.84579.27.fits KP.20230510.84579.27.fits ... 2023a_eng       0
KP.20230510.84640.47.fits KP.20230510.84640.47.fits ... 2023a_eng       0
KP.20230510.84701.73.fits KP.20230510.84701.73.fits ... 2023a_eng       0
KP.20230510.84763.00.fits KP.20230510.84763.00.fits ... 2023a_eng       0
KP.20230510.84824.25.fits KP.20230510.84824.25.fits ... 2023a_eng       0
KP.20230510.84885.52.fits KP.20230510.84885.52.fits ... 2023a_eng       0
KP.20230510.84946.79.fits KP.20230510.84946.79.fits ... 2023a_eng       0
Length = 3066 rows

Find all arclamp calibrations on 20230203

param = dict()
param['instrument'] = 'kpf'
param['imtype'] = 'arclamp'
param['date'] ='2023-02-03'

Koa.query_criteria (param, \
    './output/bias.kpf.tbl', \
    format='ipac')

rec = Table.read ('./output/bias.kpf.tbl', format='ascii.ipac')
print (rec)

submitting request...
Result downloaded to file [./output/bias.kpf.tbl]
          koaid                     ofname          ...   semid   propint
------------------------- ------------------------- ... --------- -------
KP.20230203.06672.53.fits KP.20230203.06672.53.fits ... 2023a_eng       0
KP.20230203.06736.64.fits KP.20230203.06736.64.fits ... 2023a_eng       0
KP.20230203.06800.93.fits KP.20230203.06800.93.fits ... 2023a_eng       0
KP.20230203.06864.97.fits KP.20230203.06864.97.fits ... 2023a_eng       0
KP.20230203.06929.06.fits KP.20230203.06929.06.fits ... 2023a_eng       0
KP.20230203.07023.70.fits KP.20230203.07023.70.fits ... 2023a_eng       0
KP.20230203.07087.92.fits KP.20230203.07087.92.fits ... 2023a_eng       0
KP.20230203.07152.01.fits KP.20230203.07152.01.fits ... 2023a_eng       0
KP.20230203.07216.04.fits KP.20230203.07216.04.fits ... 2023a_eng       0
KP.20230203.07280.31.fits KP.20230203.07280.31.fits ... 2023a_eng       0
                      ...                       ... ...       ...     ...
KP.20230203.67773.96.fits KP.20230203.67773.96.fits ... 2023a_eng       0
KP.20230203.67841.04.fits KP.20230203.67841.04.fits ... 2023a_eng       0
KP.20230203.71445.41.fits KP.20230203.71445.41.fits ... 2023a_eng       0
KP.20230203.71517.44.fits KP.20230203.71517.44.fits ... 2023a_eng       0
KP.20230203.71589.50.fits KP.20230203.71589.50.fits ... 2023a_eng       0
KP.20230203.71687.88.fits KP.20230203.71687.88.fits ... 2023a_eng       0
KP.20230203.71950.07.fits KP.20230203.71950.07.fits ... 2023a_eng       0
KP.20230203.72212.26.fits KP.20230203.72212.26.fits ... 2023a_eng       0
KP.20230203.74611.03.fits KP.20230203.74611.03.fits ... 2023a_eng       0
KP.20230203.74681.56.fits KP.20230203.74681.56.fits ... 2023a_eng       0
KP.20230203.74752.55.fits KP.20230203.74752.55.fits ... 2023a_eng       0
Length = 95 rows

query ="select koaid, filehand, imtype from koa_kpf  where (progid='ENG')" 

Koa.query_adql (query, \
    './output/bias_info.tbl', overwrite=True, \
    format='ipac')

rec = Table.read('./output/bias_info.tbl', format='ascii.ipac')
print (rec)

submitting request...
Result downloaded to file [./output/bias_info.tbl]
          koaid           ...  imtype 
------------------------- ... --------
KP.20230107.03784.63.fits ...  Arclamp
KP.20230111.76566.47.fits ...  Arclamp
KP.20230111.76688.90.fits ...  Arclamp
KP.20230111.76811.08.fits ...  Arclamp
KP.20230111.76933.40.fits ...  Arclamp
KP.20230111.77055.69.fits ...  Arclamp
KP.20230111.78521.87.fits ...  Arclamp
KP.20230111.78644.13.fits ...  Arclamp
KP.20230111.79092.79.fits ...  Arclamp
KP.20230111.79215.11.fits ...  Arclamp
                      ... ...      ...
KP.20230509.64269.75.fits ...     None
KP.20230509.65752.46.fits ...  Arclamp
KP.20230509.65874.44.fits ...  Arclamp
KP.20230509.65996.69.fits ...  Arclamp
KP.20230509.66344.34.fits ...  Arclamp
KP.20230509.66411.12.fits ...  Arclamp
KP.20230509.66478.26.fits ...  Arclamp
KP.20230509.66545.36.fits ...  Arclamp
KP.20230509.67964.69.fits ...  Arclamp
KP.20230509.68086.72.fits ...  Arclamp
KP.20230509.69247.48.fits ... Flatlamp
Length = 28263 rows

General Metadata Queries With the IVOA Astronomical Data Query Langage (ADQL).

A query made with ADQL enables general and complex queries against the archive. If you wish to download data discovered via ADQL, explicitly include the koaid, instrument, and filehandle in the query. A number of examples are shown below.

Return a count of the total number of calibration files

Calibration files are all written to the 'ENG' (engineering) program ID

query ="select count(koaid) from koa_kpf  where (progid='ENG')" 

Koa.query_adql (query, \
    './output/count_info.tbl', overwrite=True, \
    format='ipac')

rec = Table.read('./output/count_info.tbl', format='ascii.ipac')
print (rec) 

submitting request...
Result downloaded to file [./output/count_info.tbl]
count(koaid)
------------
       28263

Query for metadata and order by UT time.

query ="select koaid, filehand, utdatetime from koa_kpf  where (progid='ENG') order by utdatetime" 

Koa.query_adql (query, \
    './output/program_info.tbl', overwrite=True, \
    format='ipac')

rec = Table.read('./output/program_info.tbl', format='ascii.ipac')
print (rec)

submitting request...
Result downloaded to file [./output/program_info.tbl]
          koaid           ...         utdatetime        
------------------------- ... --------------------------
KP.20220911.85781.49.fits ... 2022-09-11 23:49:36.300000
KP.20220911.85853.29.fits ... 2022-09-11 23:50:48.210000
KP.20220912.69869.02.fits ... 2022-09-12 19:24:23.840000
KP.20220914.70364.06.fits ... 2022-09-14 19:32:34.010000
KP.20220914.70541.19.fits ... 2022-09-14 19:35:30.980000
KP.20220914.71035.09.fits ... 2022-09-14 19:43:44.660000
KP.20220914.71380.21.fits ... 2022-09-14 19:49:29.930000
KP.20220915.58117.20.fits ... 2022-09-15 16:08:26.990000
KP.20220915.58267.08.fits ... 2022-09-15 16:10:56.930000
KP.20220915.58466.63.fits ... 2022-09-15 16:14:16.340000
                      ... ...                        ...
KP.20230511.00859.35.fits ... 2023-05-11 00:14:19.180000
KP.20230511.00923.58.fits ... 2023-05-11 00:15:23.440000
KP.20230511.01020.32.fits ... 2023-05-11 00:17:00.340000
KP.20230511.01084.46.fits ... 2023-05-11 00:18:04.090000
KP.20230511.01148.62.fits ... 2023-05-11 00:19:08.350000
KP.20230511.01212.66.fits ... 2023-05-11 00:20:12.610000
KP.20230511.01276.64.fits ... 2023-05-11 00:21:16.360000
KP.20230511.01371.84.fits ... 2023-05-11 00:22:51.730000
KP.20230511.01494.06.fits ... 2023-05-11 00:24:53.620000
KP.20230511.01616.13.fits ... 2023-05-11 00:26:56.020000
KP.20230511.01802.38.fits ... 2023-05-11 00:30:02.170000
Length = 28263 rows

The full set of KPF metadata can be found with the ADQL query below. Queries can be made against any combination of these metadata, as shown in the complex query that follows.

query="select * from TAP_SCHEMA.columns where table_name='koa_kpf'  "
Koa.query_adql (query, './output/columns.tbl', overwrite=True, format='ipac')

rec = Table.read ('./output/columns.tbl',format='ipac')
print (rec)

submitting request...
Result downloaded to file [./output/columns.tbl]
table_name column_name datatype arraysize ... principal std column_index format
---------- ----------- -------- --------- ... --------- --- ------------ ------
   koa_kpf       koaid     char        32 ...         1   0            0    32s
   koa_kpf     propint      int        -- ...         0   0            1    16d
   koa_kpf    koaimtyp     char        32 ...         1   0            2    32s
   koa_kpf    adctrack     char        32 ...         0   0            3    32s
   koa_kpf     agitsta     char        32 ...         0   0            4    32s
   koa_kpf     airmass   double        -- ...         0   0            5  18.5f
   koa_kpf      ampsec     char        32 ...         0   0            6    32s
   koa_kpf    autactiv     char        16 ...         0   0            7    16s
   koa_kpf     autfwhm   double        -- ...         0   0            8  18.5f
   koa_kpf    autxcent   double        -- ...         0   0            9  18.5f
       ...         ...      ...       ... ...       ... ...          ...    ...
   koa_kpf    timshttr     char        32 ...         0   0          232    32s
   koa_kpf    tracking     char        16 ...         0   0          233    16s
   koa_kpf    trigtarg     char        32 ...         0   0          234    32s
   koa_kpf    tubeflex   double        -- ...         0   0          235  18.5f
   koa_kpf    tubetemp   double        -- ...         0   0          236  18.5f
   koa_kpf       uauon     char        32 ...         0   0          237    32s
   koa_kpf      uauton      int        -- ...         0   0          238    16d
   koa_kpf      udayon     char        32 ...         0   0          239    32s
   koa_kpf     udayton   double        -- ...         0   0          240  18.5f
   koa_kpf      uneday     char        32 ...         0   0          241    32s
   koa_kpf       unegd     char        32 ...         0   0          242    32s
Length = 274 rows

query = "select koaid, cameras, ofname, instrume, currinst, object, targname, koaimtyp, frameno, ra, dec, \
        to_char(date_obs,'YYYY-MM-DD') as date_obs, ut, elaptime, waveblue, wavered, equinox, red, green,\
        ca_hk, progid, proginst, progpi, progtitl, filehand, lower(semid) \
        as semid, propint from koa_kpf where ((targname like 'DOME FLATS')) order by utdatetime"

Koa.query_adql (query, \
    './output/adql.kpf.tbl', \
    format='ipac')


rec = Table.read('./output/adql.kpf.tbl', format='ascii.ipac')
print (rec)

submitting request...
Result downloaded to file [./output/adql.kpf.tbl]
          koaid           cameras ...   semid   propint
------------------------- ------- ... --------- -------
KP.20220920.67983.24.fits       2 ... 2022b_eng       0
KP.20220920.69908.92.fits       2 ... 2022b_eng       0
KP.20220920.70155.26.fits       2 ... 2022b_eng       0
KP.20220920.77545.14.fits       2 ... 2022b_eng       0
KP.20221022.58425.52.fits       3 ... 2022b_eng       0
KP.20221022.58576.24.fits       3 ... 2022b_eng       0
KP.20221022.58726.92.fits       3 ... 2022b_eng       0
KP.20221022.58877.69.fits       3 ... 2022b_eng       0
KP.20221022.59028.51.fits       3 ... 2022b_eng       0
KP.20221022.59179.33.fits       3 ... 2022b_eng       0
                      ...     ... ...       ...     ...
KP.20230509.69309.75.fits       3 ... 2023a_eng       0
KP.20230509.69372.02.fits       3 ... 2023a_eng       0
KP.20230509.69434.29.fits       3 ... 2023a_eng       0
KP.20230509.69528.96.fits       3 ... 2023a_eng       0
KP.20230509.69591.25.fits       3 ... 2023a_eng       0
KP.20230509.69653.76.fits       3 ... 2023a_eng       0
KP.20230509.69716.03.fits       3 ... 2023a_eng       0
KP.20230509.69778.38.fits       3 ... 2023a_eng       0
KP.20230509.69871.52.fits       3 ... 2023a_eng       0
KP.20230509.69993.61.fits       3 ... 2023a_eng       0
KP.20230509.70115.83.fits       3 ... 2023a_eng       0
Length = 1909 rows

Download data

Download a subset of results from the "query by date range" example above

KPF creates files bigger than 100 MB, and so downloads can be slow. The example below shows how to download two files as a simple example. Remove the 'start_row' and 'end_row' fields to download all files. Currently, this example will not return any associated calibration files because the science data are still protected.

Please note that if a file is already stored in your directory, it won't be overwritten

Koa.download ('./output/datetime.kpf.tbl',  
    'ipac', \
    'dnload_dir_kpf', \
    calibfile=1, \
    start_row=152, \
    end_row=153 )

Start downloading 2 koaid data you requested;
please check your outdir: dnload_dir_kpf for  progress ....
No associated calibration list for KP.20230203.70579.69.fits
No associated calibration list for KP.20230203.70642.20.fits

A total of 0 new lev0 FITS files downloaded.
0 new calibration list downloaded.
0 new calibration FITS files downloaded.

Access to KPF Data for Keck PIs

Keck PIs can login with their KOA credentials, assigned when the data were acquired, and access their protected data. While logged in, they can access all public data as well. Koa.login creates the cookie file: the userid below is a dummy ID to illustrate the login.

Koa.login ('./tapcookie.txt')

The following are example queries and do not return data.

To access your protected data, simply include include in your query the 'cookiepath' returned by Koa.login, as in these examples:

Koa.query_datetime ('kpf', \
    '2015-09-01 00:00:00/2015-09-30 23:59:59', \
    './output/kpf_daterange.tbl', overwrite=True, format='ipac', \
    cookiepath='./tapcookie.txt')

rec = Table.read ('./output/kpf_daterange.tbl',format='ipac')
print (rec)
                    

Koa.query_position ('kpf', \
    'circle 90.0 45.0 1.0', \
    './output/pos.kpf.tbl', \
    format='ipac' )


rec = Table.read ('./output/pos.kpf.tbl',format='ipac')
print (rec)

---

Visit KOA at https://koa.ipac.caltech.edu

Please acknowledge the use of KOA by including this text in your publications: "This research has made use of the Keck Observatory Archive (KOA), which is operated by the W. M. Keck Observatory and the NASA Exoplanet Science Institute (NExScI), under contract with the National Aeronautics and Space Administration."

Please also acknowledge the PI(s) of datasets that have been obtained through KOA, and please contact the KOA Help Desk if you publish archival data:

KOA Help Desk

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk