Keck Observatory Archive (KOA) Python Client - Tutorial: Access to ESI Raw Data

May 2021 - pykoa v1.4.5

PyKOA offers access to public raw science and calibration files acquired at the W. M. Keck Observatory Archive, and for Keck Observatory PIs, secure access to their protected data with the KOA credentials assigned to them. PyKOA also supports queries to the nexsciTAP Table Access Protocol (TAP) server https://github.com/Caltech-IPAC/nexsciTAP. The PyKOA client thus enables a rich variety of searches, including cone, box, polygon, or all-sky spatial searches; temporal searches; searches on program information; and complex searches on multiple attributes.

This Jupyter Notebook provides examples of how to discover and access raw science and calibration data acquired with the DEep Imaging Multi-Object Spectrograph (ESI) instrument with the methods supported by PyKOA, and examples of how Keck PIs may access their protected data.

Installation

PyKOA can be installed from PyPI:

$ pip install --upgrade pykoa

Requirements

Requires Python 3.6 (or above), plus table read and write functions from Astropy. We have tested with Astropy 4.0.1. We recommend using the Anaconda Python distribution.

Overview of this Tutorial

PyKOA supports methods for discovering and downloading public and private data archived at KOA. It writes the output metadata data to an output file, in the IPAC ASCII, VOTable, CSV or TSV data formats. A dedicated method enables downloads of data discovered through queries.

This tutorial illustrates methods for discovering and accessing public and private raw science and calibration files for ESI:

  • Query by date or date range (with examples for each file format).
  • Query by position.
  • Query by object.
  • Query by program information.
  • Query by by combinations of the above search criteria.
  • Download raw science and associated calibration files, or a subset of data, corresponding to a collection of metadata.
  • General, complex metadata queries in the IVOA Astronomical Data Query Langage (ADQL).
  • Queries for protected data (available to Keck PIs only).

The number of records returned by each query may differ from the number returned in this Notebook because new data are made public daily.

In [1]:
import sys
import io
import os
from pykoa.koa import Koa 
from astropy.table import Table,Column

View the help file

In [3]:
help(Koa)
Help on Archive in module pykoa.koa.core object:

class Archive(builtins.object)
 |  The 'Archive' class provides functions for accessing data stored in the 
 |  Keck Observatory Archive (KOA). Queries are performed via the nexsciTAP
 |  server.
 |  
 |  Keck PIs can use the KOA credentials assigned to them when data were 
 |  acquired (given at login) to search for their proprietary data.
 |  
 |  Example:
 |  --------
 |  
 |  import os
 |  import sys 
 |  
 |  from pykoa.koa import Koa 
 |  
 |  Koa.query_datetime ('hires', '2018-03-16 00:00:00/2018-03-18 00:00:00', outpath= './meta.xml', format='ipac')
 |  
 |  Methods defined here:
 |  
 |  __init__(self, **kwargs)
 |      'init' method initializes the class with optional debugfile flag.
 |      
 |      Optional inputs:
 |      ----------------
 |      debugfile: a file path for the debug output
 |  
 |  download(self, metapath, format, outdir, **kwargs)
 |      'download' method allows download of FITS files (and/or) 
 |      associated calibration files shown in their metadata file.
 |      
 |      *** Requirement: To download files, the following three columns: 
 |          instrume, koaid, and filehand must be included in the input
 |          metadata file.
 |      
 |      Required input:
 |      -----
 |      metapath (string): a full path metadata table obtained from running
 |                query methods    
 |      
 |      format (string):   metadata table's format: ipac, votable, csv, or tsv.
 |      
 |      outdir (string):   the directory for depositing the returned files      
 |      
 |      
 |      Optional input:
 |      ----------------
 |      cookiepath (string): cookie file path for downloading the proprietary 
 |                           KOA data;
 |      
 |      start_row (integer): default is start_row = 0;
 |      
 |      end_row (integer): default is end_row = nrows - 1 where nrows is the 
 |                         number of rows in the metadata file;
 |      
 |      calibfile (integer): whether to download the associated calibration 
 |          files (0: do not download; 1: download);
 |          default is 0.
 |  
 |  login(self, cookiepath, **kwargs)
 |      'login' method validates a user has a valid KOA account; it takes two
 |      'keyword' arguments: userid and password. If the inputs are not 
 |      provided in the keyword, the login method prompts for inputs.
 |      
 |      Required input:
 |      ---------------     
 |      cookiepath (string): a file path provided by the user to save 
 |               returned cookie (in login method) or to serve
 |               as input parameter for the subsequent koa 
 |               query and download methods.
 |      
 |      Keyword input:
 |      ---------------     
 |      userid     (string): a valid user id assigned by KOA;
 |      
 |      password   (string): a valid password in the KOA's user table; 
 |      
 |      
 |      Calling synopsis: 
 |      
 |      koa.login (cookiepath, userid='xxxx', password='xxxxxx'), or
 |      
 |      koa.login (cookiepath): and the program will prompt for 
 |                               userid and password
 |  
 |  print_data(self)
 |  
 |  query_adql(self, query, outpath, **kwargs)
 |      'query_adql' method receives a qualified ADQL query string from
 |      the user input.
 |      
 |      Required Inputs:
 |      ---------------    
 |          query (string):  an ADQL query
 |      
 |          outpath (string): the output filename of the returned metadata table
 |      
 |      Optional inputs:
 |      ----------------
 |          cookiepath (string): cookie file path for query the proprietary 
 |                               KOA data.
 |      
 |          format (string):  output format: votable, ipac, csv, or tsv 
 |                   (default: ipac)
 |      
 |          maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_criteria(self, param, outpath, **kwargs)
 |      'query_criteria' method allows searches of KOA by multiple
 |      parameters specified in a python dictionary (param).
 |      
 |      param: a dictionary containing a list of acceptable parameters:
 |      
 |          instrument (required): HIRES
 |      
 |          datetime (string): a datetime range string in the format of 
 |              datetime1/datetime2, '/' being the separator between first
 |              and second datetime valaues where datetime format is 
 |              'yyyy-mm-dd hh:mm:ss'
 |          
 |          date (string): a date range string in the format of 
 |              date1/date2, '/' being the separator between first
 |              and second date valaues where date format is 'yyyy-mm-dd'
 |          
 |          pos (string): a position string in the format of 
 |      
 |              1.  circle ra dec radius;
 |      
 |              2.  polygon ra1 dec1 ra2 dec2 ra3 dec3 ra4 dec4;
 |      
 |              3.  box ra dec width height;
 |      
 |              all RA Dec in J2000 coordinate.
 |           
 |          target (string): target name used in the project, this will be 
 |              searched against the database -- not SIMBAD or NED.
 |      
 |      outpath (string): file path for the returned metadata table 
 |      
 |      Optional parameters:
 |      --------------------
 |      cookiepath (string): cookie file path obtained via login method, only
 |                           required for querying the proprietary KOA data.
 |      
 |      format (string): output table format -- votable, ipac, csv, or tsv;
 |          default: ipac
 |          
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_date(self, instrument, date, outpath, **kwargs)
 |      'query_date' method searches KOA by 'date_obs' range
 |      
 |      Required Inputs:
 |      ---------------    
 |      instrument (string): HIRES
 |      
 |      date (string): a date_obs string in the format of 
 |          date1/date2 where '/' separates the two date values` 
 |          of format 'yyyy-mm-dd'
 |      
 |          the following inputs are acceptable:
 |      
 |          date1/: will search data with date later than (>=) 
 |                      date1
 |          
 |          /date2: will search data with date earlier than (<=)
 |                      date2
 |      
 |          date1: will search data with date equal to (=) date1
 |      
 |      outpath (string): a full output filepath of the returned metadata 
 |          table
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16/2018-03-18' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16/' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '/2018-03-18' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16' 
 |      
 |      Optional inputs:
 |      ----------------
 |      cookiepath (string): cookie file path for querying the proprietary 
 |                           KOA data
 |      
 |      format (string):  Output format: votable, ipac, csv, or tsv 
 |                        (default: ipac)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_datetime(self, instrument, datetime, outpath, **kwargs)
 |      'query_datetime' method searches KOA by 'datetime' range
 |      
 |      Required Inputs:
 |      ---------------    
 |      instrument (string): HIRES
 |      
 |      datetime (string): a datetime string in the format of 
 |          datetime1/datetime2 where '/' separates the two datetime values
 |          of format 'yyyy-mm-dd hh:mm:ss'
 |      
 |          the following inputs are acceptable:
 |      
 |          datetime1/: will search data with datetime later than (>=) 
 |                      datetime1
 |          
 |          /datetime2: will search data with datetime earlier than (<=)
 |                      datetime2
 |      
 |          datetime1: will search data with datetime equal to (=) datetime1
 |      
 |      outpath (string): a full output filepath of the returned metadata 
 |          table
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16 06:10:55/2018-03-18 00:00:00' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16 06:10:55/' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '/2018-03-18 00:00:00' 
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          datetime = '2018-03-16 06:10:55' 
 |      
 |      Optional inputs:
 |      ----------------
 |      cookiepath (string): cookie file path for query the proprietary 
 |                           KOA data
 |      
 |      format (string):  Output format: votable, ipac, csv, or tsv 
 |                        (default: ipac)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_object(self, instrument, object, outpath, **kwargs)
 |      'query_object' method searches KOA by 'object name' 
 |      
 |      Required Inputs:
 |      ---------------    
 |      
 |      instrument: HIRES
 |      
 |      object (string): an object name resolvable by SIMBAD, NED, and
 |          ExoPlanet's name_resolve; 
 |      
 |      This method resolves the object name into coordiates to be used as the
 |      center of the circle position search with default radius of 0.5 deg.
 |      
 |      e.g. 
 |          instrument = 'hires',
 |          object = 'WD 1145+017'
 |      
 |      Optional Input:
 |      ---------------    
 |      cookiepath (string): cookie file path for query the proprietary 
 |                           KOA data.
 |      
 |      format (string):  Output format: votable, ipac, csv, tsv (default: ipac)
 |      
 |      radius (float): search radius in deg (default = 0.5 deg)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  query_position(self, instrument, pos, outpath, **kwargs)
 |      'query_position' method searches KOA by 'position' 
 |      
 |      Required Inputs:
 |      ---------------    
 |      
 |      instrument (string): HIRES
 |      
 |      pos (string): a position string in the format of 
 |      
 |      1.  circle ra dec radius;
 |      
 |      2.  polygon ra1 dec1 ra2 dec2 ra3 dec3 ra4 dec4;
 |      
 |      3.  box ra dec width height;
 |      
 |      All RA Dec in J2000 coordinate.
 |           
 |      e.g. 
 |          instrument = 'hires',
 |          pos = 'circle 230.0 45.0 0.5'
 |      
 |      outpath (string): a full filepath for the returned metadata table
 |      
 |      Optional Input:
 |      ---------------    
 |      cookiepath (string): cookie file path for querying the proprietary 
 |                           KOA data.
 |      
 |      format (string): votable, ipac, csv, tsv  (default: ipac)
 |      
 |      maxrec (integer):  maximum records to be returned 
 |               default: -1 or not specified will return all requested records
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)
 |  
 |  ----------------------------------------------------------------------
 |  Data and other attributes defined here:
 |  
 |  astropytbl = None
 |  
 |  content_type = ''
 |  
 |  debug = 0
 |  
 |  debugfname = './koa.debug'
 |  
 |  format = 'ipac'
 |  
 |  maxrec = -1
 |  
 |  msg = ''
 |  
 |  ncaliblist = 0
 |  
 |  ndnloaded = 0
 |  
 |  ndnloaded_calib = 0
 |  
 |  outdir = ''
 |  
 |  outpath = ''
 |  
 |  parampath = ''
 |  
 |  query = ''
 |  
 |  status = ''
 |  
 |  tap = None

Create output directory

In [3]:
try:
    os.mkdir('./output')
except:
    print(" Directory exists already", flush=True)

Anonymous access

Query by date

In [4]:
Koa.query_date ('esi', \
    '2015-09-02', \
    './output/ESI_date.tbl', overwrite=True, format='ipac' )

rec = Table.read ('./output/ESI_date.tbl',format='ipac')
print (rec)
submitting request...
Result downloaded to file [./output/ESI_date.tbl]
        koaid          instrume targname ...   semid   propint
                                         ...                  
---------------------- -------- -------- ... --------- -------
ES.20150902.10478.fits      ESI  unknown ... 2015b_eng      18
ES.20150902.10550.fits      ESI  unknown ... 2015b_eng      18
ES.20150902.45958.fits      ESI  unknown ... 2015b_eng      18
ES.20150902.46025.fits      ESI  unknown ... 2015b_eng      18
ES.20150902.46101.fits      ESI  unknown ... 2015b_eng      18
ES.20150902.46168.fits      ESI  unknown ... 2015b_eng      18
ES.20150902.46410.fits      ESI  unknown ... 2015b_eng      18
ES.20150902.85368.fits      ESI       -- ... 2015b_eng      18
ES.20150902.85427.fits      ESI       -- ... 2015b_eng      18
ES.20150902.85487.fits      ESI       -- ... 2015b_eng      18
ES.20150902.85546.fits      ESI       -- ... 2015b_eng      18
ES.20150902.85605.fits      ESI       -- ... 2015b_eng      18
ES.20150902.85664.fits      ESI       -- ... 2015b_eng      18

Query by date range

Query by date range; output in IPAC ASCII format (default)

In [4]:
Koa.query_datetime ('esi', \
    '2015-09-01 00:00:00/2015-09-30 23:59:59', \
    './output/ESI_datetime.tbl', overwrite=True, format='ipac' )

rec = Table.read ('./output/ESI_datetime.tbl',format='ipac')
print (rec)
submitting request...
Result downloaded to file [./output/ESI_datetime.tbl]
        koaid          instrume   targname  ...    semid    propint
                                            ...                    
---------------------- -------- ----------- ... ----------- -------
ES.20150902.10478.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.10550.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.45958.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46025.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46101.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46168.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46410.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.85368.fits      ESI          -- ...   2015b_eng      18
ES.20150902.85427.fits      ESI          -- ...   2015b_eng      18
ES.20150902.85487.fits      ESI          -- ...   2015b_eng      18
                   ...      ...         ... ...         ...     ...
ES.20150911.26693.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.35455.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.42470.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.44332.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.46194.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.48056.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.50078.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.51339.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.52604.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.53867.fits      ESI  J0246-0825 ... 2015b_u077e      18
Length = 225 rows

Query by date range; output in VOTable format

In [5]:
Koa.query_datetime ('esi', \
    '2015-09-01 00:00:00/2015-09-30 23:59:59', \
    './output/ESI_datetime.vot', overwrite=True, format='votable')

rec = Table.read ('./output/ESI_datetime.vot',format='votable')
print (rec)
submitting request...
Result downloaded to file [./output/ESI_datetime.vot]
        koaid          instrume   targname  ...    semid    propint
---------------------- -------- ----------- ... ----------- -------
ES.20150902.10478.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.10550.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.45958.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46025.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46101.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46168.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46410.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.85368.fits      ESI             ...   2015b_eng      18
ES.20150902.85427.fits      ESI             ...   2015b_eng      18
ES.20150902.85487.fits      ESI             ...   2015b_eng      18
                   ...      ...         ... ...         ...     ...
ES.20150911.24831.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.26693.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.35455.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.42470.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.44332.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.46194.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.48056.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.50078.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.51339.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.52604.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.53867.fits      ESI  J0246-0825 ... 2015b_u077e      18
Length = 225 rows

Query by date range; output in CSV format

In [6]:
Koa.query_datetime ('esi', \
    '2015-09-01 00:00:00/2015-09-30 23:59:59', \
    './output/ESI_datetime.csv', overwrite=True, format='csv')


rec = Table.read ('./output/ESI_datetime.csv',format='csv')
print (rec)
submitting request...
Result downloaded to file [./output/ESI_datetime.csv]
        koaid          instrume   targname  ...    semid    propint
---------------------- -------- ----------- ... ----------- -------
ES.20150902.10478.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.10550.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.45958.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46025.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46101.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46168.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46410.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.85368.fits      ESI          -- ...   2015b_eng      18
ES.20150902.85427.fits      ESI          -- ...   2015b_eng      18
ES.20150902.85487.fits      ESI          -- ...   2015b_eng      18
                   ...      ...         ... ...         ...     ...
ES.20150911.24831.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.26693.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.35455.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.42470.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.44332.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.46194.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.48056.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.50078.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.51339.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.52604.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.53867.fits      ESI  J0246-0825 ... 2015b_u077e      18
Length = 225 rows

Query by date range; output in TSV format

In [7]:
Koa.query_datetime ('esi', \
    '2015-09-01 00:00:00/2015-09-30 23:59:59', \
    './output/ESI_datetime.tsv', overwrite=True, format='tsv')

rec = Table.read ('./output/ESI_datetime.tsv',format='ascii.fast_tab')
print (rec)
submitting request...
Result downloaded to file [./output/ESI_datetime.tsv]
        koaid          instrume   targname  ...    semid    propint
---------------------- -------- ----------- ... ----------- -------
ES.20150902.10478.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.10550.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.45958.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46025.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46101.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46168.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.46410.fits      ESI     unknown ...   2015b_eng      18
ES.20150902.85368.fits      ESI          -- ...   2015b_eng      18
ES.20150902.85427.fits      ESI          -- ...   2015b_eng      18
ES.20150902.85487.fits      ESI          -- ...   2015b_eng      18
                   ...      ...         ... ...         ...     ...
ES.20150911.24831.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.26693.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.35455.fits      ESI HS2209+1914 ... 2015b_u077e      18
ES.20150911.42470.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.44332.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.46194.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.48056.fits      ESI HE0047-1756 ... 2015b_u077e      18
ES.20150911.50078.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.51339.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.52604.fits      ESI  J0246-0825 ... 2015b_u077e      18
ES.20150911.53867.fits      ESI  J0246-0825 ... 2015b_u077e      18
Length = 225 rows

Query by position

In [8]:
Koa.query_position ('esi', \
                  'circle 255.03913 -11.82214 1.0', \
                  './output/position_search.tbl', overwrite=True )

rec = Table.read ('./output/position_search.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/position_search.tbl]
        koaid          instrume   targname  ...    semid    propint
                                            ...                    
---------------------- -------- ----------- ... ----------- -------
ES.20140724.19862.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.19954.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20187.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20277.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20549.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20649.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20925.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.21025.fits      ESI s17002-1149 ... 2014a_k197e      18

Query by object

In [9]:
Koa.query_object ('esi', \
                  'M32', \
                  './output/M32.tbl', overwrite=True)

rec = Table.read ('./output/M32.tbl', format='ascii.ipac')
print (rec)
object name resolved: ra2000= 10.67427083, dec2000=40.86516944
submitting request...
Result downloaded to file [./output/M32.tbl]
        koaid          instrume  targname  ...    semid    propint
                                           ...                    
---------------------- -------- ---------- ... ----------- -------
ES.19990830.50746.fits      ESI        M32 ...   1999b_eng      18
ES.19990830.50857.fits      ESI        M32 ...   1999b_eng      18
ES.19990830.51267.fits      ESI        M32 ...   1999b_eng      18
ES.20000922.27039.fits      ESI        M32 ...  2000b_u61e      18
ES.20000922.29502.fits      ESI        M32 ...  2000b_u61e      18
ES.20000922.30362.fits      ESI        M32 ...  2000b_u61e      18
ES.20000922.31237.fits      ESI        M32 ...  2000b_u61e      18
ES.20000922.32249.fits      ESI        M32 ...  2000b_u61e      18
ES.20000922.32359.fits      ESI        M32 ...  2000b_u61e      18
ES.20000922.32721.fits      ESI        M32 ...  2000b_u61e      18
                   ...      ...        ... ...         ...     ...
ES.20000924.50625.fits      ESI  M32_ref_c ...  2000b_u63e      18
ES.20000924.51491.fits      ESI M32_center ...  2000b_u63e      18
ES.20000924.51877.fits      ESI M32_center ...  2000b_u63e      18
ES.20000924.52153.fits      ESI M32_center ...  2000b_u63e      18
ES.20000927.50348.fits      ESI        M32 ...  2000b_u46e      18
ES.20000927.51003.fits      ESI        M32 ...  2000b_u46e      18
ES.20060928.48245.fits      ESI        m31 ... 2006b_u001e      18
ES.20111002.22121.fits      ESI  4184-1120 ... 2011b_h259e      18
ES.20111002.23976.fits      ESI  4184-1120 ... 2011b_h259e      18
ES.20111003.23278.fits      ESI  4184-1120 ... 2011b_h259e      18
Length = 80 rows

Query for program information

In [10]:
query ="select koaid, filehand, progid from koa_esi where (progid = 'K197E') " 

Koa.query_adql (query, \
    './output/program_info.tbl', overwrite=True, \
    format='ipac')

rec = Table.read('./output/program_info.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/program_info.tbl]
        koaid          ... progid
---------------------- ... ------
ES.20140724.08809.fits ...  K197E
ES.20140724.09553.fits ...  K197E
ES.20140724.10762.fits ...  K197E
ES.20140724.10811.fits ...  K197E
ES.20140724.11148.fits ...  K197E
ES.20140724.11286.fits ...  K197E
ES.20140724.11391.fits ...  K197E
ES.20140724.11496.fits ...  K197E
ES.20140724.11615.fits ...  K197E
ES.20140724.11721.fits ...  K197E
                   ... ...    ...
ES.20140724.33747.fits ...  K197E
ES.20140724.33833.fits ...  K197E
ES.20140724.33919.fits ...  K197E
ES.20140724.34019.fits ...  K197E
ES.20140724.34713.fits ...  K197E
ES.20140724.35396.fits ...  K197E
ES.20140724.36090.fits ...  K197E
ES.20140724.36465.fits ...  K197E
ES.20140724.36862.fits ...  K197E
ES.20140724.36948.fits ...  K197E
ES.20140724.37034.fits ...  K197E
Length = 105 rows

Query by instrument, date, and position

In [11]:
param = dict()
param['instrument'] = 'esi'
param['datetime'] = '2014-07-24 00:00:00/2014-07-24 23:59:59'
param['pos'] = 'circle 255.03913 -11.82214 1.0'

Koa.query_criteria (param, \
                  './output/parameters.tbl', overwrite=True)

rec = Table.read ('./output/parameters.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/parameters.tbl]
        koaid          instrume   targname  ...    semid    propint
                                            ...                    
---------------------- -------- ----------- ... ----------- -------
ES.20140724.19862.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.19954.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20187.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20277.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20549.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20649.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.20925.fits      ESI s17002-1149 ... 2014a_k197e      18
ES.20140724.21025.fits      ESI s17002-1149 ... 2014a_k197e      18

General Metadata Queries With the Astronomical Data Query Langage (ADQL) Method.

A TAP query made with the ADQL method enables you to make general and complex queries against the archive. If you wish to download data discovered via the ADQL query made against KOA, you must include explicitly include the koaid, instrument , and filehandle in the query.


Spatial cone search query with column selection; order by UT Time.

In [3]:
query =  "select koaid, object, koaimtyp, frameno, ra, dec, \
            to_char(date_obs,'YYYY-MM-DD') as date_obs, ut  \
            elaptime, waveblue, wavered, filter, apmsknam, slmsknam, obsmode, \
            progid, proginst,  progpi, progtitl, semester, ofname, filehand \
            from koa_esi where \
            (contains(point('J2000',ra ,dec ), circle('J2000', 23.48 ,30.60, 1))=1) \
            order by utdatetime"



Koa.query_adql (query, \
    './output/adql_cone_UTorder.tbl', overwrite=True, \
    format='ipac')

rec = Table.read ('./output/adql_cone_UTorder.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/adql_cone_UTorder.tbl]
        koaid          ...                       filehand                     
                       ...                                                    
---------------------- ... ---------------------------------------------------
ES.19991128.35903.fits ... /koadata36/ESI/19991128/lev0/ES.19991128.35903.fits
ES.19991128.36009.fits ... /koadata36/ESI/19991128/lev0/ES.19991128.36009.fits
ES.19991128.36188.fits ... /koadata36/ESI/19991128/lev0/ES.19991128.36188.fits
ES.19991128.36425.fits ... /koadata36/ESI/19991128/lev0/ES.19991128.36425.fits
ES.19991128.36717.fits ... /koadata36/ESI/19991128/lev0/ES.19991128.36717.fits
ES.20001018.34771.fits ... /koadata36/ESI/20001018/lev0/ES.20001018.34771.fits
ES.20001018.36255.fits ... /koadata36/ESI/20001018/lev0/ES.20001018.36255.fits
ES.20001018.37509.fits ... /koadata36/ESI/20001018/lev0/ES.20001018.37509.fits
ES.20001018.38907.fits ... /koadata36/ESI/20001018/lev0/ES.20001018.38907.fits
ES.20001018.40161.fits ... /koadata36/ESI/20001018/lev0/ES.20001018.40161.fits
                   ... ...                                                 ...
ES.20111003.34729.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.34729.fits
ES.20111003.36701.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.36701.fits
ES.20111003.38565.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.38565.fits
ES.20111003.40733.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.40733.fits
ES.20111003.42593.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.42593.fits
ES.20111003.44449.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.44449.fits
ES.20111003.46307.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.46307.fits
ES.20111003.48708.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.48708.fits
ES.20111003.50567.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.50567.fits
ES.20111003.52438.fits ... /koadata36/ESI/20111003/lev0/ES.20111003.52438.fits
Length = 120 rows
In [13]:
query =  "select koaid from koa_esi where \
           (contains(point('J2000',ra ,dec ), box('J2000', 23.48 ,30.60, 1, 1))=1) "

Koa.query_adql (query, \
    './output/adql_radec.tbl',overwrite=True, \
    format='ipac')

    
rec = Table.read ('./output/adql_radec.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/adql_radec.tbl]
        koaid         
----------------------
ES.19991128.35903.fits
ES.19991128.36009.fits
ES.19991128.36188.fits
ES.19991128.36425.fits
ES.19991128.36717.fits
ES.20001018.34771.fits
ES.20001018.36255.fits
ES.20001018.37509.fits
ES.20001018.38907.fits
ES.20001018.40161.fits
                   ...
ES.20111003.29129.fits
ES.20111003.31010.fits
ES.20111003.32868.fits
ES.20111003.34729.fits
ES.20111003.40733.fits
ES.20111003.42593.fits
ES.20111003.44449.fits
ES.20111003.46307.fits
ES.20111003.48708.fits
ES.20111003.50567.fits
ES.20111003.52438.fits
Length = 96 rows

Select top 10 records in a spatial box search with column selection; order results in descending UT time.

In [14]:
query =  "select top 10 koaid, ra ,dec, utdatetime from koa_esi \
           where (contains(point('J2000',ra ,dec),  \
           box('J2000', 23.48 ,30.60, 1, 1)) =1) order by utdatetime desc "
Koa.query_adql (query, \
    './output/adql_radec.tbl',overwrite=True, \
    format='ipac')

rec = Table.read ('./output/adql_radec.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/adql_radec.tbl]
        koaid             ra      dec            utdatetime        
---------------------- -------- -------- --------------------------
ES.20001019.45897.fits 23.38388 30.55814 2000-10-19 12:44:57.590000
ES.20001019.44622.fits 23.38388 30.55811 2000-10-19 12:23:42.140000
ES.20001019.43118.fits 23.36121 30.55583 2000-10-19 11:58:38.380000
ES.20001018.37509.fits 23.48954 30.76078 2000-10-18 10:25:09.800000
ES.20001018.36255.fits 23.48954 30.76078 2000-10-18 10:04:15.290000
ES.19991128.36717.fits 23.49087 30.59231 1999-11-28 10:11:57.470000
ES.19991128.36425.fits 23.49087 30.59231 1999-11-28 10:07:05.740000
ES.19991128.36188.fits 23.49087 30.59231 1999-11-28 10:03:08.710000
ES.19991128.36009.fits 23.49087 30.59231 1999-11-28 10:00:09.560000
ES.19991128.35903.fits 23.49087 30.59231 1999-11-28 09:58:23.760000
In [15]:
query =  "select count(koaid) from koa_esi \
    where (contains(point('J2000',ra,dec), box('J2000', 23.48 ,30.60, 1, 1 ))=1) "

Koa.query_adql (query, \
    './output/adql_count.tbl',overwrite=True, \
    format='ipac')

rec = Table.read ('./output/adql_count.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/adql_count.tbl]
count(koaid)
------------
          96
In [16]:
query = "select koaid, filehand, ra, dec from koa_esi\
        where contains(point('icrs', ra, dec), \
        polygon('icrs',209.80225,53.34894,209.80225,55.34894,211.80225,54.34894)) = 1"

Koa.query_adql (query, \
    './output/polygon.tbl',overwrite=True, \
    format='ipac')

rec = Table.read ('./output/polygon.tbl', format='ascii.ipac')
print (rec)
submitting request...
Result downloaded to file [./output/polygon.tbl]
        koaid          ...   dec   
---------------------- ... --------
ES.20120315.48168.fits ... 54.41872
ES.20120315.49127.fits ... 54.41872

Download Data

Download a subset of results from the "query by multiple parameters: instrument, date, and position" example above

Please note that if a file is already stored in your directory, it won't be overwritten

In [17]:
Koa.download ('./output/parameters.tbl', \
               'ipac', \
               './dnload_dir1', \
                start_row=1, \
                end_row=3)
Start downloading 3 FITS data you requested;
please check your outdir: ./dnload_dir1 for  progress.
A total of new 3 FITS files downloaded.

Download the full data set from the "query by multiple parameters: instrument, date, and position" example above

In [18]:
Koa.download ('./output/parameters.tbl', \
               'ipac', \
               './dnload_dir2')                   
Start downloading 8 FITS data you requested;
please check your outdir: ./dnload_dir2 for  progress.
A total of new 8 FITS files downloaded.

Download three science FITS files from Query by instrument, date, and position

In [19]:
Koa.download ('./output/parameters.tbl',  
    'ipac', \
    'dnload_dir3',  \
   start_row=0, \
    end_row=2)
Start downloading 3 FITS data you requested;
please check your outdir: dnload_dir3 for  progress.
A total of new 3 FITS files downloaded.

Download three science FITS files and all associated calibration files and calibration file lists from the query by instrument, date, and position

In [20]:
Koa.download ('./output/parameters.tbl',  
    'ipac', \
    'dnload_dir4',  \
   start_row=0, \
    end_row=2, \
    calibfile=1)
 
Start downloading 3 FITS data you requested;
please check your outdir: dnload_dir4 for  progress.
A total of new 3 FITS files downloaded.
3 new calibration list downloaded.
173 new calibration FITS files downloaded.

Login access

The next query shows how a PI can login with their KOA credentials, assigned when the data were acquired, and access their protected data. The example is a query for public data to show the syntax. Please login with your KOA supplied credentials to access your private data. While logged in, you can access all public data as well. Koa.login creates the cookie file at login.

Note: if files have been downloaded already, they will not be downloaded again and overwritten.

In [2]:
Koa.login ('./tapcookie.txt')
Userid: tapdemo
Password: ········
Successfully login as tapdemo

Include the cookiepath in your query to access your data, as in this example:

In [3]:
Koa.query_datetime ('esi', \
    '2014-07-24 00:00:00/2014-07-24 23:59:59', \
    './output/ESI_login.tbl', overwrite=True, format='ipac', \
    cookiepath='./tapcookie.txt' )

rec = Table.read ('./output/ESI_login.tbl',format='ipac')
print (rec)
                    
submitting request...
Result downloaded to file [./output/ESI_login.tbl]
        koaid          instrume  targname  ...    semid    propint
                                           ...                    
---------------------- -------- ---------- ... ----------- -------
ES.20140724.05955.fits      ESI    unknown ...   2014a_eng      18
ES.20140724.06168.fits      ESI    unknown ...   2014a_eng      18
ES.20140724.06258.fits      ESI    unknown ...   2014a_eng      18
ES.20140724.06553.fits      ESI    unknown ...   2014a_eng      18
ES.20140724.06635.fits      ESI    unknown ...   2014a_eng      18
ES.20140724.06805.fits      ESI    unknown ...   2014a_eng      18
ES.20140724.07169.fits      ESI    unknown ...   2014a_eng      18
ES.20140724.08809.fits      ESI    unknown ... 2014a_k197e      18
ES.20140724.09553.fits      ESI    unknown ... 2014a_k197e      18
ES.20140724.10762.fits      ESI    unknown ... 2014a_k197e      18
                   ...      ...        ... ...         ...     ...
ES.20140724.61851.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.61957.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62061.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62170.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62243.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62318.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62392.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62467.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62543.fits      ESI DOME PHLAT ... 2014a_k188e      18
ES.20140724.62616.fits      ESI DOME PHLAT ... 2014a_k188e      18
Length = 257 rows

Visit KOA at https://koa.ipac.caltech.edu

Please acknowledge the use of KOA by including this text in your publications: "This research has made use of the Keck Observatory Archive (KOA), which is operated by the W. M. Keck Observatory and the NASA Exoplanet Science Institute (NExScI), under contract with the National Aeronautics and Space Administration."

Please also acknowledge the PI(s) of datasets that have been obtained through KOA, and please contact the KOA Help Desk if you publish archival data:

KOA Help Desk

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).

Need help? Submit your questions to the KOA Help Desk