Packaging Files for Download

• Select Files
• Including Calibrated/Extracted/Reduced Data
• Including NIRC2 _unp Data
• Including NIRC2 TRS Data
• Filename Options
• Submit the Download Form
• Confirm Your Request
• Retrieve TAR Files from Workspace
• Unpacking TAR Files
• Sample File Names

Select Files:

By default, every file on the results page is selected for download. You can customize your download package by:

• Checking or unchecking the box to the left of each KOAID
• Clicking on the Select All and De-select All links in the science and calibration results tabs. See Understanding Search Results.

If you searched for science files and checked "Include associated calibration files," you can choose between two methods of download packaging:

Figure 1: Download Packaging Options (calibration association)

1. Automatic
   
   If "Automatic" is selected as the File Packaging Method, you may select the science files you wish to download, and all associated calibration files will automatically be included in the package. In this mode, you do not need to check the boxes next to calibration files of interest; you only need to choose the science files you wish to download.
   
   Note: In automatic mode, any user-selected calibration files will be ignored by the packaging program; only calibration files associated with the selected science files will be packaged.
2. Manual
   
   In this mode, the user has complete control over which files of either type will be included in the download package. Calibration files that are associated with a science file will not be automatically included in your download package. On page load all the calibration files for a night are checked, but if you only want a subset, you must click on the Calibration Files tab and click the lower right tab button "De-Select All Calibration Files" (after Reset Filters if you have any column filters set) and check the desired calibration files by hand.
   
   Your download package will still include the ".caliblist" text files for each selected science file. This will allow you to go back and manually retrieve any additional calibration files you wish at a later date.

Including Calibrated/Extracted/Reduced Data

If your search results include science files, you will have the option to include extracted/calibrated data along with the raw (uncalibrated) science files. Please note that extracted/calibrated data are not available for all science files.

Figure 2: Including Extracted Spectra

See DEIMOS Reduced Data, HIRES Extracted Spectra, KCWI Reduced Data, LWS Reduced Data, NIRC2 Calibrated Data, NIRSPEC Extracted Spectra, and OSIRIS Reduced Data

Including NIRC2 _unp Data

The multi-dimensional NIRC2 _unp files can be many times larger than the 2-d NIRC2 level 0 files and including them will significantly increase the size of your download. The _unp files must also be selected in the table to be included in the download. If this option is not checked, the _unp files will not be included in the download even if selected in the results table

Including NIRC2 TRS Data

TRS (Telemetry Recording System) data are available for NIRC2 science data taken beginning 18 Aug 2019. Users who wish to include these files for all available NIRC2 science files, should check the "Include TRS data" box. Note that these file have much higher data volume compared to NIRC2 and will significantly increase the size of the download.

Filename Options

By default, your download package will contain files with the same filenames as shown in the KOAID column on the search results page. These are unique filenames, assigned by the Keck Observatory Archive.

For observers with logs referring to the original telescope-assigned filenames, or other researchers with data reduction tools that expect those filenames, it may be desirable to retrieve the files in that format instead. Select "Non-unique filename, assigned at the telescope" to download your files with telescope-assigned filenames.

We also provide a standalone UNIX shell script, koa_translator, that converts KOAID-style filenames to telescope-assigned filenames for data that users have already retrieved. Additional documentation on telescope-assigned filenames is available at that link.

Figure 3: Filename Options

Submit the Download Form:

• At the top of the form, enter a valid email address. This will be used to notify you when the files are ready to be downloaded; we do not store or give out email addresses.
• Make sure the desired files have been selected (see Selecting Files above)
• Click the "Download Files" button to submit the request
• If you do not see a confirmation request page, please contact KOA Support

Figure 4: Request for Confirmation

Confirm your Request:

• Before download continues, you are asked to confirm your e-mail address and the number of selected files (Figure 4). Please double-check that this information is correct.
• Hit "Continue" to start file packaging
• Note the "Job Number" displayed on the final confirmation page (Figure 5); this number will be referenced in the email you receive.
• If you do not see a "File Packaging in Progress" page with a Job Number, please contact KOA Support.

Figure 5: File Packaging in Progress

Retrieve TAR Files from Workspace:

You will receive email notification when your files are ready; this process can take from less than a minute to several hours, depending on how many files have been selected, and how busy the server is. (Selecting 2 or 3 files may take only a minute; selecting hundreds can take up to several hours.)

If you have not received an email message after a reasonable amount of time, please contact KOA Support to inquire after the status of your job.

The email message contains a link to a download page, like the one shown in Figure 6.

Figure 6: Sample Download Page

The download page includes:

• Links to TAR files. TAR file names include a prefix denoting instrument, file type (sci or cal), their job ID, and a sequential counter such as: HIRES_sci_79849_1.tar.
  
  To download a single TAR file, click on the filename. (You may be prompted to log in.)
  
  To the right of each filename is the number of KOAIDs included in the TAR file, which links to a text file that lists the raw FITS files (identified by KOAID) inside the TAR file.
  
  
• A shell script (curl_job ID.sh or wget_jobID) for users who wish to retrieve all their TAR files at once. Please see "CURL and WGET Scripts to Download File Packages" for more information.
  
  
• [DEIMOS only] When reduced data is selected and available, a link to a shell script you can use to download them all at once. This script is also included in the first tarfile of the science data.
  
  
• [KCWI only] When reduced data is selected and available, a link to a shell script you can use to download them all at once. This script is also included in the first tarfile of the science data.
  
  
• [OSIRIS only] Links to download rectification matrices from WMKO, and the link to shell script you can use to download them all at once. This script is also included in the first tarfile of the science data.

Any errors that occurred during file packaging will be described at the bottom of the download page.

Note: Packaged files are held in a workspace area on the KOA server; files older than a few days are routinely deleted. The download page displays the date and time after which your packages may be deleted.

Unpacking TAR Files:

To unpack the TAR file, use the command "tar xvf filename"

All the TAR files from a given Job ID will unpack into a directory named "KOA_jobid", with subdirectories for each instrument. The rest of the directory structure is determined by the file naming convention requested for the downloaded data: either KOAID or original file names.

KOAID File Names

KOA_jobid/INSTRUMENT/raw/sci/

Contains g-zipped science FITS files (KOAID.fits.gz), as well as JPEG previews where available. If you searched for "Science Files with Associated Calibration", this subdirectory will also include the "caliblist" text files for each science file, even if no calibration files were actually downloaded.

KOA_jobid/INSTRUMENT/raw/cal/

Contains g-zipped calibration FITS files (KOAID.fits.gz), as well as JPEG previews where available.

KOA_jobid/INSTRUMENT/<extracted data product type>

If you chose to include extracted data products, they will be in a subdirectory named calibrated, extracted, reduced, or reduced_cubes, depending on the instrument.

Telescope-assigned File Names

KOA_jobid/INSTRUMENT/YYYYmonDD/raw/

Contains g-zipped science and calibration FITS files, as well as JPEG previews where available. If you searched for "Science Files with Associated Calibration", this subdirectory will also include the "caliblist" text files for each science file, even if no calibration files were actually downloaded. However, the caliblist contents are KOAID-based, so if you are relying on the archive's calibration association it is recommended you use the KOAID file name scheme for download.

KOA_jobid/INSTRUMENT/YYYYmonDD/<extracted data product type>

If you chose to include extracted data products, they will be in a subdirectory named calibrated, extracted, reduced, or reduced_cubes, depending on the instrument. Note: All data products of a given type (raw or processed) are placed in the same subdirectory for a given night, meaning that hundreds or thousands of HIRES extraction products may be in a single directory.

Special Files

If you download DEIMOS reduced data, a curl and wget script for retrieving lev1 and/or lev2 data (named DEIMOS_reduced_curl(wget)jobid.sh) will be included in the first DEIMOS science tarball, and will unpack into the KOA_jobid/DEIMOS directory.

If you download KCWI reduced data, a curl and wget script for retrieving lev1 and/or lev2 data (named KCWI_reduced_curl(wget)jobid.sh) will be included in the first KCWI science tarball, and will unpack into the KOA_jobid/KCWI directory.

If you downloaded HIRES extracted data, the file README_extracted.txt will be included in the first HIRES science tarball, and will unpack into the KOA_jobid/HIRES directory.

If you downloaded OSIRIS science data, a wget script for retrieving rectification matrices (named recmat_wget_jobid.sh) will be included in the first OSIRIS science tarball, and will unpack into the KOA_jobid/OSIRIS directory.

Sample File Names:

The following table shows some sample files in their KOAID-style and Telescope-assigned names and locations beneath the KOA_jobid directory: