CURL and WGET Scripts to Download File Packages
- Download Page Example
- Using the WGET Script to Download File Packages
- Using the CURL Script to Download File Packages
- Curl Instructions for altering a WGET script
- Reduced Science Quality Data Downloads
Download Page Example
Using the WGET Script to Download File Packages
GNU wget is a command-line tool for retrieving files via HTTP. If your download package includes more than one TAR file, it may be easier to download and run the wget script to retrieve your files, rather than downloading them one at a time.
The wget script generated by the KOA download packaging process includes commands to retrieve each of your TAR files, and sends the appropriate cookie so that you can access proprietary data.
If wget is unavailable, see the Curl Instructions below.
To use the wget script:
- Right-click on the link to the wget script at the bottom of the download page, and save it to your local machine. Save it inside the directory where you want to place the TAR files.
- Make sure the wget file is executable. If not, change the permissions by typing "chmod 755 wget_NNNN.sh" (where NNNN is the job ID number, included in the wget script filename).
- Run the script by typing "./wget_NNNN.sh" from inside the same directory as the script.
If you don't already have wget installed, you can download it from one of the following sites:
- UNIX source code: http://ftp.gnu.org/pub/gnu/wget
- Windows binaries: http://xoomer.alice.it/hherold
Note:
We have discovered that wget version 1.14 running in an XTerm (281) through XQuartz 2.7.4 on OSX 10.8.4 produces erroneous output on the terminal. The wget command will print out Saving to: â, using â rather than the name of the file being saved (eg., HIRES_sci_27166_1.tar). The downloaded file will be saved with the correct name, so this is purely a cosmetic printing error using this version of the xterm. We have not seen this printing error in older versions of xterm nor do we see it using Terminal (v2.3) on OSX 10.8.4.
Using the CURL Script to Download File Packages
Curl is a command-line tool for retrieving files via HTTP. If your download package includes more than one TAR file, it may be easier to download and run the curl script to retrieve your files, rather than downloading them one at a time.The curl script generated by the KOA download packaging process includes commands to retrieve each of your TAR files, and sends the appropriate cookie so that you can access proprietary data.
If curl is unavailable, see the WGET Instructions above.
To use the curl script:
- Right-click on the link to the curl script at the bottom of the download page, and save it to your local machine. Save it inside the directory where you want to place the TAR files.
- Make sure the curl file is executable. If not, change the permissions by typing "chmod 755 curl_NNNN.sh" (where NNNN is the job ID number, included in the wget script filename).
- Run the script by typing "./curl_NNNN.sh" from inside the same directory as the script.
If you don't already have curl installed, you can download it from Curl Instructions for altering a WGET script
If you have an existing wget script (and not a curl script) and wish to convert to using curl, first create the cookie file with the following at the comand line, substituting your KOA login username and password for "USER" and "PASS":
curl -b ./mycookie.txt -c ./mycookie.txt --data "userid=USER&password=PASS&ref=login_user&ref_page=/" https://koa.ipac.caltech.edu/cgi-bin/KOA/nph-KOAlogin
Then for each tar file you wish to download, use the full path to the tar file available in the wget_nnnn.sh script and redirect the output to the tarfile name as in the form of the following:
curl -b ./mycookie.txt 'https://koa.ipac.caltech.edu/cgi-bin/KOADownload/nph-KOADownload?ref=\ISIS_WORKDIR\/TMP_XXXxxxxxx/KOA/HIRES_sci_12345_1.tar' > HIRES_sci_12345_1.tar
Reduced Science Quality Data Downloads
When selecting reduced data downloads for DEIMOS and KCWI, curl and wget scripts are included on the download results page and packaged with the first science tarfile. These scripts will prompt you for your KOA login and a cookie will be saved locally for use by the script. If you are downloading public reduced data (no login is required), then when prompted for the username and password you can just press tthe return key.
Please acknowledge the use of KOA by including this text in your publications:
This research has made use of the Keck Observatory Archive (KOA), which is operated by the W. M. Keck Observatory and the NASA Exoplanet Science Institute (NExScI), under contract with the National Aeronautics and Space Administration.
Please also acknowledge the PI(s) of datasets that have been obtained through KOA, and please contact the KOA Help Desk if you publish archival data.
If your research uses data reduced by the DEIMOS DRP or MOSFIRE DRP, please use the PypeIt Citation.
Privacy Policy Image Use Policy
The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).
