Observers' Data Access Portal

Introduction

The Observers' Data Access Portal ("ODAP") is a web-based tool for Keck observers to track and download their data as the files are transfered to the Keck Observatory Archive (KOA) during the course of their observations. It is not a replacement for the other methods to access data through KOA.

• Login and Authorization
• Setting Up for Automatic File Downloads
• Manually Downloading Data
• Using the Interactive Table
• End of Night/Run
• Troubleshooting

Login and Authorization

Users should use their KOA credentials to log into the ODAP. In order to avoid potential login issues at the telescope, we recommend checking your KOA login credentials on the KOA site a few days before your observations are scheduled. If you are unable to log in or retrieve a new password using the provided tools, then contact the KOA Help Desk for assistance. PIs should ensure that any CoI who might require access to the data through the ODAP was granted data rights access on the proposal coversheet. If data access rights are required for a CoI, then the PI is encouraged to use self-service Co-I access tool that is visible after logging onto the KOA website or logging onto the ODAP . PIs may also contact the KOA Help Desk with the request at least two working days before the observations; that request needs to include the CoI name, CoI email address, and program ID.

Upon logging in to the ODAP, you will be presented with list of programs (SEMIDs) for which you have data rights access and are on the schedule for the night. The nights are based on UT Date, and the SEMIDs switch at 00:00 UTC; the current UT is presented at the bottom of the browser window. You may also notice a SEMID listed for an engineering program associated with your scheduled instrument. The data files you will see from this program will consist of publicly available calibration files -- like flats or darks -- that you will need your data reduction.

Setting Up for Automatic File Downloads

All raw files ingested into KOA are assigned a unique KOAID that allows for files from different nights or programs to coexist in the same directory or download package without file name conflicts. Understandably, observers may not want to have the names of their data files modified. Users of the ODAP have the option to download the data using the assigned KOAID or the file name as originally saved by the instrument ("OFNAME").

By using the toggle switches, you can select to automatically download any combination of level 0, level 1, and level 2 data files as they are ingested into the archive. When using these file download options, please remember to use a browser on your own computer or laptop and not one running on a remote observing computer.

• Level 0 are raw files nearly identical to the files that are written by the instrument. The only difference is the addition of a few KOA-critical keywords to the header -- no existing keywords are modified and no data are changed. Level 0 data should appear in this interface within a minute of being written to disk by the instrument.
• Level 1 data are partially processed files that are meant only for a quick look at the data; they are generally not suited for direct science use. The speed at which these files appear in the UI depends on the complexity of the data products and pipeline speed, but typically appear within 5-10 minutes of the file being written by the instrument.
• Level 2 data are files that have been fully calibrated and can immediately be used for science. The DRPs that create the level 2 products start in the morning and will fully process any science file for which all of the needed calibration files are available.

Because the downloads happen through the browser, the preferences in your browser will determine where the files are placed on your local drive. Therefore, in order to ensure the smoothest transfer of data to your machine, in your browser preferences, you should turn off the option in which the browser asks where to save each file and you should set the local directory to where you want these files to be stored. At the end of the night, be sure to set your preferences back to your pre-observing settings.

Finally, it is important to ensure your computer does not enter a “sleep mode” in which the internet can be disconnected; this will cause the connection to the ODAP to be lost and the file streaming will be stopped.

NOTE -- users of Chrome-based browsers have reported an issue where the browser becomes disconnected to the file streaming backend service under certain conditions. Users of Firefox and Safari have not reported any disconnections. We recommend using Firefox or Safari while we investigate this issue.

Manually Downloading Data

In addition to the automatic data streaming discussed above, there are two other methods for downloading data.

Single files can be downloaded using the link connected to the "Download" column. These files will be saved according to the preferences set in your browser and will be saved with the file name given in this column.

The current contents of the table can be downloaded using a wget script. Clicking on the button generates one or more shell scripts with wget commands for downloading the data; one script is created for each SEMID listed. Any filters that are applied to the table are captured in the list of files in the wget script (see the next section for more details). In order to run one of these scripts, you must first make the file executable:

chmod a+x INST_SEMID_UTDATE.sh

Where INST is the name of the instrument, SEMID is the SEMID, and UTDATE is the UT Date at the time the script was created. To run the script, you can call the file directly:

./INST_SEMID_UTDATE.sh

Using the Interactive Table

The primary function of the table is to provide you with a realtime look at when files are transfered from WMKO to the KOA disks at NExScI. However, the interactive table provides a few options and features if you need more flexibility.

• The table can be sorted by the contents of any column by clicking on the column name.
• You can search for a string anywhere in the table using the "Search" function. Although the table contents are modified to only show rows with the matching string, this type of "filtering" is not captured by the wget scripts.
• Filtering on the table rows can be done using the "Filter" function. Only string-type filters are allowed in this table, meaning a filter like "right ascension is greater than 12 hours" is not possible. In addition, currently only a single column can be filtered at a time, so if you create a new filter when one is in place, the new filter will override the old filter. If you "Download WGET Script(s)" while a filter is applied to the table using this function, the files in the wget scripts will honor that filter. Please note that while the RA and DEC columns are displayed in the ODAP as sexagesimal strings to make them more easily human readable, they are actually stored in the database in decimal degrees. Table filters with 'RA equals' or 'DEC equals' need to be translated from sexagesimal values to decimal degrees for the database query, which could result in a mismatch between the ODAP contents and the files in the wget script.
• "Export" can be used to download the entire table as a comma-separated value (CSV) table.
• The spacing between rows can be changed with the "Density" function. The number of rows displayed does not change, however.
• Use the "Columns" function if you want to add or remove columns from the table display. Currently, all available columns are shown by default. In a future update, we plan to include additional columns.
• The link in the first column of the table can be used to download individual files to your computer.

End of Night/Run

At the end of your night, if you are expecting more calibration data to be taken after you go to bed, then you can leave the ODAP running. If you return to the ODAP after the switchover in UT date, you will receive a notification in the browser that your session has ended. Under normal conditions, you should have continued to receive data until the date switchover. If you have another night of observing, you will need to log off and log back in.

At the end of your last night of observing, once you are certain you have downloaded all of the data from your night, you should log off the OPAD using the logout button at the bottom of the window. This will ensure that you do not inadvertently download calibration data from other science programs using the same instrument. Be sure to also revert your browser's download settings back to their pre-observing values.

Troubleshooting

Problem	Possible Solution
The SEMID for your program isn't listed after you log in.	If you are the PI for the observing program, then contact your Support Astronomer and KOA. If you are a CoI on the observing program, then it is possible that you were not given access to the program. In this case, the PI will need to request to KOA to provide you access.
The downloaded files are not where you expect them.	The automatic download feature works through the browser, so make sure your browser preferences are set correctly.
Your data are being downloaded, but the process is slower than expected.	A slowdown in the download speed could occur if multiple files are trying to be downloaded at once. You can try to alleviate this issue by turning off the downloading of certain file types (eg., level 1). All data that appear in the ODAP are available in the usual KOA query interfaces, so they can be downloaded at a later time if they are not immediately required.
The interactive table is not being updated with new files.	Check that the "Status" message at the bottom of the browser window says "Connected". If it does not, try reloading the page or logging out and back in. If that does not correct the connection, then contact your Support Astronomer. -- There is a good connection to the real time ingestion server. -- There is a poor or no connection to the real time ingestion server.