ALS Data and Simulation Portal

Intro

SPOT provides a REST API that can be used in variety of languages (javascript, BASH, python, JAVA etc). The output is typically JSON formatted data.

Authentication and Use

To use the SPOT REST API, you must first authenticate to NEWT using your NIM credentials. NEWT will soon support Multi-Factor Authentication (MFA) and so will SPOT's REST API. In order to simplify the API and ensure security in the SPOT back-end, the authentication process requires two steps by users of the SPOT API:

Authenticate to NEWT, using your NIM username and password. If you have MFA enabled, your password will consist of your NIM password and (immediately following with no spaces) your OTP (one time password) from your NIM MFA token. From this call you will receive a newt_sessionid.

Authenticate to the SPOT API using your username and the newt_sessionid received from NEWT.

Once authenticated to the SPOT API, the newt_sessionid is used for future calls to the API. For curl commands a cookies file is used, whereas in Python the session object will take care of it.

Examples of how to do this in Bash (using curl) and Python are shown below.

For long running sessions, if either the NEWT or API session lifetimes expires, then re-authenticating will be necessary.

Basic Usage in BASH

Authentication

Illustrated example authenticating to NEWT:

$ curl -sS -X POST -d "username=<your_username>&password=<your_nim_password><nim_mfa_otp>" https://newt.nersc.gov/newt/auth | jq .

{
"username": "[your_username]",
"session_lifetime": 43199,
"auth": true,
"newt_sessionid": "43cf49b59ab3608cc2e7ff380c7b02a5"
}

Confirm that values are correct (auth is true, username is yours, lifetime is > 0) then use newt_sessionid as password for authenticating to the SPOT API:

$ curl -c als_cookies.txt -sS -X POST -d "username=<your_username>&password=43cf49b59ab3608cc2e7ff380c7b02a5" https://portal-auth.nersc.gov/als/auth | jq .

{
"username": "[your_username]",
"session_lifetime": 566317,
"auth": true
}

The output of this command is just informative, the als_cookies.txt file is what to use for the real calls to the SPOT API:

$ curl -b als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/search?limitnum=1&end_station=bl832" | jq .

[
  {
    "fs": {
      "size": 4655890616,
      "derivatives": [
        {
          "dstage": "fast-tomopy",
...

A more concise example:

# Credentials:
$ user="ksb"
$ otp="1234" # If MFA enabled for your NIM account
$ pw="Ch@ngeM3"

# Auth to NEWT:
$ newt_sessionid=$(curl -sS -X POST -d "username=${user}&password=${pw}${otp}" https://newt.nersc.gov/newt/auth | jq -r .newt_sessionid)

# Auth to SPOT:
$ curl -c api_cookies.txt -sS -X POST -d "username=${user}&password=${newt_sessionid}" https://portal-auth.nersc.gov/als/auth > /dev/null

# Make REST call:
$ curl -b als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/search?limitnum=1&end_station=bl832" | jq .

SPOT API Reference

Search Datasets

GET

URL:
https://portal-auth.nersc.gov/als/hdf/search

Arguments:
limitnum: number of results to show
skipnum: number of results to skip
sortterm: database field on which to sort (commonly fs.stage_date or appmetadata.sdate)
sorttype: desc or asc
search: query

Result:
JSON List

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/search?skipnum=0&limitnum=10&sortterm=fs.stage_date&sorttype=desc&search=end_station=bl832"

Find Derived Datasets (norm, sino, gridrec, imgrec) from raw dataset

GET

URL:
https://portal-auth.nersc.gov/als/hdf/dataset

Arguments:
dataset: raw dataset

Result:
JSON List

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/dataset?dataset=20130713_185717_Chilarchaea_quellon_F_9053427_IKI_"

View Attributes for Single Dataset and Image Within Dataset

GET

URL:
https://portal-auth.nersc.gov/als/hdf/attributes/[dataset]

Arguments:
group: path in hdf5 file. Set to "/" for overall attributes

Result:
JSON Array

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/attributes/als/bl832/hmwood/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/raw/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_.h5?group=/"

List Images Within Dataset

GET

URL:
https://portal-auth.nersc.gov/als/hdf/listimages/[dataset]

Arguments:
None

Result:
JSON List (paths to images within the HDF5 file)

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/listimages/als/bl832/hmwood/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/raw/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_.h5"

Stage Dataset From Tape to Disk if Required

GET

URL:
https://portal-auth.nersc.gov/als/hdf/stageifneeded/[dataset]

Arguments:
None

Result:
JSON Array

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/stageifneeded/als/bl832/hmwood/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/raw/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_.h5"

Download Dataset

GET

URL:
https://portal-auth.nersc.gov/als/hdf/download/[dataset]

Arguments:
None

Result:
H5 File

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/download/als/bl832/hmwood/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/raw/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_.h" > file.h5

Download Rawdata For Individual Image

GET

URL:
https://portal-auth.nersc.gov/als/hdf/rawdata/[dataset]

Arguments:
group: path in HDF5 file to image

Result:
JSON Array

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/rawdata/als/bl832/hmwood/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/norm/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_-norm-20130714_192637.h5?group=/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/20130713_185717_Chilarchaea_quellon_F_9053427_IKI__0000_0640.tif"

Get Download URLs for .tif and .png files for individual image

GET

URL:
https://portal-auth.nersc.gov/als/hdf/image/[dataset]

Arguments:
group: path in HDF5 file to image

Result:
JSON Array

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/image/als/bl832/hmwood/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/norm/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_-norm-20130714_192637.h5?group=/20130713_185717_Chilarchaea_quellon_F_9053427_IKI_/20130713_185717_Chilarchaea_quellon_F_9053427_IKI__0000_0640.tif"

Run TomoPy on an existing dataset

GET

URL:
https://portal-auth.nersc.gov/als/hdf/tomopyjob

Arguments:
dataset: raw dataset

Result:
JSON List

Example:

$ curl -c als_cookies.txt -sS -X GET "https://portal-auth.nersc.gov/als/hdf/tomopyjob?dataset=20130713_185717_Chilarchaea_quellon_F_9053427_IKI_"

Basic Usage in Python

Use the python requests module to interact with the API:

import requests

Authenticate with NEWT:

newt_session = requests.Session()
newt_response = newt_session.post("https://newt.nersc.gov/newt/auth", {"username": "ksb", "password": "<NIM password>" + "<MFA OTP>"})
newt_response.json()

{'auth': True,
'newt_sessionid': 'bda90e628b81752wx27dca7b0a21e8z0',
'session_lifetime': 43199,
'username': 'ksb'}

You should confirm that "auth" is True, "session_lifetime" is > 0, "newt_sessionid" is not null and "username" is correct, then save the "newt_sessionid":

newt_sessionid = newt_response.json()['newt_sessionid']

Now in a new session, authenticate to the SPOT API using newt_sessionid as your password:

api_session = requests.Session()
api_response = api_session.post("https://portal-auth.nersc.gov/als/auth", {"username": "ksb", "password": newt_sessionid})
api_response.json()

{"username": "spotdev", "session_lifetime": 43200, "auth": True}

Again, confirm username, lifetime & auth, then make calls to the API in that session:

api_response = api_session.get("https://portal-auth.nersc.gov/als/hdf/search?skipnum=0&limitnum=2&sortterm=fs.stage_date&sorttype=desc&search=end_station=bl832")

Look at results:

api_response.json()

See the BASH section above for more API commands.