Skip to content

DRAFT cloud foundations content update #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
sarahw-earthscope wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

DRAFT cloud foundations content update #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cdd9f7b
scaffolding and output updates
sarahw-earthscope May 22, 2026
2c06220
rename fundamentals -> foundations
sarahw-earthscope May 27, 2026
3576f1c
move intro, git, env to moodle; renumber notebooks
sarahw-earthscope May 27, 2026
60367d6
remove rinex data from seismic exercise
sarahw-earthscope May 27, 2026
47b6a99
typo in README
sarahw-earthscope May 27, 2026
caaa707
updated es_sdk download examples for es_sk-1.4
sparafina-earthscope May 27, 2026
4a1bb9d
NGF language update
sarahw-earthscope May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added tutorials/.DS_Store
View file
Open in desktop
Binary file not shown.
10 changes: 7 additions & 3 deletions tutorials/fundamentals/3_authorization.ipynb → tutorials/foundations/1_authorization.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,15 @@
"source": [
"# Getting a Token to Access EarthScope Services\n",
"\n",
"EarthScope currently offers data through web services and, in the future, direct access to seismic and geodetic data in the cloud. In this module, we'll learn how to request an authorization token for downloading or accessing data from EarthScope.\n",
"The NSF National Geophysical Facility currently offers data through web services and, in the future, direct access to seismic and geodetic data in the cloud. In this module, we'll learn how to request an authorization token for downloading or accessing data directly from EarthScope. \n",
"\n",
"```{note}\n",
"An access token is not required to access data using obspy.\n",
"```\n",
"\n",
"## Getting an EarthScope Token\n",
"\n",
"EarthScope's funders require tracking data usage. To this end, EarthScope issues a token to access resources. Tokens are tied to your EarthScope account (which you used to log into GeoLab). Tokens are generated using either the EarthScope SDK (Software Development Kit) or CLI (Command Line Interface). Both packages are installed in GeoLab. The simplest way to generate a token is to use the EarthScope CLI. Copy and paste `es login` into a terminal, then follow the instructions to get a token.\n",
"The NSF requires EarthScope tracking data usage. To this end, EarthScope issues a token to access resources. Tokens are tied to your EarthScope account (which you used to log into GeoLab). Tokens are generated using either the EarthScope SDK (Software Development Kit) or CLI (Command Line Interface). Both packages are installed in GeoLab. The simplest way to generate a token is to use the EarthScope CLI. **Copy and paste `es login` into a terminal, then open the link to get a token.**\n",
"\n",
"```bash\n",
"es login\n",
Expand Down Expand Up @@ -121,7 +125,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.5"
"version": "3.12.5"
}
},
"nbformat": 4,
Expand Down
66 changes: 44 additions & 22 deletions tutorials/fundamentals/4_seismic_data.ipynb → .../foundations/2_seismic_data_example.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"source": [
"This section details how to get seismic data from EarthScope. This section requires a working knowledge of Python. You should have an understanding of data types and string formatting, creating file paths and file naming, web requests, and creating Python functions. In addition to Python, you should have an understanding of miniSEED data distributed through the International Federation of Digital Seismograph Networks (FDSN). \n",
"\n",
"The notebook material is informational and is useful for completing the seismic and geodetic exercises."
"**This notebook assumes you have already run the `es login` command to obtain an access token.** You can learn more about the login flow and authentication tokens in the authorization notebook."
]
},
{
Expand All @@ -25,15 +25,18 @@
"source": [
"## Getting Seismic Data from EarthScope\n",
"\n",
"Seismic data is available through third party packages such as [obspy](https://docs.obspy.org/) or through EarthScope's `dataselect` web service.\n",
"\n",
"![](images/Web_Services_Data_Flow.png)\n",
"Seismic data is available through EarthScope's `dataselect` web service, or through third party packages such as [`obspy`](https://docs.obspy.org/) (which use dataselect under the hood).\n",
"\n",
"Currently, the simplest way to access EarthScope's seismic data is to use a third party package such as obspy. If you want to work directly with miniSEED files, EarthScope's dataselect service supports selecting sorting event data and returns data in the miniSEED format. \n",
"\n",
"![](images/cloud_native_data_access.png)\n",
"```{note}\n",
"Because GeoLab runs in the same AWS region as the data is stored in, `dataselect` routes the data to GeoLab without any egress from the cloud. You can follow these same instructions to run `dataselect` and `obspy` on your local machine, but they are _significantly faster_ when run on cloud resources in AWS US-east-2, such as GeoLab. \n",
"```\n",
"\n",
"At the end of this notebook, we also provide an overview of how data is stored in AWS S3 and an example of how to access data from a public S3 bucket. The NSF NGF does not currently have data available in this format, but many other data providers do (e.g., [NCEDC](https://ncedc.org/) and [SCEDC](https://scedc.caltech.edu/). The NSF NGF plans to offer public S3 bucket access in the future and will update this tutorial and our documentation when it is available. \n",
"\n",
"In the future, data will be available from EarthScopes cloud services hosted in Amazon's S3 storage service. We provide an overview of how data is stored in S3 and an example of how to access data from a public S3 bucket. "
"Until NSF NGF data is available in public buckets, it can be accessed directly from EarthScope's AWS S3 storage service. This advanced approach is not covered in this tutorial, but you can learn more about the S3 direct access program by visiting the [SDK docs](https://docs.earthscope.org/sdk/s3-direct-access-tutorial).\n",
"\n"
]
},
{
Expand All @@ -45,27 +48,36 @@
"\n",
"Obspy is a Python framework (or package) for processing seismic data. It provides parsers for common file formats, clients to access data centers and seismological signal processing routines which allow the manipulation of seismological time series.\n",
"\n",
"We'll start by importing modules from the obspy package. It's not necessary to import the entire package, just the function from the module, e.g., obspy.clients.fdsn. Next we create a client that connects to EarthScope's services and send a query based on start and end time, and the minimum magnitude of an event.\n",
"We'll start by importing modules from the obspy package. It's not necessary to import the entire package, just the function from the module, e.g., obspy.clients.fdsn. Next we create a client that connects to EarthScope's services and send a query based on start and end time, and the minimum magnitude of an event. The data is returned in a catalog, which is a list-like container for events. \n",
"\n",
"```python\n",
"An example of instantiating a client and retrieving data is below. To learn more about working with `obspy`, see the documentation and tutorials on their [web site].(https://docs.obspy.org/) \n",
"\n",
"Note, a token is not required to access data using obspy. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4b317fb6-f98e-4ce3-b8f5-38a4ef89d88a",
"metadata": {},
"outputs": [],
"source": [
"from obspy.clients.fdsn import Client\n",
"from obspy.clients.fdsn.header import URL_MAPPINGS\n",
"from obspy import UTCDateTime\n",
"import warnings\n",
"import cartopy\n",
"\n",
"warnings.filterwarnings('ignore')\n",
"warnings.simplefilter('ignore')\n",
"\n",
"# creates a client that connects to the IRIS data center\n",
"# creates a client that connects to the IRIS (NGF) data center\n",
"client = Client(\"IRIS\")\n",
"\n",
"starttime = UTCDateTime(\"2020-01-01\")\n",
"endtime = UTCDateTime(\"2025-12-31\")\n",
"catalog = client.get_events(starttime=starttime, endtime=endtime, minmagnitude=7)\n",
"```\n",
"\n",
"The data is returned in a catalog, which is a list-like container for events. To learn more about working with obspy, see the documentation and tutorials on the obspy [web site](chttps://docs.obspy.org/) "
"print(catalog)"
]
},
{
Expand Down Expand Up @@ -93,7 +105,9 @@
"\n",
"To download a file, we can use the `requests` package to send the HTTP request to the dataselect web service. As discussed in the previous section, the request must include an authorization token using the `get_token` function.\n",
"\n",
"The `download_data` function requires the query parameters required by the FDSN Web Service specification, and where to write the data. The function does several things. First, it requests an authorization token. Next, it creates a file name for the data. Finally, it sends the request to dataselect and writes the data to a file in a directory."
"The `download_data` function requires the query parameters required by the FDSN Web Service specification, and where to write the data. The function does several things. First, it requests an authorization token. Next, it creates a file name for the data. Finally, it sends the request to dataselect and writes the data to a file in a directory.\n",
"\n",
"**Note, you'll need an authentication token to successfully run the cell below. See the Authorization Notebook for instructions.**"
]
},
{
Expand All @@ -108,7 +122,7 @@
"from datetime import datetime\n",
"from earthscope_sdk import EarthScopeClient\n",
"\n",
"# SAGE archive\n",
"# NSF NGF Seismic Archive\n",
"URL = \"http://service.iris.edu/fdsnws/dataselect/1/query?\"\n",
"\n",
"# function to get authorization token \n",
Expand Down Expand Up @@ -165,6 +179,14 @@
"download_data(params, data_directory)\n"
]
},
{
"cell_type": "markdown",
"id": "00592210-5c7f-413e-adef-5f2370a010b8",
"metadata": {},
"source": [
"The function above will copy data into a location specified by the variable `data_directory`. Check in your filetree: you should have a new directory called `miniseed_data` with one `.mseed` file in it. "
]
},
{
"cell_type": "markdown",
"id": "729eab32",
Expand All @@ -182,13 +204,13 @@
"\n",
"> s3:ncedc-pds/continuous_waveforms/BK/2022/2022.231/MERC.BK.HNZ.00.D.2022.231\n",
"\n",
"- s3 - service name\n",
"- ncedc-pds - bucket name\n",
"- continuous_waveforms - prefix\n",
"- BK - (prefix) seismic network name \n",
"- 2022 - (prefix) year \n",
"- 2022.231 - (prefix) year and day of year\n",
"- MERC.BK.HNZ.00.D.2022.231 - (key) station.network.channel.location.year.day of year\n",
"- `s3` - service name\n",
"- `ncedc-pds` - bucket name\n",
"- `continuous_waveforms` - prefix\n",
"- `BK` - (prefix) seismic network name \n",
"- `2022` - (prefix) year \n",
"- `2022.231` - (prefix) year and day of year\n",
"- `MERC.BK.HNZ.00.D.2022.231` - (key) station.network.channel.location.year.day of year\n",
"\n",
"Similar to a web service file URL, the ARN is used to request data."
]
Expand Down Expand Up @@ -273,7 +295,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.5"
"version": "3.12.12"
}
},
"nbformat": 4,
Expand Down
33 changes: 12 additions & 21 deletions ...als/fundamentals/5_seismic_exercise.ipynb → ...ials/foundations/3_seismic_exercise.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"source": [
"## Exercise A: Download miniSEED files from dataselect\n",
"\n",
"This exercise is a script for downloading seismic data from EarthScope in the miniSEED file format. All the functions are covered in the previous `4_seismic.ipynb` notebook. The download function builds a SAGE web service request. Fill in the blanks to complete the script. Download earthquake event data by the query parameters for the SAGE web service. You can use the example in the previous notebook or use another event and station. \n",
"This exercise is a script for downloading seismic data from EarthScope in the miniSEED file format. All the functions are covered in the previous `2_seismic_data_example.ipynb` notebook. The download function builds a web service request. Fill in the blanks to complete the script. Download earthquake event data by the query parameters for the web service. You can use the example in the previous notebook or use another event and station. \n",
"\n",
"If the script is correctly completed you will have a new directory called `/data` containing a miniSEED file."
]
Expand Down Expand Up @@ -66,8 +66,8 @@
"client = ____\n",
"\n",
"\n",
"# Fill this with the service endpoint for SAGE data\n",
"SAGE_URL = ____\n",
"# Fill this with the service endpoint for NSF NGF Seismic data\n",
"URL = ____\n",
"\n",
"DATA_DIR = Path(\"./data\")\n",
"DATA_DIR.mkdir(parents=True, exist_ok=True)\n",
Expand All @@ -94,11 +94,11 @@
" return ____\n",
"\n",
"\n",
"# ---------- Download a file from EarthScope's web services ----------\n",
"# ---------- Define a function to download a file from EarthScope's web services ----------\n",
"# --------------------------------------------------------------------\n",
"def download_data(url, data_directory, params={}): # params is an optional query parameter\n",
" \"\"\"\n",
" Sends GET with query parameters to the SAGE web service and saves the response \n",
" Sends GET with query parameters to the NSF NGF web service and saves the response \n",
" body to a file. Expected params include: net, sta, loc, cha, start (ISO), end (ISO), etc.\n",
" \"\"\"\n",
" # get authorization token\n",
Expand Down Expand Up @@ -145,7 +145,7 @@
" print(f\"failure: {r.status_code}, {r.reason}\")\n",
" return None\n",
"\n",
"# Download a miniSEED file for an event\n",
"# Describe the parameters for a miniSEED file for an event\n",
"# \n",
"params = {\"net\" : 'IU',\n",
" \"sta\" : 'ANMO',\n",
Expand All @@ -154,17 +154,8 @@
" \"start\": '2010-02-27T06:30:00',\n",
" \"end\": '2010-02-27T10:30:00'}\n",
"\n",
"# Try the parameterized request\n",
"download_data(____, DATA_DIR, ___)\n",
"\n",
"# Download a RINEX file\n",
"year = 2025\n",
"day = 1\n",
"station = 'p034'\n",
"doy = '%03d'.format(day)\n",
"compression = 'd.Z' # or \".Z\" / \"\" depending on the archive\n",
"url = create_url(year, ____, station, compression)\n",
"download_data(url, ____)"
"# Use the function you created above to execute the parameterized request\n",
"download_data(____, DATA_DIR, ___)\n"
]
},
{
Expand Down Expand Up @@ -236,8 +227,8 @@
"# ---------- Create an EarthScope client to get token ----------\n",
"client = EarthScopeClient()\n",
"\n",
"# Fill this with the service endpoint for SAGE data\n",
"SAGE_URL = \"http://service.iris.edu/fdsnws/dataselect/1/query?\"\n",
"# Fill this with the service endpoint for NSF NGF Seismic data\n",
"URL = \"http://service.iris.edu/fdsnws/dataselect/1/query?\"\n",
"\n",
"# create a directory for rinex data\n",
"DATA_DIR = \"./data\"\n",
Expand All @@ -262,7 +253,7 @@
" return {\"authorization\": f\"Bearer {token}\"}\n",
"\n",
"\n",
"# ---------- Download a file from EarthScope's web services ----------\n",
"# ---------- Define a function to download a file from EarthScope's web services ----------\n",
"# --------------------------------------------------------------------\n",
"def download_data(url, data_directory, params={}):\n",
" \"\"\"\n",
Expand Down Expand Up @@ -317,7 +308,7 @@
" \"end\": '2010-02-27T10:30:00'}\n",
"\n",
"# Try the parameterized request\n",
"download_data(SAGE_URL, DATA_DIR, params)\n",
"download_data(URL, DATA_DIR, params)\n",
"</PRE>\n",
"```"
]
Expand Down
Loading