BEA Economic Data Pipeline

A Python automation pipeline that pulls economic datasets from the Bureau of Economic Analysis (BEA) REST API, writes timestamped snapshots to disk as CSV and Pickle, and uploads them to AWS S3.

Demonstrates practical patterns for:

• Wrapping an external REST API with clean error handling and type hints
• Managing cloud storage (S3) via boto3 with full create / list / upload / teardown lifecycle
• Parameterized CLI tooling (argparse) in place of hardcoded config
• Integration testing with mocked AWS (moto) and mocked HTTP (unittest.mock) — no live credentials required
• Containerized dev environment with LocalStack for local S3 (no AWS account needed to develop)
• GitHub Actions CI pipeline: lint → test → docker build, gated in sequence

Highlights

• Fails loudly, never silently: bad API key, BEA errors, network or S3 failures exit non-zero with a clear message (no tracebacks, no false success)
• Safe requests: query parameters are URL-encoded, so user input can't inject extra parameters
• Region-aware S3: works outside us-east-1; existence checks use HeadBucket (no account-wide ListBuckets permission needed)
• 100% test coverage with mocked AWS (moto) and BEA API; tests stay hermetic even inside the LocalStack devcontainer
• Production container: multi-stage build, pinned runtime deps only, non-root user, read-only filesystem
• CI on Python 3.11–3.14 plus a Docker build and smoke test; Dependabot keeps deps current

---

Quick Start

make            # list every task
make check      # create .venv, then lint + typecheck + tests
make docker-run BUCKET=my-bucket   # build and run the production image (uses .env)

Fastest path — devcontainer (recommended):

git clone <repo>
cd sample_code_2

cp example.env .env
# Add your BEA_API_KEY to .env (free — see Prerequisites below)

# Open in Cursor or VS Code → Cmd+Shift+P → "Dev Containers: Reopen in Container"
# LocalStack S3 starts automatically. AWS credentials are pre-configured.

Without Docker:

cp example.env .env          # fill in BEA_API_KEY and AWS credentials
make check
make run BUCKET=my-bucket DATASET=gdp YEAR=2023

---

Project Structure

sample_code_2/
├── sample_2/
│   ├── main.py               # CLI entrypoint (argparse)
│   └── libs/
│       ├── utils.py          # BEA_Wrapper — REST API client + DataFrame output
│       └── storages.py       # AwsS3 — create / upload / list / delete
├── tests/
│   ├── conftest.py           # keeps AWS env hermetic for every test
│   └── test_sample.py        # 30 tests: BEA API, S3 (moto), CLI exit codes
├── .devcontainer/
│   ├── Dockerfile            # Python 3.14 + Docker CLI + AWS CLI
│   ├── docker-compose.yml    # devcontainer service + LocalStack S3
│   └── scripts/
│       ├── post-create.sh    # installs deps, configures AWS CLI, seeds S3 bucket
│       └── devcontainer-ps1.sh  # custom PS1: repo | branch | path
├── .github/workflows/ci.yml  # lint → test → docker build
├── Dockerfile-{dev,prod}
├── docker-compose.yml        # app profiles: dev / prod
├── Makefile
├── pyproject.toml            # packaging, ruff, mypy, pytest config
└── .pre-commit-config.yaml

---

Architecture

BEA REST API  (https://apps.bea.gov/api/data)
      │
      ▼
BEA_Wrapper                      sample_2/libs/utils.py
      │  list_datasets()          → DataFrame of all available datasets
      │  fetch_gdp_by_industry()  → DataFrame of GDP by industry for a given year
      │
      ▼
Local disk                        ./data/<timestamp>.<dataset>.{csv,pkl}
      │
      ▼
AwsS3                            sample_2/libs/storages.py
      │  create_bucket()
      │  copy_to_bucket()
      │  list_bucket_files()
      │  delete_bucket_and_contents()
      │
      ▼
AWS S3  (or LocalStack in dev)

---

Stack

Layer	Tech
Language	Python 3.11+
HTTP client	requests
Data	pandas
Cloud storage	boto3 / AWS S3
Local AWS	LocalStack 3 (S3)
Packaging	Docker (python:3.14-slim, multi-stage), docker compose
Testing	pytest, moto (S3), unittest.mock (BEA API)
Linting	ruff (lint + format), mypy, pre-commit
CI/CD	GitHub Actions
Dev environment	VS Code / Cursor devcontainer (Debian, Python 3.14)

---

Prerequisites

BEA API key (required, free)

1. Register at apps.bea.gov/API/signup
2. Click the activation link in the confirmation email
3. Add the key to .env:

   BEA_API_KEY=your_key_here
   

AWS credentials

Devcontainer — not required. LocalStack provides a local S3 endpoint and credentials are pre-configured as test/test.

Local / production — add real credentials to .env:

AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_DEFAULT_REGION=us-east-1

---

Usage

CLI

# Fetch GDP by Industry for 2023 → save locally → upload to S3
python -m sample_2.main --bucket my-bucket --dataset gdp --year 2023

# List all available BEA datasets instead
python -m sample_2.main --bucket my-bucket --dataset datasets

# Change output directory
python -m sample_2.main --bucket my-bucket --dataset gdp --dest /tmp/bea

# Tear down the S3 bucket and all objects
python -m sample_2.main --bucket my-bucket --delete-bucket

Flag	Default	Description
--bucket	(required)	S3 bucket name
--dest	./data	Local output directory
--dataset	gdp	gdp or datasets
--year	2023	Year for GDP data
--delete-bucket	—	Delete bucket and all objects, then exit

Docker (app images)

make docker-build # build the production image
make docker-run BUCKET=my-bucket  # run it with .env
make docker-test  # lint + typecheck + tests in the dev image

---

Testing

make test

No live AWS credentials or BEA API key needed. All external calls are mocked.

Class	What's tested
TestBEAWrapper	list_datasets() parses response into DataFrame; request URL is HTTPS and contains no & entities; fetch_gdp_by_industry() extracts rows correctly
TestAwsS3	create_bucket() creates and is idempotent; copy_to_bucket() + list_bucket_files() round-trip; delete_bucket_and_contents() tears down; safe no-op on missing bucket

---

Devcontainer

A fully configured devcontainer is included for VS Code and Cursor. Opens into a ready-to-use Python 3.14 (Debian) environment — LocalStack S3 starts automatically, AWS CLI is pre-pointed at it, and a default bucket is created.

What's inside the image:

OS	Debian (python:3.14-slim)
Python	3.14, pip; package installed with .[dev]
AWS	CLI v2 + boto3; LocalStack S3 at http://localstack:4566
Docker	CLI + Compose plugin (mounts OrbStack / Docker socket)
Linting	ruff, mypy, pre-commit
Shell	repo | branch | path $ prompt

Open:

Cmd+Shift+P → "Dev Containers: Reopen in Container"

Or via Make:

make dc-build   # build image
make dc-up      # start devcontainer + localstack
make dc-shell   # attach a bash shell
make dc-down    # stop and remove containers

LocalStack S3 inside the container:

AWS_ENDPOINT_URL=http://localstack:4566 is set automatically — both aws CLI and boto3 use it without extra flags:

aws s3 ls                                                        # list buckets
aws s3 ls s3://sample-sized-bucket-local                        # list objects

python -m sample_2.main --bucket sample-sized-bucket-local --dataset gdp

Port 4566 is forwarded to the host, so you can also hit LocalStack from outside the container:

aws --endpoint-url http://localhost:4566 s3 ls

To switch to real AWS, set AWS_ENDPOINT_URL= (empty string) in .env.

---

Linting & pre-commit

make lint            # ruff check + ruff format --check
make lint-fix        # auto-fix everything
make pre-commit-install   # install git hooks

Hooks run on every git commit: ruff (with autofix), ruff format, whitespace and private-key checks.

Config lives in pyproject.toml ([tool.ruff], [tool.mypy], [tool.pytest.ini_options]).

---

CI/CD

GitHub Actions runs on every push and pull request to master. Jobs are gated in sequence:

test (Python 3.11–3.14)  →  docker build + smoke test

1. test — ruff, mypy, pytest with mocked AWS and BEA API (no credentials needed)
2. docker — builds Dockerfile-prod and checks --help and the clean missing-key exit

See .github/workflows/ci.yml.