onsrap

A simple Pipeline orchestration package.

Where this documentation refers to the root folder we mean where this README.md is
located.

What is onsrap?

Add a summary of your project here.

Getting started

To start using this project, first make sure your system meets its requirements.

It's suggested that you install this package and its requirements within a virtual environment.

Requirements

• Python 3.9+ installed
• a .secrets file with the required secrets and credentials
• to have loaded environment variables from .env

Contributors have some additional requirements - please see our contributing guidance.

Installing the package

Whilst in the root folder, in a terminal, you can install the package and its Python dependencies using:

python -m pip install -U pip setuptools
pip install -e .

Install for contributors (Python only)

To install the contributing requirements, use:

python -m pip install -U pip setuptools
pip install -e .[dev]
pre-commit install

This installs an editable version of the package. This means that when you update the package code you do not have to reinstall it for the changes to take effect. This saves a lot of time when you test your code.

Remember to update the setup and requirement files inline with any changes to your package.

Running the pipeline (Python only)

The main runnable example now lives in examples/pipeline_1/main.py. It builds a three-stage pipeline from numbered scripts under examples/pipeline_1/scripts/. To run the example, use:

python examples/pipeline_1/main.py

Alternatively, most Python IDEs allow you to run the code directly using a run button.

Required secrets and credentials

To run this project, you need a .secrets file with secrets/credentials as environmental variables. The secrets/credentials should have the following environment variable name(s):

Secret/credential	Environment variable name	Description
Secret 1	SECRET_VARIABLE_1	Plain English description of Secret 1.
Credential 1	CREDENTIAL_VARIABLE_1	Plain English description of Credential 1.

Once you've added them, load these environment variables using .env.

Project structure layout

The cookiecutter template generated for each project will follow this folder structure:

.
├── onsrap/
│   ├── data/
│   │   ├── raw/
│   │   ├── interim/
│   │   └── processed/
│   └── onsrap/
│       ├── example_modules/
│       │   ├── __init__.py
│       │   └── example_module.py
│       ├── __init__.py
│       ├── example_config.yml
│       └── run_pipeline.py
└── ...

Licence

Unless stated otherwise, the codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation. The documentation is © Crown copyright and available under the terms of the Open Government 3.0 licence.

Contributing

If you want to help us build and improve onsrap, please take a look at our contributing guidelines.

Acknowledgements

This project structure is based on the govcookiecutter template project.