Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
7ec075a
Update python to 3.9+ and Poetry to 1.0+
emmleroy Jun 24, 2026
1895093
Remove deprecated is_datetime_or_timedelta_dtype from pandas
emmleroy Jun 24, 2026
cdd235d
Bump actions/cache to v4 and release-drafter to v6
emmleroy Jun 24, 2026
68e06db
Bump github runner image to ubuntu-latest
emmleroy Jun 24, 2026
67f04d1
Add Python 3.11 to gh-actions version matrix
emmleroy Jun 24, 2026
4070047
Bump version from 0.5.0 to 1.0.0rc0
emmleroy Jun 24, 2026
2954ed8
Remove redundant timestamp re-conversion
emmleroy Jun 24, 2026
df5cefc
Merge pull request #28 from quant-aq/emmleroy-sc-20241-update-poetry-…
emmleroy Jun 24, 2026
a876d71
Add modulair-X rawsd and cloudapi data to tests
emmleroy Jun 25, 2026
2d9aacf
Merge pull request #29 from quant-aq/emmleroy-sc-20243-test-X-series
emmleroy Jun 25, 2026
019e9d0
Port david's resampling function
emmleroy Jun 26, 2026
f897777
Let exceptions propagate
emmleroy Jun 26, 2026
5366b8e
Add loguru as dependency
emmleroy Jun 26, 2026
45849c5
Postpone evaluation of annotations for compatibility with Python 3.9
emmleroy Jun 26, 2026
c144360
Add logging for edge case where u/v cols are present but nan'd
emmleroy Jun 29, 2026
9046c41
Add more resampling tests for variety of quantaq products
emmleroy Jun 29, 2026
577eb00
Fix: remove all Unnamed columns in safe_load()
emmleroy Jun 29, 2026
8b95643
Remove relative imports
emmleroy Jun 29, 2026
9235f1a
Merge pull request #31 from quant-aq/emmleroy-sc-20239-modify-resample
emmleroy Jun 30, 2026
6c74da7
Use namedtuple for FLAG and FLAG_CRITERIA
emmleroy Jun 26, 2026
02606fb
Replace single-filter flag command with multi-filter flagging
emmleroy Jun 26, 2026
28197e1
Add util for determining the timestamp col
emmleroy Jun 26, 2026
1f8f9ba
Refactor - rename bit to flag_value, fix click commands
emmleroy Jun 29, 2026
79e26bf
Update tests
emmleroy Jun 29, 2026
003d5ad
Fix indentation
emmleroy Jun 29, 2026
d4b2c97
Remove relative imports
emmleroy Jun 30, 2026
22b7560
Remove arisense flagging functionality
emmleroy Jun 30, 2026
61f6d34
Modify safe_loader() to add device serial number to rawSD columns
emmleroy Jun 30, 2026
f0b40aa
Add func to infer data model from the sn column
emmleroy Jun 30, 2026
5ad1d1f
Add func to infer data source from the dataframe sampling frequency (…
emmleroy Jun 30, 2026
045e34b
Infer data model and source in flag_dataframe() instead of taking args
emmleroy Jun 30, 2026
d04686a
Define identical FLAG_DEFINITIONS for QuantAQ products and modifiable…
emmleroy Jun 30, 2026
e19c028
Convert model-specific FLAG_DEFINITIONS into one generic list
emmleroy Jun 30, 2026
fe018eb
Remove tests for rawSD flagging - currently not implemented
emmleroy Jun 30, 2026
30646f7
Upgrade deps to Click8.x
emmleroy Jun 30, 2026
89a2ce6
Port expunge_dataframe() and echo_flag_table() from Isoprene
emmleroy Jun 30, 2026
49c186a
Move flag_summary and echo_flag_table to flag.py and fix all_nan_colu…
emmleroy Jun 30, 2026
cae63d5
Remove arisense from test_expunge() and add some expunge tests from I…
emmleroy Jun 30, 2026
2e67857
Remove feather tests and fix --wind unpack error in resample CLI
emmleroy Jun 30, 2026
eb2c043
Remove data model and data source as required args to flag_command
emmleroy Jul 1, 2026
04d34ed
Remove data model and data source as required args in cli logic and t…
emmleroy Jul 1, 2026
9e729a7
Fix cli args again
emmleroy Jul 1, 2026
cf1fe9a
Add jupyter for playbook notebooks
emmleroy Jul 1, 2026
61ccdea
Rename quantaq_cli to quantaq_py
emmleroy Jul 1, 2026
7d09cc4
Add logging configuration file
emmleroy Jul 1, 2026
b11c7c5
Create importable concat_files() function and move concat_command() t…
emmleroy Jul 1, 2026
554d0fe
Create importable merge_files() function and move merge_command() to …
emmleroy Jul 1, 2026
6afab69
Create importable resample_dataframe() function and move resample_com…
emmleroy Jul 1, 2026
736358d
Add importable flag_dataframe() and echo_flag_table(), move flag_comm…
emmleroy Jul 1, 2026
6db884c
Create importable expunge_dataframe(), move expunge_command to cli.py
emmleroy Jul 1, 2026
ace9101
Create importable clean_file(), move clean_command to cli.py
emmleroy Jul 1, 2026
134f7b0
Remove feather functionality for parquet
emmleroy Jul 1, 2026
63fee0d
Make add_sn_column an optional arg
emmleroy Jul 1, 2026
ee78d7f
Remove console/commands sub-dirs
emmleroy Jul 1, 2026
4f332ba
Add tests for parquet file support
emmleroy Jul 1, 2026
bfe66fb
Move library function into quantaq_py/toolkit
emmleroy Jul 1, 2026
c50de58
Add pandera
emmleroy Jul 2, 2026
a625733
Add Pandera schema checks, always add sn column when safe_loading raw…
emmleroy Jul 2, 2026
9966b89
Define dtypes and schema checks in schema.py
emmleroy Jul 2, 2026
8162df4
Remove tests for arisense
emmleroy Jul 2, 2026
349b388
Update docstrings with types, add helper for fixing timestamps
emmleroy Jul 2, 2026
2ca1564
Add MOD-UFP to sn_to_model docstrings and tests
emmleroy Jul 2, 2026
d838aaa
Infer data source based on the frequency of tdiffs, add tests for mod…
emmleroy Jul 2, 2026
583a903
Fix: only local timezone if tscol is timestamp or timestamp_iso, and …
emmleroy Jul 2, 2026
84d4fc0
Fix: raise error if no files found before fixing timestamps and log e…
emmleroy Jul 2, 2026
44a89e5
Remove echo_flag_table() from flag_command, rely on logging in _add_f…
emmleroy Jul 2, 2026
7cf009c
Create a save_file utility function
emmleroy Jul 2, 2026
e739f6d
Passed desired suffixes for overlapping columns, and add option to ke…
emmleroy Jul 2, 2026
a49159a
Fix: merge_df bug and add error for attempting to merge >2 files
emmleroy Jul 2, 2026
1e1d347
Ensure all errors are logged and that errors are referenced when raised
emmleroy Jul 6, 2026
c3a2060
More informative docstring for clean_dataframe()
emmleroy Jul 6, 2026
405bc1d
Move drop_unnamed() to a helper func used in safe_load and clean_data…
emmleroy Jul 6, 2026
69b39e2
Update docstring
emmleroy Jul 6, 2026
1311341
Specify that u/v columns don't need be present in dataframe, but the …
emmleroy Jul 6, 2026
537bb1e
Add intercept handler to route standard logging calls through loguru
emmleroy Jul 6, 2026
e981a49
Rename quantaq_py to quantaq_tools
emmleroy Jul 6, 2026
8839d93
Set log level for noisy loggers to WARNING
emmleroy Jul 8, 2026
a294b6a
Add init file
emmleroy Jul 8, 2026
06e2356
Undo package rename
emmleroy Jul 8, 2026
3201cd4
Merge pull request #32 from quant-aq/emmleroy-sc-20240-library-sc-202…
emmleroy Jul 8, 2026
127adb4
Update deprecated error msg --> safe_load always adds sn to dataframe
emmleroy Jul 9, 2026
947a3cc
fix: mod-X db file mistakenly labeled as cloudAPI -- added tests
emmleroy Jul 9, 2026
4a07866
Define separate FLAG_CRITERIA for database, cloudAPI, and rawSD
emmleroy Jul 9, 2026
8d0f92e
Fix: only add FLAG_STARTUP for modulair and modulair-x devives
emmleroy Jul 9, 2026
74b98c8
Add remaining flags (FLAG_OPC, FLAG_OVERHEATED, FLAG_NEPH, gas FLAGS)…
emmleroy Jul 10, 2026
e8d9876
Implement optional flag-aware resampling that follows the firmware ag…
emmleroy Jul 10, 2026
efed2d2
Fix: add error for flag-aware resampling if 'flag' not in dataframe c…
emmleroy Jul 10, 2026
f3cf262
Remove unused imports
emmleroy Jul 10, 2026
109d4f3
Add schema standardization; move safe_load to toolkit/load.py to avoi…
emmleroy Jul 13, 2026
365feac
Remove cloudAPI from flag criteria
emmleroy Jul 13, 2026
21603c5
Test different options for nonnumeric_how
emmleroy Jul 13, 2026
0de29fd
Add tests for schema migration
emmleroy Jul 13, 2026
eaccfbf
Add safe_load to list of functions that can be directly imported
emmleroy Jul 13, 2026
8f1fbf6
Add column names and dtypes to validate_schema() and call this before…
emmleroy Jul 14, 2026
e9aeffb
Merge pull request #33 from quant-aq/sdcard-reflagging-sc-20242
emmleroy Jul 14, 2026
609c38a
Attempt to update docs
emmleroy Jul 8, 2026
8b9a47c
Another attempt at making docs
emmleroy Jul 8, 2026
20072db
Actually succeed at updating docs
emmleroy Jul 8, 2026
e443b00
Undo package re-name
emmleroy Jul 13, 2026
36eaf77
Update list of supported Python versions
emmleroy Jul 13, 2026
49189e5
gitignore vscode settings
emmleroy Jul 13, 2026
8581201
Add overview of library funcs
emmleroy Jul 13, 2026
b3e23f8
fix: add optional args to merge command
emmleroy Jul 13, 2026
14cfe15
Update usage docs
emmleroy Jul 13, 2026
2a9fe0b
Remove unused loguru levels
emmleroy Jul 14, 2026
d14bc68
Change default suffixes for keep=left/left to empty string (columns r…
emmleroy Jul 14, 2026
af3461e
Format usage.rst
emmleroy Jul 14, 2026
f6b323f
Delete test log files
emmleroy Jul 14, 2026
92504b5
Update log usage without renamed package
emmleroy Jul 14, 2026
4b20e02
Add logging configs and imports
emmleroy Jul 14, 2026
f142999
Allow imports from toolkit module
emmleroy Jul 14, 2026
156b4f9
Don't coerce dtypes when expunging and move flag nan check to flag_da…
emmleroy Jul 14, 2026
aa77d34
Don't coerce dtypes when expunging and move flag nan check to flag_da…
emmleroy Jul 14, 2026
28deb3b
For duplicated columns after merging, collapse if the duplicated colu…
emmleroy Jul 14, 2026
b8b9825
Modify merge tests to account for collapsed duplicated, identical col…
emmleroy Jul 14, 2026
5c0d811
Create example notebook for using the Python library
emmleroy Jul 14, 2026
f1002fc
Fix docstring indentation issue
emmleroy Jul 14, 2026
c549454
Fix docstring indentation issue
emmleroy Jul 14, 2026
05027dc
Don't cross-reference options
emmleroy Jul 14, 2026
e4b29e1
Re-run notebook
emmleroy Jul 14, 2026
3ecaf2a
Move merge columns assertion to a helper func; and undo no safe_loadi…
emmleroy Jul 14, 2026
e8666b0
Add tests for keep left/right
emmleroy Jul 14, 2026
83d7cdb
Don't use default merge test assertion if keep=left/right
emmleroy Jul 14, 2026
8cb2013
Remove extra cells
emmleroy Jul 14, 2026
bbf3120
Fix indentation issues again
emmleroy Jul 14, 2026
1bbdad5
Re-introduce default schema validation for expunge and flag
emmleroy Jul 14, 2026
4f9531b
Add test for nan flags
emmleroy Jul 14, 2026
4f74cc7
Modify public vs private functions
emmleroy Jul 14, 2026
8cbf4e7
Remove extra cells again
emmleroy Jul 14, 2026
cf5d278
fix: fix cli suffixes specification and add tests for custom suffixes
emmleroy Jul 14, 2026
c46a7a2
Re-order imports
emmleroy Jul 14, 2026
14da369
Move jupyter to dev deps
emmleroy Jul 15, 2026
82f39cc
Bump poetry version to 1.8.4 in gh-actions
emmleroy Jul 15, 2026
23f2ab6
Add a docs preview artifact for PRs
emmleroy Jul 15, 2026
bdf0106
Merge pull request #34 from quant-aq/emmleroy-sc-20244-update-docs
emmleroy Jul 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/build-and-deploy-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ on:
jobs:
build-docs:
name: Build docs and push to gh-pages
runs-on: ubuntu-18.04
runs-on: ubuntu-latest
steps:
- name: Checkout branch
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup python
uses: actions/setup-python@master
Expand All @@ -19,7 +19,7 @@ jobs:
- name: Install poetry
uses: snok/install-poetry@v1.3.3
with:
version: 1.2.2
version: 1.8.4
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true
Expand Down
53 changes: 53 additions & 0 deletions .github/workflows/docs-preview.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: docs/preview
on:
pull_request:
paths:
- 'docs/**'
- '**.py'

jobs:
build-docs-preview:
name: Build docs preview
runs-on: ubuntu-latest
steps:
- name: Checkout branch
uses: actions/checkout@v4

- name: Setup python
uses: actions/setup-python@v5
with:
python-version: 3.10.5

- name: Install poetry
uses: snok/install-poetry@v1.3.3
with:
version: 1.8.4
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true

- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v4
with:
path: .venv
key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}

- name: Install dependencies
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
run: poetry install --no-interaction --no-root

- name: Build docs
run: |
source .venv/bin/activate
cd docs
make clean
make html
cd ..

- name: Upload docs artifact
uses: actions/upload-artifact@v4
with:
name: docs-preview-${{ github.event.pull_request.number }}
path: docs/_build/html
retention-days: 7
2 changes: 1 addition & 1 deletion .github/workflows/release-drafter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
update_release_draft:
runs-on: ubuntu-latest
steps:
- uses: release-drafter/release-drafter@v5
- uses: release-drafter/release-drafter@v6

env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
6 changes: 3 additions & 3 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ on:
jobs:
build-n-publish:
name: Build and publish to PyPI
runs-on: ubuntu-18.04
runs-on: ubuntu-latest
steps:
- name: Checkout branch
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup python
uses: actions/setup-python@v4
Expand All @@ -19,7 +19,7 @@ jobs:
- name: Install poetry
uses: snok/install-poetry@v1.3.3
with:
version: 1.2.2
version: 1.8.4
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/test-and-verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python: ["3.8", "3.9", "3.10"]
python: ["3.9", "3.10", "3.11"]
name: Python ${{ matrix.python }} tests
steps:
- name: Checkout branch
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup Python ${{ matrix.python }}
uses: actions/setup-python@master
Expand All @@ -19,14 +19,14 @@ jobs:
- name: Install poetry
uses: snok/install-poetry@v1
with:
version: 1.2.2
version: 1.8.4
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true

- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v2
uses: actions/cache@v4
with:
path: .venv
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,9 @@ venv.bak/
.spyderproject
.spyproject

# VScode workspace settings
.vscode

# Rope project settings
.ropeproject

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ Full documentation can be found [here](https://quant-aq.github.io/cli/).

## Dependencies

This tool is built for Python 3.6.1+ and has the following key dependencies
This tool is built for Python 3.9+ and has the following key dependencies

```
python = ">=3.8,<4.0"
pandas = ">=1.0.4"
python = ">=3.9, <3.14"
pandas = ">=2.1.4,<2.2"
```

More details can be found in the `pyproject.toml` file at the base of this repository.
Expand Down
58 changes: 56 additions & 2 deletions docs/api.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,60 @@
API Reference
=============

.. click:: quantaq_cli.console:main
CLI
---

.. click:: quantaq_cli.cli:main
:prog: quantaq-cli
:show-nested:
:nested: full

quantaq_cli.toolkit
----------------------

concat
~~~~~~

.. automodule:: quantaq_cli.toolkit.concat
:members:
:undoc-members:
:show-inheritance:

expunge
~~~~~~~

.. automodule:: quantaq_cli.toolkit.expunge
:members:
:undoc-members:
:show-inheritance:

flag
~~~~

.. automodule:: quantaq_cli.toolkit.flag
:members:
:undoc-members:
:show-inheritance:

merge
~~~~~

.. automodule:: quantaq_cli.toolkit.merge
:members:
:undoc-members:
:show-inheritance:

munge
~~~~~

.. automodule:: quantaq_cli.toolkit.munge
:members:
:undoc-members:
:show-inheritance:

resample
~~~~~~~~

.. automodule:: quantaq_cli.toolkit.resample
:members:
:undoc-members:
:show-inheritance:
9 changes: 8 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,18 @@


# -- General configuration ---------------------------------------------------
autosummary_generate = True

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.autosummary',
'sphinx.ext.napoleon', # google-style docstrings
'sphinx_click.ext',
'sphinx.ext.intersphinx',
'recommonmark',
'myst_parser',
"sphinx_rtd_theme"
]

Expand All @@ -56,6 +58,11 @@
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']

napoleon_google_docstring = True
napoleon_numpy_docstring = False
napoleon_include_init_with_doc = True
napoleon_use_param = True
napoleon_use_rtype = True

# -- Options for HTML output -------------------------------------------------

Expand Down
4 changes: 2 additions & 2 deletions docs/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ Testing

$ poetry run pytest -rP

We support a number of Python versions. Currently, we support and test for 3.8,
3.9, and 3.10. These tests are automated and run as GitHub actions on every
We support a number of Python versions. Currently, we support and test for 3.9,
3.10, and 3.11. These tests are automated and run as GitHub actions on every
pull request.

.. _issue tracker: https://github.com/quant-aq/cli/issues
Expand Down
Binary file modified docs/flag-output.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/flag-output2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 2 additions & 4 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
quantaq-cli
============

**QuantAQ-CLI** provides an easy-to-use command line interface for munging and
interacting with QuantAQ air quality sensor data.
**QuantAQ-CLI** provides an easy-to-use Python library and command line
interface for munging and interacting with QuantAQ air quality sensor data.


.. toctree::
Expand All @@ -17,5 +17,3 @@ interacting with QuantAQ air quality sensor data.
usage
contributing
api


Loading
Loading