fix: preserve dataframe context across pages (#73)

Author: karlwaldman

CHANGELOG.md

@@ -7,6 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- Historical DataFrame helpers now accept a `per_page` value from 1 to 1000
+  and fetch all pages automatically. The `client.prices.to_dataframe(...)`
+  convenience path forwards the same option for date-range queries.
+
+### Fixed
+
+- Preserve each API record's currency and unit in current and historical
+  DataFrames instead of labeling a missing currency as USD.
+- Remove exact duplicate records introduced by overlapping page boundaries,
+  stop safely on empty pages with stale continuation metadata, and return a
+  stable schema for empty historical DataFrames.
+
 ## [1.11.0] - 2026-07-19
 
 ### Changed

README.md

@@ -137,6 +137,31 @@ print(
 Use the raw first-request pattern when downstream logic requires the exact
 source and timestamp-field semantics from the API response.
 
+## Complete pandas DataFrames
+
+Install the optional pandas support, then request a historical DataFrame:
+
+```python
+df = client.historical.to_dataframe(
+    commodity="BRENT_CRUDE_USD",
+    start="2026-01-01",
+    end="2026-06-30",
+    per_page=500,
+)
+```
+
+`to_dataframe()` fetches every page automatically. `per_page` controls the
+request page size, not the total result size, and must be an integer from 1 to
+1000. The DataFrame preserves each API row's `currency` and `unit`; a missing
+currency remains missing rather than being labeled USD. Exact records repeated
+by an overlapping page boundary are returned once, while distinct records are
+retained. Empty results have a stable schema with a `date` index.
+
+The same `per_page` behavior applies to date-range queries through
+`client.prices.to_dataframe(...)`. See the
+[DataFrames and pagination guide](docs/DATAFRAMES.md) for the complete
+contract.
+
 ## Recovery
 
 The package exposes typed errors for the customer-recoverable boundaries:

docs/DATAFRAMES.md

@@ -0,0 +1,80 @@
+# DataFrames and pagination
+
+The pandas helpers preserve the source context returned by the API and fetch
+complete result sets without requiring a manual page loop.
+
+Install the optional dependency:
+
+```bash
+python -m pip install "oilpriceapi[pandas]"
+```
+
+## Historical data
+
+```python
+from oilpriceapi import OilPriceAPI
+
+client = OilPriceAPI()
+df = client.historical.to_dataframe(
+    commodity="BRENT_CRUDE_USD",
+    start="2026-01-01",
+    end="2026-06-30",
+    interval="daily",
+    per_page=500,
+)
+```
+
+The historical DataFrame contract is:
+
+- Every page is fetched automatically until the API reports completion.
+- `per_page` is an integer from 1 to 1000 and defaults to 500.
+- `per_page` controls each request; it does not limit the total rows returned.
+- `currency` and `unit` are taken from each API record. A missing currency
+  remains null and is never converted to USD.
+- An exact record repeated on a later, overlapping page is emitted once.
+  Distinct records, including records sharing a timestamp, remain intact.
+- An empty page terminates pagination even if stale metadata says another page
+  exists.
+- The result is sorted by its `date` index. Empty results retain the columns
+  `commodity`, `value`, `currency`, `unit`, and `type_name`.
+
+The convenience resource uses the same pagination behavior:
+
+```python
+df = client.prices.to_dataframe(
+    commodity="EU_CARBON_EUR",
+    start="2026-01-01",
+    end="2026-06-30",
+    per_page=250,
+)
+```
+
+For model objects instead of pandas, use `client.historical.get_all(...)`.
+It accepts the same `per_page` range and automatically fetches every page.
+
+## Current prices
+
+Calling `client.prices.to_dataframe()` with no commodity returns all current
+prices and automatically follows the API's pagination headers:
+
+```python
+df = client.prices.to_dataframe(per_page=250)
+```
+
+Each row keeps the API-provided currency and unit. Exact records repeated on a
+later page are returned once.
+
+## Manual page control
+
+Use `client.historical.get(...)` when a single page is intentional:
+
+```python
+page = client.historical.get(
+    commodity="BRENT_CRUDE_USD",
+    page=2,
+    per_page=100,
+)
+```
+
+Use `client.historical.iter_pages(...)` to process complete results one page
+at a time without retaining the entire response in memory.

docs/index.md

@@ -57,18 +57,25 @@ Access years of historical price data for backtesting and analysis:
 
 ```python
 # Get historical data
-df = client.prices.to_dataframe(
-    commodity="BRENT_CRUDE_USD",
-    start="2024-01-01",
-    end="2024-12-31",
-    interval="daily"
-)
-
-# Analyze trends
-print(df.describe())
-```
-
-**[Learn about historical endpoints →](https://docs.oilpriceapi.com/api-reference/historical)**
+df = client.prices.to_dataframe(
+    commodity="BRENT_CRUDE_USD",
+    start="2024-01-01",
+    end="2024-12-31",
+    interval="daily",
+    per_page=500
+)
+
+# Analyze trends
+print(df.describe())
+```
+
+The DataFrame helper fetches every page and preserves the `currency` and
+`unit` returned for each record. `per_page` may be set from 1 to 1000 and
+controls request size rather than total results. See
+**[DataFrames and pagination →](DATAFRAMES.md)** for empty-result and page
+boundary behavior.
+
+**[Learn about historical endpoints →](https://docs.oilpriceapi.com/api-reference/historical)**
 
 ### Technical Analysis
 

mkdocs.yml

@@ -15,6 +15,7 @@ theme:
 
 nav:
   - Home: index.md
+  - DataFrames and Pagination: DATAFRAMES.md
   - Performance Guide: PERFORMANCE_GUIDE.md
   - API Reference:
       - Client: reference/client.md

oilpriceapi/_pagination.py

@@ -0,0 +1,15 @@
+"""Shared pagination validation for public SDK helpers."""
+
+MIN_PAGE_SIZE = 1
+MAX_PAGE_SIZE = 1000
+
+
+def validate_page_size(per_page: int) -> int:
+    """Return a valid API page size or fail before making a request."""
+    if (
+        isinstance(per_page, bool)
+        or not isinstance(per_page, int)
+        or not MIN_PAGE_SIZE <= per_page <= MAX_PAGE_SIZE
+    ):
+        raise ValueError(f"per_page must be an integer between {MIN_PAGE_SIZE} and {MAX_PAGE_SIZE}")
+    return per_page