From 71cb12eea2ec37892edf2d01bf94a0369576a77f Mon Sep 17 00:00:00 2001 From: Cevat Batuhan Tolon Date: Wed, 19 Aug 2026 23:15:01 +0300 Subject: [PATCH 1/3] failure events arrive natively, and a wake can no longer be lost MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit events() streams refusals without a poll: the EventNotifier parks in wait_for_event and the engine wakes it for installs and refused reloads alike; failure_poll parses, warns, and does nothing. On the way, the suite caught a wake that could be LOST: the notifier thread read its baseline generation on its own first instruction, so an install racing the thread's birth folded into the baseline and was never reported — the baseline now reads on the caller's thread under the registration lock, in both notifiers. Wheels build on engine 0.8 and the 0.8 store crates with the dev-only patch block gone, and CI runs only where review happens: on the pull request. --- .github/workflows/ci.yml | 59 +++++- .github/workflows/nightly.yml | 26 +++ .github/workflows/publish-dry-run.yml | 2 +- .github/workflows/release.yml | 23 +- .github/workflows/security.yml | 22 +- Cargo.lock | 196 ++++++++---------- Cargo.toml | 21 +- README.md | 10 +- RELEASING.md | 11 +- SECURITY.md | 20 +- book/src/async.md | 16 +- book/src/callbacks.md | 6 + book/src/limitations.md | 26 +-- book/src/patterns.md | 6 + book/src/pre-forking.md | 62 +++++- book/src/reference.md | 2 +- book/src/remote-wheel.md | 2 +- dynamic-config-python-remote/CHANGELOG.md | 6 + dynamic-config-python-remote/Cargo.toml | 6 +- dynamic-config-python/CHANGELOG.md | 37 ++++ dynamic-config-python/Cargo.toml | 2 +- .../examples/27_reload_events.py | 7 +- dynamic-config-python/examples/README.md | 2 +- .../python/dynamic_config/_config.py | 135 ++++++------ .../python/dynamic_config/_core.pyi | 5 + .../python/dynamic_config/_notify.py | 103 ++++++++- dynamic-config-python/src/config/inner.rs | 23 ++ dynamic-config-python/src/config/mod.rs | 84 ++++++++ dynamic-config-python/tests/test_notify.py | 34 ++- scripts/conformance.py | 120 +++++++++++ scripts/leak.py | 76 +++++++ 31 files changed, 875 insertions(+), 275 deletions(-) create mode 100644 .github/workflows/nightly.yml create mode 100644 scripts/conformance.py create mode 100644 scripts/leak.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6fb528b..bfd46d3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,14 +1,11 @@ name: CI -# No push trigger for `dev`: everything on dev travels through a pull -# request anyway, and a push run would twin the PR run on the same commit — -# under *required* checks, a deduplicating cancellation poisons the gate -# (a cancelled twin's "CI is green" reads as failure and blocks the merge). -# `main` keeps its push trigger because a rebase-merge mints a new SHA that -# no PR run ever checked. +# No push trigger at all: everything travels through a pull request, and +# main only ever receives a SQUASH of a green PR — the squashed tree is +# byte-identical to the PR head the checks already ran on, so a main +# re-run would test the same content twice. Merging runs release.yml +# alone; workflow_dispatch stays as the manual escape hatch. on: - push: - branches: [main] pull_request: workflow_dispatch: @@ -29,6 +26,50 @@ env: RUST_BACKTRACE: 1 jobs: + conformance: + name: the three languages agree + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + # The engine's conformance cases, at a PINNED ref: bumping the pin is + # a visible PR diff, which is the point — semantics move only when + # somebody says so. + - run: | + ref="v0.7.1" # the engine release these wheels build against + curl -sSL "https://github.com/dynamic-config-rs/dynamic-config/archive/refs/tags/${ref}.tar.gz" \ + | tar -xz --strip-components=1 --wildcards '*/conformance/' + - uses: actions/setup-python@2e3e4b15a884dc73a63f962bff250a855150a234 # v6 + with: + python-version: "3.12" + - run: pip install dynamic-config-py + - run: CONFORMANCE_DIR=conformance/cases python scripts/conformance.py + + msrv: + name: the declared floor compiles + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # master + with: + toolchain: stable + # The stores reach `etcd-client`, whose build script compiles protos. + - run: sudo apt-get update && sudo apt-get install -y protobuf-compiler + # A lockfile resolved by stable, then checked by the floor — the + # combination `pip install` + our build actually produces. This job + # did not exist before the 1.88 raise; a floor nobody compiles + # against is a number in a manifest. + - run: cargo generate-lockfile + - uses: dtolnay/rust-toolchain@6c977a6ca4077a0ceb28ffbe03f59d46e9ac8772 # master + with: + toolchain: "1.88" + - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2 + with: + shared-key: msrv + - run: cargo +1.88 check -p dynamic-config-python --locked --lib + - run: cargo +1.88 check -p dynamic-config-python-remote --locked --lib + # What changed, so the expensive jobs below can sit out a run that cannot # affect them. Deliberately a *job* rather than `on.pull_request.paths`: # a workflow filtered out at the trigger never reports at all, and a @@ -419,6 +460,8 @@ jobs: - python-free-threaded - docs-links - actionlint + - msrv + - conformance if: always() runs-on: ubuntu-latest timeout-minutes: 5 diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml new file mode 100644 index 0000000..68b3c13 --- /dev/null +++ b/.github/workflows/nightly.yml @@ -0,0 +1,26 @@ +name: Nightly + +# The binding-side leak budget: callback and notifier lifecycles are +# where leaks hide even when the Rust core is clean. The engine's +# nightly carries the 5-hour soak; these legs churn the binding's own +# machinery a million times and hold the deltas to a budget. +on: + schedule: + - cron: "23 1 * * *" + workflow_dispatch: + +permissions: + contents: read + +jobs: + leak: + name: a million reloads leak nothing + runs-on: ubuntu-latest + timeout-minutes: 90 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: actions/setup-python@2e3e4b15a884dc73a63f962bff250a855150a234 # v6 + with: + python-version: "3.12" + - run: pip install dynamic-config-py + - run: LEAK_RELOADS=1000000 python scripts/leak.py diff --git a/.github/workflows/publish-dry-run.yml b/.github/workflows/publish-dry-run.yml index d2354c3..4b978c9 100644 --- a/.github/workflows/publish-dry-run.yml +++ b/.github/workflows/publish-dry-run.yml @@ -6,7 +6,7 @@ name: Publish dry run # manifests change, not when the code does. on: push: - branches: [main, dev] + branches: [dev] paths: - "**/Cargo.toml" - "**/README.md" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3acc8e1..cabea34 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -293,22 +293,27 @@ jobs: && (needs.decide.outputs.release == 'true' || github.event_name == 'workflow_dispatch') runs-on: ubuntu-latest timeout-minutes: 15 + permissions: + # Trusted publishing: PyPI verifies THIS workflow's OIDC identity + # and mints a short-lived token — no PYPI_TOKEN to store, leak or + # rotate. Both projects (`dynamic-config-py`, + # `dynamic-config-py-remote`) carry a Trusted Publisher entry on + # PyPI naming this repository and this file; that console entry is + # a release-train prerequisite and OUTSTANDING.md spells it out. + id-token: write steps: - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: pattern: wheels-* merge-multiple: true path: dist - - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + # `skip-existing`: a version on PyPI is permanent, so the only way + # forward after a partial failure is a rerun that publishes the + # remainder. + - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 with: - python-version: "3.13" - # what is already uploaded: a version on PyPI is permanent, so the - # only way forward after a partial failure is a rerun that publishes - # the remainder. - - run: pip install maturin - - run: maturin upload --skip-existing dist/* - env: - MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }} + packages-dir: dist + skip-existing: true tag-and-release: name: tag and GitHub release diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml index eaa224e..b84d40e 100644 --- a/.github/workflows/security.yml +++ b/.github/workflows/security.yml @@ -7,8 +7,6 @@ name: Security # No push trigger for `dev` — same reasoning as ci.yml: dev travels through # pull requests, and a push twin under required checks poisons the gate. on: - push: - branches: [main] pull_request: schedule: - cron: "17 6 * * 1" @@ -23,6 +21,24 @@ permissions: contents: read jobs: + policy: + name: the security policy names the current line + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + # The stale-table drift this org has already lived through once: + # SECURITY.md said 0.0.x while the workspace shipped 0.7. The table's + # top row must name the line that is actually published. + - run: | + version=$(grep -m1 '^version = ' dynamic-config-python/Cargo.toml | cut -d'"' -f2) + line="${version%.*}.x" + if ! grep -qF "| ${line}" SECURITY.md; then + echo "::error file=SECURITY.md::the supported-versions table does not name ${line} (workspace is ${version})" + exit 1 + fi + echo "SECURITY.md names ${line} — current" + deny: name: advisories, licences, sources runs-on: ubuntu-latest @@ -149,7 +165,7 @@ jobs: # — which is the whole point of running it. security-ok: name: Security is green - needs: [deny, unsafe, osv, supply-chain] + needs: [policy, deny, unsafe, osv, supply-chain] if: always() runs-on: ubuntu-latest timeout-minutes: 5 diff --git a/Cargo.lock b/Cargo.lock index d39778e..f25823c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -114,16 +114,16 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" [[package]] name = "aws-config" -version = "1.8.11" +version = "1.8.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0149602eeaf915158e14029ba0c78dedb8c08d554b024d54c8f239aab46511d" +checksum = "c456581cb3c77fafcc8c67204a70680d40b61112d6da78c77bd31d945b65f1b5" dependencies = [ "aws-credential-types", "aws-runtime", "aws-sdk-sts", "aws-smithy-async", - "aws-smithy-http 0.62.5", - "aws-smithy-json 0.61.7", + "aws-smithy-http", + "aws-smithy-json", "aws-smithy-runtime", "aws-smithy-runtime-api", "aws-smithy-types", @@ -182,7 +182,7 @@ dependencies = [ "aws-sigv4", "aws-smithy-async", "aws-smithy-eventstream", - "aws-smithy-http 0.63.3", + "aws-smithy-http", "aws-smithy-runtime", "aws-smithy-runtime-api", "aws-smithy-types", @@ -211,8 +211,8 @@ dependencies = [ "aws-smithy-async", "aws-smithy-checksums", "aws-smithy-eventstream", - "aws-smithy-http 0.63.3", - "aws-smithy-json 0.62.3", + "aws-smithy-http", + "aws-smithy-json", "aws-smithy-observability", "aws-smithy-runtime", "aws-smithy-runtime-api", @@ -236,15 +236,16 @@ dependencies = [ [[package]] name = "aws-sdk-sts" -version = "1.94.0" +version = "1.97.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1be6fbbfa1a57724788853a623378223fe828fc4c09b146c992f0c95b6256174" +checksum = "e6443ccadc777095d5ed13e21f5c364878c9f5bad4e35187a6cdbd863b0afcad" dependencies = [ "aws-credential-types", "aws-runtime", "aws-smithy-async", - "aws-smithy-http 0.62.5", - "aws-smithy-json 0.61.7", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", "aws-smithy-query", "aws-smithy-runtime", "aws-smithy-runtime-api", @@ -253,6 +254,7 @@ dependencies = [ "aws-types", "fastrand", "http 0.2.12", + "http 1.5.0", "regex-lite", "tracing", ] @@ -265,7 +267,7 @@ checksum = "efa49f3c607b92daae0c078d48a4571f599f966dce3caee5f1ea55c4d9073f99" dependencies = [ "aws-credential-types", "aws-smithy-eventstream", - "aws-smithy-http 0.63.3", + "aws-smithy-http", "aws-smithy-runtime-api", "aws-smithy-types", "bytes", @@ -297,7 +299,7 @@ version = "0.64.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ddcf418858f9f3edd228acb8759d77394fed7531cce78d02bdda499025368439" dependencies = [ - "aws-smithy-http 0.63.3", + "aws-smithy-http", "aws-smithy-types", "bytes", "crc-fast", @@ -323,27 +325,6 @@ dependencies = [ "crc32fast", ] -[[package]] -name = "aws-smithy-http" -version = "0.62.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "445d5d720c99eed0b4aa674ed00d835d9b1427dd73e04adaf2f94c6b2d6f9fca" -dependencies = [ - "aws-smithy-runtime-api", - "aws-smithy-types", - "bytes", - "bytes-utils", - "futures-core", - "futures-util", - "http 0.2.12", - "http 1.5.0", - "http-body 0.4.6", - "percent-encoding", - "pin-project-lite", - "pin-utils", - "tracing", -] - [[package]] name = "aws-smithy-http" version = "0.63.3" @@ -390,15 +371,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "aws-smithy-json" -version = "0.61.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2db31f727935fc63c6eeae8b37b438847639ec330a9161ece694efba257e0c54" -dependencies = [ - "aws-smithy-types", -] - [[package]] name = "aws-smithy-json" version = "0.62.3" @@ -419,9 +391,9 @@ dependencies = [ [[package]] name = "aws-smithy-query" -version = "0.60.8" +version = "0.60.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d28a63441360c477465f80c7abac3b9c4d075ca638f982e605b7dc2a2c7156c9" +checksum = "0cebbddb6f3a5bd81553643e9c7daf3cc3dc5b0b5f398ac668630e8a84e6fff0" dependencies = [ "aws-smithy-types", "urlencoding", @@ -434,7 +406,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f3df87c14f0127a0d77eb261c3bc45d5b4833e2a1f63583ebfb728e4852134ee" dependencies = [ "aws-smithy-async", - "aws-smithy-http 0.63.3", + "aws-smithy-http", "aws-smithy-http-client", "aws-smithy-observability", "aws-smithy-runtime-api", @@ -773,9 +745,9 @@ dependencies = [ [[package]] name = "cookie_store" -version = "0.22.0" +version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3fc4bff745c9b4c7fb1e97b25d13153da2bc7796260141df62378998d070207f" +checksum = "15b2c103cf610ec6cae3da84a766285b42fd16aad564758459e6ecf128c75206" dependencies = [ "cookie", "document-features", @@ -977,7 +949,6 @@ version = "0.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" dependencies = [ - "powerfmt", "serde_core", ] @@ -1020,9 +991,9 @@ checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" [[package]] name = "dynamic-config" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fbdeb12a8bbb3450e8aa517b26a5dc2aed69abd54119be2333212e38f1be3c5e" +checksum = "583c63e4677f90e11d393fd86450c6e02cdd5b24bcf823f9591afa39eedce8c3" dependencies = [ "arc-swap", "dynamic-config-macros", @@ -1038,9 +1009,9 @@ dependencies = [ [[package]] name = "dynamic-config-consul" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6aa5075daa9b3ad2742520d683930421f1b0b47092412ff8b1a6be8631e5502a" +checksum = "e11823ded0b8a11400a788f8ce238938d8a15b186af38110c276b7fc50ad5fad" dependencies = [ "base64 0.23.1", "dynamic-config", @@ -1051,9 +1022,9 @@ dependencies = [ [[package]] name = "dynamic-config-etcd" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e702f764ae212e52ee1524f737224fa9f88c4628404e23e2f237af5ca556c9a" +checksum = "0863e16e125192bffe5fe99dd4575d43f9a8ae8c803096b263de8e80f6d5be01" dependencies = [ "dynamic-config", "dynamic-config-store-core", @@ -1063,9 +1034,9 @@ dependencies = [ [[package]] name = "dynamic-config-firestore" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "952204f3d8bee319a42be759fdc06fc56c8afa90cfc74504dc3efdd506e2a824" +checksum = "474e53b14085e7ec5ec4da6c7473ba9ca79fe82df2171f2e0206215932292c00" dependencies = [ "dynamic-config", "dynamic-config-store-core", @@ -1075,9 +1046,9 @@ dependencies = [ [[package]] name = "dynamic-config-git" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93aff8de36d5af90bee33ef0989327f702b4450b9ca480d014ea89c532b298b0" +checksum = "b2d7e51a28df5e04dfbe2287bd540cff089b62801c93ec428e4c1c7deff9950b" dependencies = [ "dynamic-config", "dynamic-config-store-core", @@ -1087,9 +1058,9 @@ dependencies = [ [[package]] name = "dynamic-config-macros" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a09f2c8214115242f7cd6004669cac6cfeabd32d3ba32f2e5a14522b89020e7" +checksum = "4914a2922931028329295898184036a863248e014f5fbeca861efc9e15aca395" dependencies = [ "proc-macro-crate", "proc-macro2", @@ -1099,9 +1070,9 @@ dependencies = [ [[package]] name = "dynamic-config-nats" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7c6ea6f768c8959e5c63a01d36a960aafabebeeef37b972861acdd66d39c55a" +checksum = "3b9ff7396c936480c40fc006ec2cee34ab0f074cddfe1adc3cf1538d9a84bd4e" dependencies = [ "async-nats", "dynamic-config", @@ -1141,9 +1112,9 @@ dependencies = [ [[package]] name = "dynamic-config-redis" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b14d6966377d382c531e20cb038a4612dfe11fd058218a790a8ec50b3c5fcb0" +checksum = "f5232459dd1763435e5fa1f07ebe7b6d7a98d754cd58cb5b13cf350ebaa2125e" dependencies = [ "dynamic-config", "dynamic-config-store-core", @@ -1152,9 +1123,9 @@ dependencies = [ [[package]] name = "dynamic-config-s3" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ba7a0174d6b3530132a9b5a9e784bc7890bde6485bb8189da425032d01cc648" +checksum = "f6a884d66ac17194af478492e18d3427a6eb6934c91d0901fd86f9c79fbc91ea" dependencies = [ "aws-config", "aws-sdk-s3", @@ -1167,18 +1138,18 @@ dependencies = [ [[package]] name = "dynamic-config-store-core" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3fb3568d62ec661bb9f74ad1603ba4b696b7a501c324b1ba677aeae5b4dfb2c" +checksum = "b803ed1de57336bebd49a4559b4b70f5718df84265fa42763b6790bf1f83e50e" dependencies = [ "dynamic-config", ] [[package]] name = "dynamic-config-vault" -version = "0.7.0" +version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48aae6195afd0c7cda3335da3e72e0f593ec01b0f087cea55f43a27d531b7f10" +checksum = "d9c816d3f4afb7f8ddca1389f44b6a361d323114ebc96425b5f495fd16b99117" dependencies = [ "dynamic-config", "dynamic-config-store-core", @@ -1236,7 +1207,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -2246,9 +2217,9 @@ dependencies = [ [[package]] name = "h2" -version = "0.4.16" +version = "0.4.17" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27" +checksum = "9f877e75f39e9827ec50a572dd592684ac28c029578726c85f1b2aa6ab807449" dependencies = [ "atomic-waker", "bytes", @@ -2477,12 +2448,13 @@ dependencies = [ [[package]] name = "icu_collections" -version = "2.1.1" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c6b649701667bbe825c3b7e6388cb521c23d88644678e83c0c4d0a621a34b43" +checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513" dependencies = [ "displaydoc", "potential_utf", + "utf8_iter", "yoke", "zerofrom", "zerovec", @@ -2490,9 +2462,9 @@ dependencies = [ [[package]] name = "icu_locale_core" -version = "2.1.1" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edba7861004dd3714265b4db54a3c390e880ab658fec5f7db895fae2046b5bb6" +checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb" dependencies = [ "displaydoc", "litemap", @@ -2503,9 +2475,9 @@ dependencies = [ [[package]] name = "icu_normalizer" -version = "2.1.1" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f6c8828b67bf8908d82127b2054ea1b4427ff0230ee9141c54251934ab1b599" +checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f" dependencies = [ "icu_collections", "icu_normalizer_data", @@ -2517,16 +2489,17 @@ dependencies = [ [[package]] name = "icu_normalizer_data" -version = "2.1.1" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a" +checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0" [[package]] name = "icu_properties" -version = "2.1.2" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "020bfc02fe870ec3a66d93e677ccca0562506e5872c650f893269e08615d74ec" +checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148" dependencies = [ + "displaydoc", "icu_collections", "icu_locale_core", "icu_properties_data", @@ -2537,15 +2510,15 @@ dependencies = [ [[package]] name = "icu_properties_data" -version = "2.1.2" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "616c294cf8d725c6afcd8f55abc17c56464ef6211f9ed59cccffe534129c77af" +checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa" [[package]] name = "icu_provider" -version = "2.1.1" +version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85962cf0ce02e1e0a629cc34e7ca3e373ce20dda4c4d7294bbd0bf1fdb59e614" +checksum = "92a7ed671a6aad807a8651a2e1782a6598fda9ce5185dd8158549e95a91c6428" dependencies = [ "displaydoc", "icu_locale_core", @@ -2569,9 +2542,9 @@ dependencies = [ [[package]] name = "idna_adapter" -version = "1.2.1" +version = "1.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3acae9609540aa318d1bc588455225fb2085b9ed0c4f6bd0d9d5bcd86f1a0344" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" dependencies = [ "icu_normalizer", "icu_properties", @@ -2969,7 +2942,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3422,7 +3395,7 @@ dependencies = [ "once_cell", "socket2", "tracing", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3652,7 +3625,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3711,7 +3684,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4076,7 +4049,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4171,7 +4144,7 @@ dependencies = [ "getrandom 0.4.3", "once_cell", "rustix", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4205,12 +4178,11 @@ dependencies = [ [[package]] name = "time" -version = "0.3.47" +version = "0.3.55" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" +checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134" dependencies = [ "deranged", - "itoa", "num-conv", "powerfmt", "serde_core", @@ -4220,15 +4192,15 @@ dependencies = [ [[package]] name = "time-core" -version = "0.1.8" +version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" [[package]] name = "time-macros" -version = "0.2.27" +version = "0.2.32" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" +checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85" dependencies = [ "num-conv", "time-core", @@ -4444,9 +4416,9 @@ checksum = "7d56353a2a665ad0f41a421187180aab746c8c325620617ad883a99a1cbe66d2" [[package]] name = "tonic" -version = "0.14.5" +version = "0.14.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fec7c61a0695dc1887c1b53952990f3ad2e3a31453e1f49f10e75424943a93ec" +checksum = "ac2a5518c70fa84342385732db33fb3f44bc4cc748936eb5833d2df34d6445ef" dependencies = [ "async-trait", "axum", @@ -4474,9 +4446,9 @@ dependencies = [ [[package]] name = "tonic-build" -version = "0.14.5" +version = "0.14.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1882ac3bf5ef12877d7ed57aad87e75154c11931c2ba7e6cde5e22d63522c734" +checksum = "c68f61875ac5293cf72e6c8cf0158086428c82c37229e98c840878f1706b0322" dependencies = [ "prettyplease", "proc-macro2", @@ -4486,9 +4458,9 @@ dependencies = [ [[package]] name = "tonic-prost" -version = "0.14.5" +version = "0.14.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a55376a0bbaa4975a3f10d009ad763d8f4108f067c7c2e74f3001fb49778d309" +checksum = "50849f68853be452acf590cde0b146665b8d507b3b8af17261df47e02c209ea0" dependencies = [ "bytes", "prost", @@ -4497,9 +4469,9 @@ dependencies = [ [[package]] name = "tonic-prost-build" -version = "0.14.5" +version = "0.14.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3144df636917574672e93d0f56d7edec49f90305749c668df5101751bb8f95a" +checksum = "654e5643eff75d7f8c99197ce1440ed19a3474eada74c12bbac488b2cafdae27" dependencies = [ "prettyplease", "proc-macro2", @@ -4913,7 +4885,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 16afe52..8e0fdf7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -23,21 +23,21 @@ authors = ["Cevat Batuhan Tolon "] # The engine. `default-features = false` because each crate names what it # needs: the base wheel carries every feature a Python caller can reach, # the remote wheel only what a document needs to travel. -dynamic-config = { version = "0.7", default-features = false } +dynamic-config = { version = "0.8", default-features = false } # The store crates' shared internals: the redaction that keeps a password # out of a store's description, by the same rule the store crates use # rather than by a second copy of it. -dynamic-config-store-core = "0.7" +dynamic-config-store-core = "0.8" # The eight stores. Caret, because a wheel built today should be able to # pick up a store's patch release without this repository being re-cut. -dynamic-config-consul = "0.7" -dynamic-config-etcd = { version = "0.7", features = ["tls"] } -dynamic-config-firestore = "0.7" -dynamic-config-git = "0.7" -dynamic-config-nats = "0.7" -dynamic-config-redis = { version = "0.7", features = ["tls"] } -dynamic-config-s3 = "0.7" -dynamic-config-vault = "0.7" +dynamic-config-consul = "0.8" +dynamic-config-etcd = { version = "0.8", features = ["tls"] } +dynamic-config-firestore = "0.8" +dynamic-config-git = "0.8" +dynamic-config-nats = "0.8" +dynamic-config-redis = { version = "0.8", features = ["tls"] } +dynamic-config-s3 = "0.8" +dynamic-config-vault = "0.8" # A wheel is downloaded, not compiled by whoever installs it, so the # release profile is the one that matters. @@ -46,4 +46,3 @@ lto = "thin" codegen-units = 1 strip = "symbols" -# ── 0.7 TRAIN, DEVELOPMENT ONLY ───────────────────────────────────────── diff --git a/README.md b/README.md index c38931a..295fa12 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ **Hot-reloadable configuration for Python: Rust resolves, your schema validates.** -[![CI](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/ci.yml) -[![Security](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/security.yml/badge.svg?branch=main)](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/security.yml) +[![CI](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/ci.yml/badge.svg?event=pull_request)](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/ci.yml) +[![Security](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/security.yml/badge.svg?event=pull_request)](https://github.com/dynamic-config-rs/dynamic-config-python/actions/workflows/security.yml) [![PyPI](https://img.shields.io/pypi/v/dynamic-config-py.svg)](https://pypi.org/project/dynamic-config-py/) [![Python](https://img.shields.io/pypi/pyversions/dynamic-config-py.svg)](https://pypi.org/project/dynamic-config-py/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) @@ -60,7 +60,7 @@ belongs to an unrelated package. The import is `dynamic_config`. ## The engine is a dependency, not a sibling -These crates name it with a caret (`dynamic-config = "0.6"`), so an engine +These crates name it with a caret (`dynamic-config = "0.7"`), so an engine patch release reaches them with no release here. The wheels version on their own schedule — a Rust-only release has nothing in it for a Python user, and a Python-only fix should not drag ten crates behind it. @@ -71,7 +71,7 @@ user, and a Python-only fix should not drag ten crates behind it. |---|---| | CPython | 3.9 – 3.14, one abi3 wheel per platform | | free-threaded | 3.14t, its own wheel, tested with the GIL disabled | -| MSRV (to build from source) | 1.85 — 1.88 for the remote wheel | +| MSRV (to build from source) | 1.88, both wheels | Raising either floor is a breaking change. @@ -81,6 +81,8 @@ Raising either floor is a breaking change. maturin; `just python-free-threaded /path/to/venv` is the second interpreter. +What you may build on and find unchanged tomorrow is written down: the [Compatibility Contract](https://dynamic-config-rs.github.io/compatibility.html). + ## License [MIT](LICENSE). diff --git a/RELEASING.md b/RELEASING.md index 4f818ca..cd9d07b 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -68,9 +68,14 @@ that is already on `main` — it dispatches the same workflow by hand. ## What an operator has to have ready -`PYPI_TOKEN` as a repository secret, with upload rights to both projects. -Nothing else: the wheels are built by `maturin-action` on GitHub's own -runners, and the token is the only thing that is not in this repository. +**No token.** The publish job authenticates through PyPI's Trusted +Publishing (OIDC): the one-time console entry per project — PyPI → +project → Settings → Publishing → *Add a trusted publisher* — names +owner `dynamic-config-rs`, repository `dynamic-config-python`, +workflow `release.yml`, for both `dynamic-config-py` and +`dynamic-config-py-remote`. The wheels are built by `maturin-action` +on GitHub's own runners; a leftover `PYPI_TOKEN` secret is inert and +should be revoked. ## Afterwards diff --git a/SECURITY.md b/SECURITY.md index b709ba4..a247098 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,12 +24,12 @@ the ones it explicitly does not. `Debug`; reload diffs, `check()` reports, unknown-key suggestions and *error messages* all report paths and types, never values, so nothing routes around the redaction. A report of a value appearing in a log or an error message is a -vulnerability. `dynamic-config/tests/security.rs` asserts each of these, and CI +vulnerability. the engine repository's [`tests/security.rs`](https://github.com/dynamic-config-rs/dynamic-config) asserts each of these, and its CI runs it as a job of its own. **The parsing surfaces are fuzzed.** The unit parsers, `Value` path lookup and the rule that decides whether a path touches a secret run under libFuzzer -(`fuzz/`, built on every CI run and run on a weekly schedule). A crash or a +(the engine repository's [`fuzz/`](https://github.com/dynamic-config-rs/dynamic-config), built on every CI run and run on a weekly schedule). A crash or a redaction rule that answers differently for the same path is a finding: report it through the process above, with the reproducing input attached. @@ -68,12 +68,16 @@ with your privileges. | Version | Supported | |---|---| -| 0.0.x | ✅ the latest patch | -| < 0.0.1 | — nothing older exists | +| 0.3.x | ✅ the latest patch | +| ≤ 0.2 | — end of life | -Before 1.0, fixes land on the latest published version and nothing is -backported: there is no version old enough to be worth pinning to. After 1.0, -the current and previous minor versions. +Security fixes land on the **latest patch of the line above** and +nothing is backported before 1.0: when a release ships, every prior +patch of its line is end-of-life the same day. Older toolchains resolve +older published versions through cargo's MSRV-aware resolver and are +explicitly unsupported. After 1.0, the current and previous minor +lines. The full promise lives in the engine book's +[Compatibility Contract](https://dynamic-config-rs.github.io/compatibility.html). ## Threat model, stated plainly @@ -100,7 +104,7 @@ runs on every change: | Claim | Enforced by | |---|---| -| Secrets stay out of diagnostics | `dynamic-config/tests/security.rs`, run as its own CI job | +| Secrets stay out of diagnostics | the engine repository's `tests/security.rs`, run as its own CI job there | | Files are created private, symlinks refused | `write::permissions` tests, same job | | A store cannot panic the process | `checked_add` on every server-supplied duration, plus a hostile-document test | | No unsafe code | `#![forbid(unsafe_code)]` in every crate, *and* a CI job that checks the attribute is still there | diff --git a/book/src/async.md b/book/src/async.md index f1d11f5..d596e2c 100644 --- a/book/src/async.md +++ b/book/src/async.md @@ -156,7 +156,7 @@ it, and so does anything else that provides a running loop. diagnostic one — what a log line, a metric or an alert is built from: ```python -async for event in config.events(failure_poll=1.0): +async for event in config.events(): match event: case Reloaded(generation=generation, changed=paths): log.info("config %s: %s", generation, ", ".join(paths)) @@ -169,12 +169,14 @@ async for event in config.events(failure_poll=1.0): the same rule `explain()` and `check()` follow, and for the same reason: a value in an event is a secret in a log. -`failure_poll` is what makes `ReloadFailed` possible. An install wakes -the stream; a refusal cannot, because the engine bumps no generation for -a load that installed nothing and there is nothing to be notified of. So -a stream that wants refusals asks for them, and pays one status read at -the interval it names. The default — `None` — starts no timer at all and -reports installs only. +A refusal wakes the stream natively: the engine's failure hook signals +the same parked thread an install does, so `ReloadFailed` arrives when +the refusal happens — no timer, no polling. Delivery is latest-wins: +refusals with nothing awake in between arrive as one event carrying the +current `consecutive` count, and a refusal followed by an install +arrives as both events, refusal first. (`failure_poll`, the interval +refusals were polled at before they could wake anything, is accepted, +ignored, and warns once.) ## Loading several configurations at once diff --git a/book/src/callbacks.md b/book/src/callbacks.md index 5cb8297..e10df3b 100644 --- a/book/src/callbacks.md +++ b/book/src/callbacks.md @@ -15,6 +15,12 @@ That is the whole idea. The rest is what the arguments mean, what a hook may and may not do, and the four other shapes the same thing takes. +The delivery semantics — edge-triggered, latest-wins, what a refused +reload does and does not wake — are one contract across Rust, Python +and Node, written once in the engine book's +[Change Notification](https://dynamic-config-rs.github.io/change-notification.html) +page. What follows is the Python surface of it. + ## The shapes | Shape | Runs | diff --git a/book/src/limitations.md b/book/src/limitations.md index 52d7c61..571565a 100644 --- a/book/src/limitations.md +++ b/book/src/limitations.md @@ -181,21 +181,17 @@ belongs to the client `fetch()` calls — `httpx.get(..., timeout=5)` — and `Ctrl-C` still works, because a `KeyboardInterrupt` out of a fetch propagates unchanged. -### A refused reload cannot wake anything - -The engine bumps a generation when a document *installs*. A load that -installed nothing does not, so nothing can be notified of it: a waiter -in `changed_async()` sleeps through a refusal, and `events()` reports -`ReloadFailed` only when it is given a `failure_poll` interval to check -the status at. - -That is a real gap for one case — a watcher reloading a file somebody has -just broken, where the next install is exactly the thing that is not -coming. `failure_poll=1.0` on an `events()` stream costs one status read -a second and closes it; a health endpoint reading -`status().consecutive_failures` closes it without any stream at all. -Closing it properly means a second wake channel in the engine, which is a -change to `dynamic-config` rather than to the binding. +### A `changes()` waiter sleeps through a refusal — by design + +A refused reload wakes `events()` natively (since engine 0.7.1, which +grew the second wake channel this section used to ask for). What it +still does not wake is `changes()` or `changed_async()`: those yield +*models*, a refusal installs none, and a service loop handed `None` +would be worse than one that slept. The split is the contract — +`changes()` for the values, `events()` for the diagnosis — and the +[engine book's Change Notification +page](https://dynamic-config-rs.github.io/change-notification.html) +holds it for all three languages. ### A notifier thread outlives its last waiter diff --git a/book/src/patterns.md b/book/src/patterns.md index 95f2bdf..3880fce 100644 --- a/book/src/patterns.md +++ b/book/src/patterns.md @@ -100,6 +100,12 @@ call that refuses to leave two of them new and one of them old. The group is lifecycle only. `db.current()` is still the read path, because nothing should sit between a program and its values. +One rule bounds what atomic means: the writes are all-or-nothing, the +reads are not — two `current()` calls are two reads, and a reader +landing between the group's installs can see new `db` beside old +`cache` for an instant. Two values that must always be read together +belong in **one** configuration, where a single snapshot carries both. + ## Testing without a filesystem ```python diff --git a/book/src/pre-forking.md b/book/src/pre-forking.md index 6ca4696..db17237 100644 --- a/book/src/pre-forking.md +++ b/book/src/pre-forking.md @@ -6,9 +6,17 @@ parent. Configuration survives that fork. A watcher does not. ## What each process gets +**Four workers are four engines are four watches.** There is no shared +snapshot, no master coordinating reloads, no cross-process anything: +each worker owns a full copy of the engine, reloads on its own schedule +from the same file, and none of them has to tell the others. Every +multi-process surprise below is that sentence, met somewhere +unexpected — a metric that counts one worker's reloads, a `set_override` +that patched one process, an `on_reload` hook that fired a quarter of +the times you expected. + Each worker gets its own copy of everything, including its own watcher, and -that is what you want — every worker reloads independently from the same -file, and none of them has to tell the others. +that is what you want. ```console $ gunicorn -w 4 --preload myapp:app @@ -67,3 +75,53 @@ print([t.name for t in threading.enumerate() if "dynamic-config" in t.name]) If a worker shows none and reloads never arrive, the watcher started before the fork. + +## The whole thing, runnable + +Three files; `gunicorn -c gunicorn.conf.py -w 4 --preload app:app` and +then edit `config.json` while it serves. + +```python +# config.py — the declaration, imported by both app and conf +from dataclasses import dataclass +from dynamic_config import DynamicConfig + + +@dataclass +class Database: + host: str = "localhost" + pool_size: int = 8 + + +database = DynamicConfig(Database, key="db").file("config.json") +``` + +```python +# app.py — a WSGI app; init at import time so --preload parses once +import json + +from config import database + +database.init() + + +def app(environ, start_response): + current = database.current() # this worker's snapshot, this instant + start_response("200 OK", [("Content-Type", "application/json")]) + + import os + return [json.dumps({"pid": os.getpid(), "pool": current.pool_size}).encode()] +``` + +```python +# gunicorn.conf.py — the one hook that matters +def post_fork(server, worker): + from config import database + + database.watch(debounce=0.25) # this worker's watcher, born after the fork +``` + +Edit `config.json` and every worker's next response carries the new +value — four pids, one document, no coordination. Delete the +`post_fork` hook and rerun to watch the failure mode itself: the same +edit changes nothing, in any worker, forever. diff --git a/book/src/reference.md b/book/src/reference.md index c046a1b..9b4497a 100644 --- a/book/src/reference.md +++ b/book/src/reference.md @@ -119,7 +119,7 @@ engine reads it on the load path and a load must not re-enter Python. | `replace(model)` | — | Installs a model you built, firing the hooks. `status()` and `snapshot()` still describe the last real load | | `changed(timeout=None)` | `changed_async(timeout=None)` | Blocks until the next install; `None` on timeout | | — | `changes()` | An async iterator over every install from here on | -| — | `events(failure_poll=None)` | An async iterator of [`Reloaded`](#reloaded-and-reloadfailed) and `ReloadFailed`. No event carries a value | +| — | `events()` | An async iterator of [`Reloaded`](#reloaded-and-reloadfailed) and `ReloadFailed` — refusals wake it natively. No event carries a value | | `watch(debounce=0.25, poll_interval=None)` | `watch_async(…)` | Starts a watcher; returns a [`Watch`](#watch) | | `watching(debounce=0.25, poll_interval=None)` | `watching_async(…)` | The same as a block, stopped on the way out | | `running(watch=True, …)` | `running_async(watch=True, …)` | init, then watch, then stop — the whole lifetime as one block, yielding the first model | diff --git a/book/src/remote-wheel.md b/book/src/remote-wheel.md index c3e48bf..c6598d9 100644 --- a/book/src/remote-wheel.md +++ b/book/src/remote-wheel.md @@ -55,7 +55,7 @@ clients in it. | Size (Linux x86-64, release) | 1.36 MB | 11.68 MB | | Extension module | `dynamic_config._core` | `dynamic_config_remote._core` | | tokio | never | one runtime, lazily | -| MSRV | 1.85 | 1.88 — `aws-sdk-sts`, `async-nats` and `redis` each ask for it. `gix` asks for 1.85 and so moves nothing | +| MSRV | 1.88 | 1.88 — one floor for the whole organisation since the 0.3.1 round | One wheel rather than one per store, and the size table is why. Eight build matrices is eight times the release runner for a saving nobody has diff --git a/dynamic-config-python-remote/CHANGELOG.md b/dynamic-config-python-remote/CHANGELOG.md index 96b1cdf..663ac64 100644 --- a/dynamic-config-python-remote/CHANGELOG.md +++ b/dynamic-config-python-remote/CHANGELOG.md @@ -36,6 +36,12 @@ is the door into this one. ### Changed +- **Built on the 0.8 store crates and engine 0.8** — the same + build-time bump as the base wheel, with the Python surface + unchanged. The development-only `[patch.crates-io]` block is gone. + +### Changed + - **Moved with the base wheel**, as it always does — the two are built from one commit and `dynamic-config-py[remote]` resolves to a pair. Nothing in this distribution changed; the compiled stores are the same diff --git a/dynamic-config-python-remote/Cargo.toml b/dynamic-config-python-remote/Cargo.toml index f1fa327..6d42f6e 100644 --- a/dynamic-config-python-remote/Cargo.toml +++ b/dynamic-config-python-remote/Cargo.toml @@ -11,10 +11,8 @@ name = "dynamic-config-python-remote" # declared in `pyproject.toml` rather than assumed. version = "0.3.0" edition.workspace = true -# The highest floor of anything linked in, measured rather than declared: -# PyO3, etcd and `gix` ask for 1.85, and `aws-sdk-sts`, `async-nats` and -# `redis` ask for 1.88 — so the wheel that carries all eight stores asks for -# 1.88. +# The organisation's one 1.88 floor — originally forced here anyway by +# `aws-sdk-sts`, `async-nats` and `redis`, measured rather than declared. rust-version = "1.88" license.workspace = true description = "The Rust remote stores for dynamic-config's Python bindings, as a second wheel" diff --git a/dynamic-config-python/CHANGELOG.md b/dynamic-config-python/CHANGELOG.md index 1f2da36..433799d 100644 --- a/dynamic-config-python/CHANGELOG.md +++ b/dynamic-config-python/CHANGELOG.md @@ -26,6 +26,43 @@ breaking. ## [Unreleased] +### Fixed + +- **A wake could be lost when an install raced the notifier thread's + first breath.** The thread read its baseline generation on its own + first instruction — so an install landing after a waiter's + check-register-check but before the thread was scheduled folded into + the baseline and was never reported; under load, `changed_async` + timed out on an install that had already happened. The baseline is + now read on the caller's thread, under the registration lock, before + `wait` returns — the same discipline for `events()`' notifier, which + carried the identical pattern from birth. + +### Changed + +- **Built on engine 0.8.** The engine's breaking release (a `LoadSpec` + field, MSRV 1.88) is a build-time fact here — the wheel embeds it + statically and the Python surface is unchanged, which is why this + release stays a patch. The development-only `[patch.crates-io]` + block is gone: wheels build from exactly what crates.io serves. + +### Changed + +- **A refused reload wakes `events()` natively.** The engine's 0.7.1 + failure hook signals the same parked notifier thread an install does, + so `ReloadFailed` arrives when the refusal happens rather than at the + next poll — and the stream starts no timer at all. Delivery is + latest-wins, like `changes()`: coalesced refusals arrive as one event + carrying the current `consecutive` count, and a refusal followed by an + install arrives as both events, refusal first. The limitations page's + "a refused reload cannot wake anything" section retires with this. + +### Deprecated + +- **`events(failure_poll=...)`** is accepted, ignored, and warns once: + the interval refusals were polled at, now that they wake the stream + themselves. Remove the argument; the parameter goes away in 0.4. + ## 0.3.0 — 2026-08-18 ### Fixed diff --git a/dynamic-config-python/Cargo.toml b/dynamic-config-python/Cargo.toml index f7f2413..1507bff 100644 --- a/dynamic-config-python/Cargo.toml +++ b/dynamic-config-python/Cargo.toml @@ -16,7 +16,7 @@ name = "dynamic-config-python" version = "0.3.0" edition.workspace = true # Its own floor, not the workspace's 1.71: PyO3 needs more than the core does. -rust-version = "1.85" +rust-version = "1.88" license.workspace = true description = "Python bindings for dynamic-config: Rust resolves, your schema validates" repository.workspace = true diff --git a/dynamic-config-python/examples/27_reload_events.py b/dynamic-config-python/examples/27_reload_events.py index fc1320c..1967725 100644 --- a/dynamic-config-python/examples/27_reload_events.py +++ b/dynamic-config-python/examples/27_reload_events.py @@ -14,10 +14,9 @@ async def watch_events(config: DynamicConfig[Database]) -> None: """What a log line, a metric or an alert is built from.""" - # `failure_poll` is what makes refusals visible: an install wakes this - # stream, and a refusal cannot — the engine bumps no generation for a - # load that installed nothing. - async for event in config.events(failure_poll=0.05): + # A refusal wakes this stream natively — the engine's failure hook + # signals the same parked thread an install does, so nothing polls. + async for event in config.events(): if isinstance(event, ReloadFailed): print( f" ✗ refused at generation {event.generation}: {event.kind}" diff --git a/dynamic-config-python/examples/README.md b/dynamic-config-python/examples/README.md index 6363449..ad50321 100644 --- a/dynamic-config-python/examples/README.md +++ b/dynamic-config-python/examples/README.md @@ -36,7 +36,7 @@ python examples/01_quick_start.py | [`24_async_callbacks`](24_async_callbacks.py) | `on_reload_async`, and what happens when installs outrun a hook: `latest` coalescing, `serial` dropping nothing, and a blocking hook moved off the installing thread with `Dispatch.EXECUTOR` | | [`25_async_remote_source`](25_async_remote_source.py) | `AsyncRemoteSource`: a store whose `fetch()` is a coroutine, awaited on the caller's loop — and what the synchronous `refresh_remote()` says about it | | [`26_atomic_group_reload`](26_atomic_group_reload.py) | `reload_atomic()`: every member validates or none installs, next to the per-member `reload()` that leaves a mixed state | -| [`27_reload_events`](27_reload_events.py) | `events()`: `Reloaded` and `ReloadFailed` as typed events for a log or an alert, `failure_poll` for the refusals nothing can wake, and no value in any of them | +| [`27_reload_events`](27_reload_events.py) | `events()`: `Reloaded` and `ReloadFailed` as typed events for a log or an alert — refusals wake the stream natively — and no value in any of them | Examples 10 to 12 need their framework installed, 15 needs pydantic-settings and 22 needs msgspec; 17 needs nothing at all, which is diff --git a/dynamic-config-python/python/dynamic_config/_config.py b/dynamic-config-python/python/dynamic_config/_config.py index dc0c84b..d0f03ce 100644 --- a/dynamic-config-python/python/dynamic_config/_config.py +++ b/dynamic-config-python/python/dynamic_config/_config.py @@ -61,7 +61,7 @@ from ._events import ConfigEvent, Reloaded, ReloadFailed from ._executor import default_executor from ._lifetime import _LIVE_CONFIGS, HookGuard, Watch, _register -from ._notify import Notifier +from ._notify import EventNotifier, Notifier from ._remote import AsyncRemoteSource, RemoteSource, _AwaitedDocument from ._schema import is_values_type, schema_for from ._settings import ( @@ -75,11 +75,6 @@ M = TypeVar("M") -#: What :meth:`DynamicConfig._tick` answers when the interval elapsed and -#: no install arrived. A sentinel rather than `None`, which already means -#: *this configuration was released*. -_TICK = object() - def _refusal(status: ConfigStatus, consecutive: int) -> ReloadFailed: """The `ReloadFailed` for whatever the last refusal was.""" @@ -117,6 +112,7 @@ class DynamicConfig(Generic[M]): "_async_remote", "_cached", "_core", + "_event_notifier_instance", "_executor", "_model", "_notifier_instance", @@ -237,6 +233,7 @@ def publish(_previous: M | None, current: M) -> None: # The event-loop bridge, built on the first await and not before: # a purely synchronous program never starts a notifier thread. self._notifier_instance: Notifier | None = None + self._event_notifier_instance: EventNotifier | None = None # The model the last `events()` turn reported, for `changed`. # Set by `remote()` when the store is an `AsyncRemoteSource`. self._async_remote: _AwaitedDocument | None = None @@ -257,6 +254,19 @@ def _notifier(self, loop: object) -> Notifier: return self._notifier_instance + def _event_notifier(self, loop: object) -> EventNotifier: + """The `events()` twin of :meth:`_notifier`, woken by refusals too. + + Separate on purpose: a configuration whose only consumers are + `changes()` keeps a thread that refusals never wake. + """ + del loop + + if self._event_notifier_instance is None: + self._event_notifier_instance = EventNotifier(self._core, self.key) + + return self._event_notifier_instance + @classmethod def from_settings( cls, @@ -1233,9 +1243,7 @@ async def changes(self) -> AsyncIterator[M]: yield model - async def events( - self, failure_poll: float | None = None - ) -> AsyncIterator[ConfigEvent]: + def events(self, failure_poll: float | None = None) -> AsyncIterator[ConfigEvent]: """Every install *and* every refusal, as typed events. :meth:`changes` is the model stream a service loop wants; this is @@ -1253,28 +1261,37 @@ async def events( only — the same rule every other diagnostic here follows, and for the same reason: a value in an event is a secret in a log. + A refusal wakes this stream **natively**: the engine's failure + hook signals the same parked thread an install does, so + :class:`ReloadFailed` arrives when the refusal happens, not at the + next poll. Delivery is latest-wins, like :meth:`changes`: several + refusals with nothing awake in between arrive as one event + carrying the current ``consecutive`` count, and a refusal followed + by an install arrives as both events, refusal first, because that + is the order they occurred in. + Parameters: - failure_poll: seconds between checks for a *refused* reload, - or ``None`` — the default — for installs only. - - An install wakes this stream; a refusal cannot, because - the engine bumps no generation for a load that installed - nothing, and there is nothing to be notified of. So a - stream that wants :class:`ReloadFailed` asks for it, and - pays a status read at the interval it names. Nothing is - started when it is ``None``: no timer, no thread, no - wake-up. - - One second is a reasonable choice for a watcher on a - deployment path. The refusal that matters — a watcher - reloading a file somebody has just broken — is otherwise - invisible here until the next successful install, which - in an outage is exactly the thing that is not coming. + failure_poll: **deprecated, ignored.** The interval refusals + used to be polled at, when a refusal could not wake + anything. It can now, so the stream no longer needs — or + starts — a timer; passing a value changes nothing and + warns once. """ + if failure_poll is not None: + warnings.warn( + "failure_poll is ignored: a refused reload wakes events() " + "natively now, and nothing is polled", + DeprecationWarning, + stacklevel=2, + ) + + return self._events() + + async def _events(self) -> AsyncIterator[ConfigEvent]: loop = asyncio.get_running_loop() - notifier = self._notifier(loop) - failures = int(self.status().consecutive_failures) + notifier = self._event_notifier(loop) seen = int(self._core.generation) + seen_refusals = int(self._core.refusals) # The baseline for `changed`, and local to this stream: two # streams on one configuration each compare against what they # themselves last reported, rather than against each other. @@ -1283,55 +1300,37 @@ async def events( while True: future = notifier.wait(loop) now = int(self._core.generation) + now_refusals = int(self._core.refusals) - if now > seen: + if now > seen or now_refusals > seen_refusals: + # Moved while the consumer's body was running, or between + # the lines above. Latest wins: what it gets is what is + # current, not a replay. future.cancel() - wake: Any = (now, self.try_current()) - elif failure_poll is None: - wake = await future + model = self.try_current() if now > seen else None + wake: Any = (now, now_refusals, model) else: - # One future across every tick, deliberately: a fresh one - # per tick would leave the old registered with nobody - # awaiting it, and the notifier would resolve a growing - # list of futures for one install. - while True: - wake = await self._tick(future, failure_poll) - - if wake is not _TICK: - break - - status = self.status() - count = int(status.consecutive_failures) - - if count > failures: - failures = count - - yield _refusal(status, count) + wake = await future if wake is None: # released return - generation, model = wake + generation, refusals, model = wake - if model is None: # released before the first install - return + if refusals > seen_refusals: + seen_refusals = refusals + status = self.status() + + yield _refusal(status, int(status.consecutive_failures)) if generation <= seen: continue + if model is None: # released before the read landed + return + seen = generation status = self.status() - count = int(status.consecutive_failures) - - if count > failures: - # A refusal and then an install, with nothing awake in - # between: both happened, so both are reported, refusal - # first because that is the order they occurred in. - failures = count - - yield _refusal(status, count) - - failures = count before, previous = previous, model yield Reloaded( @@ -1342,18 +1341,6 @@ async def events( reason=str(status.last_reason) if status.last_reason else "manual", ) - @staticmethod - async def _tick(future: Any, interval: float) -> Any: - """``future``'s result, or :data:`_TICK` if it takes longer. - - ``wait`` rather than ``wait_for``: a timeout must not cancel this - future, which stays registered with the notifier and is awaited - again on the next turn. - """ - done, _pending = await asyncio.wait({future}, timeout=interval) - - return future.result() if done else _TICK - def set_default(self, path: str, value: Any) -> None: """A fallback the program computes and a file need not state. diff --git a/dynamic-config-python/python/dynamic_config/_core.pyi b/dynamic-config-python/python/dynamic_config/_core.pyi index 7e3fbe8..79d5540 100644 --- a/dynamic-config-python/python/dynamic_config/_core.pyi +++ b/dynamic-config-python/python/dynamic_config/_core.pyi @@ -29,6 +29,8 @@ class Config: def key(self) -> str: ... @property def generation(self) -> int: ... + @property + def refusals(self) -> int: ... # Sources def file(self, path: str) -> None: ... def discover(self, name: str, paths: Sequence[str]) -> None: ... @@ -61,6 +63,9 @@ class Config: def wait_for_change( self, seen: int, timeout: float | None = ... ) -> tuple[int, Any] | None: ... + def wait_for_event( + self, seen_generation: int, seen_refusals: int, timeout: float | None = ... + ) -> tuple[int, int, Any | None] | None: ... def on_reload(self, hook: Callable[[Any, Any], None]) -> int: ... def remove_hook(self, token: int) -> bool: ... # Runtime layers diff --git a/dynamic-config-python/python/dynamic_config/_notify.py b/dynamic-config-python/python/dynamic_config/_notify.py index 45cb081..9d2b6f8 100644 --- a/dynamic-config-python/python/dynamic_config/_notify.py +++ b/dynamic-config-python/python/dynamic_config/_notify.py @@ -41,7 +41,7 @@ if TYPE_CHECKING: # pragma: no cover - typing only from . import _core -__all__ = ["Notifier"] +__all__ = ["EventNotifier", "Notifier"] class Notifier: @@ -83,18 +83,25 @@ def _start_locked(self) -> None: if self._thread is not None and self._thread.is_alive(): return + # The baseline is read HERE — on the caller's thread, under the + # registration lock, before `wait` returns — never on the new + # thread. Read there, an install landing between the caller's + # own generation check and the thread's first instruction would + # be folded into the baseline and never reported: the lost + # wake-up this module exists to prevent. + baseline = int(self._core.generation) + # Named after the configuration: a thread dump that says # `dynamic-config-notify-db` answers a question `Thread-17` does not. self._thread = threading.Thread( target=self._run, + args=(baseline,), name=f"dynamic-config-notify-{self._key}", daemon=True, ) self._thread.start() - def _run(self) -> None: - seen = int(self._core.generation) - + def _run(self, seen: int) -> None: while True: # No timeout: the wait ends on an install or on `release`, and # nothing else. This is the whole difference from polling. @@ -139,3 +146,91 @@ def close(self) -> None: def _settle(future: asyncio.Future[Any], wake: Any) -> None: if not future.done(): future.set_result(wake) + + +class EventNotifier: + """The :class:`Notifier`, widened to refusals — for ``events()``. + + Same shape, one difference: the thread parks in ``wait_for_event``, + which the Rust side wakes for an install *or* a refused reload. A + waiter's future resolves with ``(generation, refusals, model)`` — + ``model`` is ``None`` when only the refusal counter moved — or with + ``None`` alone when the configuration was released. + + A separate class rather than a mode on :class:`Notifier`, so a + configuration with only ``changes()`` consumers keeps a thread that + refusals never wake. + """ + + def __init__(self, core: _core.Config, key: str) -> None: + self._core = core + self._key = key + self._lock = threading.Lock() + self._waiters: list[tuple[asyncio.AbstractEventLoop, asyncio.Future[Any]]] = [] + self._thread: threading.Thread | None = None + self._closed = False + + def wait(self, loop: asyncio.AbstractEventLoop) -> asyncio.Future[Any]: + """A future resolved on the next install or refusal.""" + future: asyncio.Future[Any] = loop.create_future() + + with self._lock: + if self._closed: + future.set_result(None) + return future + + self._waiters.append((loop, future)) + self._start_locked() + + return future + + def _start_locked(self) -> None: + if self._thread is not None and self._thread.is_alive(): + return + + # The same baseline discipline as :class:`Notifier`: read on the + # caller's thread under the registration lock, so nothing that + # happens after a waiter's own check can fold into the baseline. + baseline = int(self._core.generation) + baseline_refusals = int(self._core.refusals) + + self._thread = threading.Thread( + target=self._run, + args=(baseline, baseline_refusals), + name=f"dynamic-config-events-{self._key}", + daemon=True, + ) + self._thread.start() + + def _run(self, seen: int, seen_refusals: int) -> None: + while True: + result = self._core.wait_for_event(seen, seen_refusals, None) + + if result is None: + self._resolve_all(None) + return + + generation, refusals, model = result + seen = int(generation) + seen_refusals = int(refusals) + self._resolve_all((seen, seen_refusals, model)) + + with self._lock: + if not self._waiters: + self._thread = None + return + + def _resolve_all(self, wake: Any) -> None: + with self._lock: + waiters, self._waiters = self._waiters, [] + + for loop, future in waiters: + with contextlib.suppress(RuntimeError): # pragma: no cover + loop.call_soon_threadsafe(_settle, future, wake) + + def close(self) -> None: + """Ends every waiter. ``release()`` on the Rust side does the rest.""" + with self._lock: + self._closed = True + + self._resolve_all(None) diff --git a/dynamic-config-python/src/config/inner.rs b/dynamic-config-python/src/config/inner.rs index a56931e..e24a40b 100644 --- a/dynamic-config-python/src/config/inner.rs +++ b/dynamic-config-python/src/config/inner.rs @@ -115,6 +115,10 @@ pub(super) struct Wake { /// event loop, and that thread has to be able to end: this is what /// ends it. pub(super) closed: AtomicBool, + /// Bumped — under the `generation` lock, so a waiter whose predicate + /// reads it cannot miss the wake — by the engine's failure hook. + /// What lets `events()` deliver a refusal without a timer. + pub(super) refusals: AtomicU64, } /// One validated tree, waiting to be published. @@ -355,6 +359,25 @@ impl Inner { }); }); + // The refusal wake: no Python, no GIL — a counter and the + // condition variable. `events()` reads `status()` itself, so + // nothing crosses here but "something was refused". + let weak: Weak = Arc::downgrade(this); + dynamic.on_reload_failed(move |_status| { + let Some(inner) = weak.upgrade() else { + return; + }; + + let guard = inner + .wake + .generation + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + inner.wake.refusals.fetch_add(1, Ordering::Release); + inner.wake.changed.notify_all(); + drop(guard); + }); + *engine = Engine::Ready(Arc::new(dynamic)); } diff --git a/dynamic-config-python/src/config/mod.rs b/dynamic-config-python/src/config/mod.rs index d694633..609993b 100644 --- a/dynamic-config-python/src/config/mod.rs +++ b/dynamic-config-python/src/config/mod.rs @@ -182,6 +182,7 @@ impl Config { generation: Mutex::new(0), changed: Condvar::new(), closed: AtomicBool::new(false), + refusals: AtomicU64::new(0), }, hooks: Mutex::new(Vec::new()), next_hook: AtomicU64::new(1), @@ -760,6 +761,14 @@ impl Config { }) } + /// How many reloads have been refused since start: bumped once per + /// refusal, never reset. A property, not a call — the baseline an + /// event stream starts from. + #[getter] + fn refusals(&self) -> u64 { + self.inner.wake.refusals.load(Ordering::Acquire) + } + /// The generation of the published model: bumped once per install. A /// property, not a call. #[getter] @@ -842,6 +851,81 @@ impl Config { Ok(self.current(py).map(|model| (reached, model))) } + /// Blocks until an install *or* a refusal: + /// `wait_for_event(seen_generation, seen_refusals, timeout=None)`. + /// + /// Answers `(generation, refusals, model_or_None)` as soon as either + /// counter is past what the caller has seen — the model rides along + /// only when the generation moved. `None` means the timeout elapsed, + /// or the configuration was released; the caller tells them apart the + /// same way `wait_for_change`'s callers do, by asking again. + /// + /// The refusal counter is this binding's, bumped by the engine's + /// failure hook under the same lock this wait sleeps on — which is + /// what retired the polling that used to stand in for it. + #[pyo3(signature = (seen_generation, seen_refusals, timeout = None))] + #[allow(clippy::type_complexity)] // the tuple IS the wire format the stub declares + fn wait_for_event( + &self, + py: Python<'_>, + seen_generation: u64, + seen_refusals: u64, + timeout: Option, + ) -> PyResult>)>> { + let timeout = timeout.map(seconds).transpose()?; + + let (generation, refusals) = py.detach(|| -> (u64, u64) { + let refusals = || self.inner.wake.refusals.load(Ordering::Acquire); + let closed = || self.inner.wake.closed.load(Ordering::Acquire); + let mut generation = self + .inner + .wake + .generation + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + + match timeout { + Some(limit) => { + let (guard, _timed_out) = self + .inner + .wake + .changed + .wait_timeout_while(generation, limit, |current| { + *current <= seen_generation && refusals() <= seen_refusals && !closed() + }) + .unwrap_or_else(std::sync::PoisonError::into_inner); + + (*guard, refusals()) + } + None => { + while *generation <= seen_generation && refusals() <= seen_refusals && !closed() + { + generation = self + .inner + .wake + .changed + .wait(generation) + .unwrap_or_else(std::sync::PoisonError::into_inner); + } + + (*generation, refusals()) + } + } + }); + + if generation <= seen_generation && refusals <= seen_refusals { + return Ok(None); + } + + let model = if generation > seen_generation { + self.current(py) + } else { + None + }; + + Ok(Some((generation, refusals, model))) + } + /// Registers a callback run after every install: `on_reload(hook)`. /// /// Answers the token that unregisters it. diff --git a/dynamic-config-python/tests/test_notify.py b/dynamic-config-python/tests/test_notify.py index 11aacb8..884bb1f 100644 --- a/dynamic-config-python/tests/test_notify.py +++ b/dynamic-config-python/tests/test_notify.py @@ -214,14 +214,15 @@ async def consume() -> None: assert event.at > 0 -async def test_events_reports_a_refusal_when_it_is_asked_to_poll( +async def test_events_reports_a_refusal_natively( workspace: Path, ) -> None: + """No poll interval: the refusal itself wakes the stream.""" config = configured() events: list[object] = [] async def consume() -> None: - async for event in config.events(failure_poll=0.05): + async for event in config.events(): events.append(event) consumer = asyncio.create_task(consume()) @@ -256,6 +257,35 @@ async def consume() -> None: assert failure.consecutive == 1 +async def test_events_failure_poll_is_accepted_and_warns( + workspace: Path, +) -> None: + """The 0.3.0 parameter still parses; it just no longer does anything.""" + config = configured() + write(2) + config.reload() + + received: list[object] = [] + + with pytest.warns(DeprecationWarning, match="failure_poll is ignored"): + stream = config.events(failure_poll=1.0) + + async def consume() -> None: + async for event in stream: + received.append(event) + break + + consumer = asyncio.create_task(consume()) + await asyncio.sleep(0.1) + + write(3) + config.reload() + await asyncio.wait_for(consumer, timeout=5) + + assert len(received) == 1 + assert isinstance(received[0], Reloaded) + + async def test_no_event_carries_a_value(workspace: Path) -> None: """The rule every diagnostic here follows: paths, never values.""" config = configured() diff --git a/scripts/conformance.py b/scripts/conformance.py new file mode 100644 index 0000000..b280086 --- /dev/null +++ b/scripts/conformance.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python3 +"""The conformance suite's Python runner — ~50 lines of glue, on purpose. + +Reads the engine repository's ``conformance/cases`` (path from +``CONFORMANCE_DIR``, defaulting to the sibling checkout), builds each +case through the public API, and compares ``current()`` to +``expected.json``. A disagreement names the case and is a FINDING — +fix the engine or the binding, never this runner. +""" + +from __future__ import annotations + +import json +import os +import pathlib +import sys + +from dynamic_config import DynamicConfig, Values + + +def plain(value: object) -> object: + """A ``Values`` tree as builtin dicts/lists/scalars, for comparison.""" + if isinstance(value, Values): + return {key: plain(value[key]) for key in value.keys()} + if isinstance(value, dict): + return {key: plain(inner) for key, inner in value.items()} + if isinstance(value, (list, tuple)): + return [plain(inner) for inner in value] + return value + + +def run_case(case: pathlib.Path) -> str | None: + args = json.loads((case / "args.json").read_text()) + env = json.loads((case / "env.json").read_text()) + expected = json.loads((case / "expected.json").read_text()) + + for key, value in env.items(): + os.environ[key] = value + + try: + config = DynamicConfig(Values, key=args["key"]).file(str(case / "config.toml")) + + if prefix := args.get("env_prefix"): + config = config.env(prefix) + if variable := args.get("profile_env"): + config = config.profile_env(variable) + if directory := args.get("secrets_dir"): + config = config.secrets_dir(str(case / directory)) + for name in args.get("env_files", []): + config = config.env_file(str(case / name)) + if args.get("whole_document"): + config = config.whole_document() + if missing := args.get("extra_missing_file"): + config = config.file(str(case / missing)) + + if defaults := args.get("defaults"): + config.set_defaults(defaults) + for path, value in flatten(args.get("set") or {}): + config.set_assignments([f"{path}={value}"]) + for path, value in flatten_json(args.get("overrides") or {}): + config.set_override(path, value) + for old, new in (args.get("aliases") or {}).items(): + config.alias(old, new) + + resolved = plain(config.init_and_current()) + finally: + for key in env: + del os.environ[key] + + if resolved != expected: + return f"resolved {json.dumps(resolved, sort_keys=True)} but expected {json.dumps(expected, sort_keys=True)}" + + return None + + +def flatten(tree: dict, prefix: str = ""): + for key, value in tree.items(): + path = f"{prefix}.{key}" if prefix else key + if isinstance(value, dict): + yield from flatten(value, path) + else: + yield path, value + + +def flatten_json(tree: dict, prefix: str = ""): + for key, value in tree.items(): + path = f"{prefix}.{key}" if prefix else key + if isinstance(value, dict) and value: + yield from flatten_json(value, path) + else: + yield path, value + + +def main() -> int: + root = pathlib.Path( + os.environ.get("CONFORMANCE_DIR", "../dynamic-config/conformance/cases") + ) + + if not root.is_dir(): + print(f"no cases at {root}; set CONFORMANCE_DIR", file=sys.stderr) + return 2 + + failures = [] + + for case in sorted(p for p in root.iterdir() if p.is_dir()): + if reason := run_case(case): + failures.append(f"{case.name}: {reason}") + + if failures: + print("conformance disagreements:") + for failure in failures: + print(f" {failure}") + return 1 + + print(f"{len(list(root.iterdir()))} cases agree") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/leak.py b/scripts/leak.py new file mode 100644 index 0000000..c4ba64e --- /dev/null +++ b/scripts/leak.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""The leak budget, binding edition: N reloads through the wheel, then +the deltas. Binding lifecycles — callbacks, notifier threads, PyO3 +references — are where leaks hide even when the Rust core is clean; +the free-threaded finalization bug this project fixed is the standing +argument. Exit 0 = within budget.""" + +from __future__ import annotations + +import json +import os +import pathlib +import resource +import sys +import tempfile + +from dynamic_config import DynamicConfig, Values + +RELOADS = int(os.environ.get("LEAK_RELOADS", "100000")) + + +def fds() -> int: + return len(os.listdir("/proc/self/fd")) + + +def main() -> int: + directory = pathlib.Path(tempfile.mkdtemp(prefix="dynamic-config-leak-")) + file = directory / "config.json" + file.write_text(json.dumps({"app": {"n": 1}})) + + config = DynamicConfig(Values, key="app").file(str(file)) + config.init() + + # A hook and a churned subscriber, so the callback path is on the meter. + seen = 0 + + def hook(_previous: object, _current: object) -> None: + nonlocal seen + seen += 1 + + config.on_reload(hook) + + rss_before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss + fds_before = fds() + + for n in range(2, RELOADS + 2): + file.write_text(json.dumps({"app": {"n": n}})) + config.reload() + + rss_after = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss + fds_after = fds() + + print( + f"leak: {RELOADS} reloads | maxrss {rss_before}->{rss_after} kB | " + f"fds {fds_before}->{fds_after} | hooks fired {seen}" + ) + + failed = False + + if rss_after > rss_before + 131_072: + print(f"LEAK: rss grew {rss_after - rss_before} kB", file=sys.stderr) + failed = True + + if fds_after > fds_before + 8: + print(f"LEAK: fds grew {fds_after - fds_before}", file=sys.stderr) + failed = True + + if seen != RELOADS: + print(f"LEAK/LOSS: {seen} hook calls for {RELOADS} reloads", file=sys.stderr) + failed = True + + return 1 if failed else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From e871ff06f6e2e22669440bd525d2ba69099d7c71 Mon Sep 17 00:00:00 2001 From: Cevat Batuhan Tolon Date: Wed, 19 Aug 2026 23:15:10 +0300 Subject: [PATCH 2/3] release dynamic-config-py and dynamic-config-py-remote 0.3.1 --- dynamic-config-python-remote/CHANGELOG.md | 2 ++ dynamic-config-python-remote/Cargo.toml | 2 +- dynamic-config-python-remote/pyproject.toml | 2 +- dynamic-config-python/CHANGELOG.md | 2 ++ dynamic-config-python/Cargo.toml | 2 +- 5 files changed, 7 insertions(+), 3 deletions(-) diff --git a/dynamic-config-python-remote/CHANGELOG.md b/dynamic-config-python-remote/CHANGELOG.md index 663ac64..a6ce476 100644 --- a/dynamic-config-python-remote/CHANGELOG.md +++ b/dynamic-config-python-remote/CHANGELOG.md @@ -30,6 +30,8 @@ is the door into this one. ## [Unreleased] +## 0.3.1 — 2026-08-19 + ## 0.3.0 — 2026-08-18 ## 0.2.0 — 2026-08-18 diff --git a/dynamic-config-python-remote/Cargo.toml b/dynamic-config-python-remote/Cargo.toml index 6d42f6e..1c5a518 100644 --- a/dynamic-config-python-remote/Cargo.toml +++ b/dynamic-config-python-remote/Cargo.toml @@ -9,7 +9,7 @@ name = "dynamic-config-python-remote" # release that moves the base wheel and not this one is fine (the extra keeps # working); the reverse is not, which is why the compatibility floor is # declared in `pyproject.toml` rather than assumed. -version = "0.3.0" +version = "0.3.1" edition.workspace = true # The organisation's one 1.88 floor — originally forced here anyway by # `aws-sdk-sts`, `async-nats` and `redis`, measured rather than declared. diff --git a/dynamic-config-python-remote/pyproject.toml b/dynamic-config-python-remote/pyproject.toml index 7e311c0..1e41995 100644 --- a/dynamic-config-python-remote/pyproject.toml +++ b/dynamic-config-python-remote/pyproject.toml @@ -55,7 +55,7 @@ classifiers = [ # remote path at all. It is not `==` for the reason the Rust crates are: a # wheel that pinned its sibling exactly would make every base-wheel patch a # forced upgrade of this one. -dependencies = ["dynamic-config-py>=0.3.0"] +dependencies = ["dynamic-config-py>=0.3.1"] dynamic = ["version"] [project.urls] diff --git a/dynamic-config-python/CHANGELOG.md b/dynamic-config-python/CHANGELOG.md index 433799d..1053e05 100644 --- a/dynamic-config-python/CHANGELOG.md +++ b/dynamic-config-python/CHANGELOG.md @@ -26,6 +26,8 @@ breaking. ## [Unreleased] +## 0.3.1 — 2026-08-19 + ### Fixed - **A wake could be lost when an install raced the notifier thread's diff --git a/dynamic-config-python/Cargo.toml b/dynamic-config-python/Cargo.toml index 1507bff..245e3f8 100644 --- a/dynamic-config-python/Cargo.toml +++ b/dynamic-config-python/Cargo.toml @@ -13,7 +13,7 @@ name = "dynamic-config-python" # So this moves when *this* changes — a new Python API, a behaviour # change, or an engine bump worth shipping — and `maturin upload # --skip-existing` makes a release that changed nothing here a no-op. -version = "0.3.0" +version = "0.3.1" edition.workspace = true # Its own floor, not the workspace's 1.71: PyO3 needs more than the core does. rust-version = "1.88" From 1a722f4d6d08fa525abdcc385f492eace3809015 Mon Sep 17 00:00:00 2001 From: Cevat Batuhan Tolon Date: Wed, 19 Aug 2026 23:20:34 +0300 Subject: [PATCH 3/3] the conformance pin names a tag that exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The job pinned the engine tarball at v0.7.1 — written before the release, which then shipped as v0.8.0 because semver said so. The published wheel agrees with all twenty v0.8.0 cases, verified before this push. --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bfd46d3..6ec8bff 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -36,7 +36,7 @@ jobs: # a visible PR diff, which is the point — semantics move only when # somebody says so. - run: | - ref="v0.7.1" # the engine release these wheels build against + ref="v0.8.0" # the engine release these wheels build against curl -sSL "https://github.com/dynamic-config-rs/dynamic-config/archive/refs/tags/${ref}.tar.gz" \ | tar -xz --strip-components=1 --wildcards '*/conformance/' - uses: actions/setup-python@2e3e4b15a884dc73a63f962bff250a855150a234 # v6