feat: add updateAddon.py automated tool to complement existing

[abdel792]

Link to issue number:

Close #37.

Summary of the issue:

Upgrading an existing add-on to follow the modern addonTemplate structure manually is a repetitive and error-prone process. Developers frequently encounter syntax issues or merge conflicts when manually porting old metadata into the new pyproject.toml structure.

Description of developer facing changes:

Introduces an automated update workflow. Developers can now run a single command (uv run updateAddon.py) from any directory on their PC to automatically migrate metadata, clean template placeholder data (like removing nvaccess from third-party authors lists), and merge project configurations while preserving custom formatting.

Description of development approach

• Script implementation: Added updateAddon.py at the repository root. The script leverages an AST-aware (Abstract Syntax Tree) approach to perform safe, robust parsing and merging of TOML structures.
• Dependency Update: Added tomlkit==0.13.0 under the # Update machinery section in pyproject.toml to support programmatic, comment-preserving TOML modifications.
• Linter Exclusion: Added updateAddon.py to the exclude lists of both [tool.ruff] and [tool.pyright] within pyproject.toml to avoid unnecessary environmental or strict typing alerts during standard repository validation.
• Documentation: Updated updatingExistingAddons.md to cleanly integrate prerequisites (Python 3.11+, Git available in PATH) and step-by-step instructions on how to use this new companion tool alongside the existing manual instructions.

Testing strategy:

The script and documentation workflow have been tested locally:

1. Initialized a legacy NVDA add-on workspace using old metadata configurations.
2. Ran git fetch template to ensure local tracking branches were up to date.
3. Executed uv run updateAddon.py from the repository root, and verified that a _bak_ backup directory was safely generated.
4. Checked the newly mutated pyproject.toml to ensure old custom metadata was correctly merged, template placeholder defaults were stripped, and file formatting/comments remained completely intact.
5. Ran uv sync followed by uv run scons to ensure the final environment synchronized properly and the .nvda-addon bundle compiled successfully with zero errors.
6. Repeated the execution from a completely different working directory outside the repository to confirm path flexibility.

Known issues with pull request:

None.

---

Code Review Checklist

• Testing: Manually tested across diverse repository paths to confirm robustness. Steps to reproduce are detailed in the testing strategy.
• API is compatible with existing add-ons: This is an infrastructure update tool for developers; it does not introduce any breaking API changes or modifications to the NVDA core ecosystem.
• Documentation: Technical developer documentation (updatingExistingAddons.md) has been explicitly updated to reflect these changes.
• UX of all users considered: N/A (Developer-facing infrastructure tool only).
• Security precautions taken: N/A (The script executes locally during development and does not interact with secure screens or lock screen states).

[abdel792]

Hi @nvdaes,

I have just added you as a collaborator to this repository.

If you have some time and interest, feel free to directly review and update the docstrings in updateAddon.py to ensure they perfectly align with the project's official documentation conventions.

Thank you so much for your help and guidance on this!

[nvdaes]

Thanks @abdel792 . I've accepted your invitation as a collaborator, and I'll try to update the docstrings.

[nvdaes]

@abdel792 , can we add a way to ignore files which shouldn't be merged?
I think that @CyrilleB79 may be interested, and i'd like to have this possibility. For example, I don't want to use the build_addon.yml workflow since it's failing for me, and I prefer to remove it.
Also, I think that instructions for the script should be placed at the start of the file, and manual instructions maybe removed or placed at the end. BTW, they contain an error of a typo. With your permission, I'll fix it.

[abdel792]

Thanks a lot @nvdaes for your quick commits and for improving the docstrings and file headers!

Regarding your suggestions:

1. Ignoring files: That's a great idea. Having a way to skip specific files (like workflows or custom configs) during the merge would make the tool much more flexible. I'll think about the best way to implement this (maybe via an ignore list/parameter) and update the script.
2. Documentation structure: Completely agree. The automated tool should definitely be the primary option at the top of the file, with the manual steps as a fallback at the end. Please feel free to reorganize the file and fix the typo you found, you have my full permission!

I'll let you know once I've drafted the file-ignoring feature so we can test it. Thanks again for your amazing collaboration!

[nvdaes]

@abdel792 , I've reorganized the documentation so that info about the script is at the start. Perhaps the information about merge should be removed=
Looking forward to see a way to ignore files/folders.
Thanks for all your amazing work.

[abdel792]

Hi @nvdaes,

Thank you so much for updating the documentation! It looks great and makes using the automated tool very clear.

Could you please do me a favor and also integrate the new feature requested by @CyrilleB79 that I just implemented in my latest commit? It allows developers to exclude specific template files during synchronization.

Here is the markdown section that needs to be added to the documentation (you can place it right before the build verification step):

### Excluding specific template files

By default, the script synchronizes every infrastructure file provided by AddonTemplate.

If you want to preserve specific files from your repository (for example, a customized GitHub workflow), you can exclude them from synchronization.

Open `updateAddon.py` and locate the `IGNORED_FILES` set near the beginning of the `main()` function:

```python
IGNORED_FILES = {
    os.path.join(".github", "workflows", "build_addon.yml").lower(),
}

Add any relative path from the template root to this set to prevent that file from being synchronized.

---

[abdel792]

Hi @nvdaes,

While looking closely at our documentation, I am considering a complete overhaul and restructuring of the documentation files to make everything cleaner, more cohesive, and easier to navigate for developers.

Before diving into this, I wanted to open a discussion here to get your thoughts, ideas, or any suggestions you might have regarding how we should reorganize the content.

Let me know what you think!

[abdel792]

I have just committed a complete overhaul and reorganization of the update documentation (updatingExistingAddons.md).

In this update, I have:

• Restructured the document to separate the recommended automated method from the manual workflow.
• Added the full comprehensive guide for the automated update script (updateAddon.py).
• Added a dedicated section explaining how to exclude specific template files using the IGNORED_FILES set, as requested by @CyrilleB79.

Please let me know if you have any feedback or further suggestions.

@nvdaes, please feel free to add, modify, or correct anything you see fit—you have the green light, of course!

[Copilot]

Pull request overview

This PR introduces a developer-facing automation tool (updateAddon.py) intended to help migrate/update existing NVDA add-ons to the modern AddonTemplate structure, alongside documentation and dependency updates to support that workflow.

Changes:

• Added updateAddon.py to clone the latest AddonTemplate, sync infrastructure files, and merge buildVars.py / pyproject.toml.
• Updated pyproject.toml dependencies (pyright bump + new tomlkit dependency) and excluded updateAddon.py from ruff/pyright checks.
• Expanded docs/managementFromGit/updatingExistingAddons.md with an “automated update” path plus revised manual instructions.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 11 comments.

File	Description
updateAddon.py	New CLI tool that syncs template infrastructure and performs AST/TOML-based merges.
pyproject.toml	Adds tomlkit dependency and excludes the new script from lint/type-checking; bumps pyright.
docs/managementFromGit/updatingExistingAddons.md	Documents the new automated update workflow and revises the manual update steps.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wmhn1872265132]

I suggest renaming updateAddon.py to updateTemplate.py to make the purpose of the file clearer.

[abdel792]

Hi @wmhn1872265132,

Thanks for the suggestion! However, I'm concerned that renaming it to updateTemplate.py might be a bit misleading.

In this context, the "template" refers specifically to the upstream nvaccess/AddonTemplate repository. We aren't actually updating the template itself; rather, we are updating the local addon repository's infrastructure to align with the latest template standards.

Keeping updateAddon.py helps clarify that the target of the update is the addon's repository, not the template.

[CyrilleB79]

Hi all! You are discussing and working a lot and I'm not able to follow all this as would be needed, sorry.

It's not clear to me if this tool is needed:

1. once for add-on X based on an old version of the template to catch up with more recent template evolution
2. or each time that add-on X need to be updated from the template

@abdel792 you write:

Upgrading an existing add-on to follow the modern addonTemplate structure manually is a repetitive and error-prone process. Developers frequently encounter syntax issues or merge conflicts when manually porting old metadata into the new pyproject.toml structure.

Do you mean in case 1 or case 2? If you meant case 2, it's not true at all. I too have had merge difficulties to update my add-ons from template, e.g. when the structure of the buildVar.py changed. But now that my add-ons are synchronized, I update quite easily from the template. I just merge from (without --squash flag) my local templateMaster branch (which follows template/master) and usually have only conflict on the readme.md, which is easy to solve since I juste need to choose my add-on's version and discard all changes from the template on this file.

To me, upgrades from template now work correctly and I do not ask personally a new tool to perform this work, and won't probably need to use it. Sorry for the confusion.

Of course, I have a lot of commits coming from the template in my add-on's history. This may be considered undesirable by some. To me, it's not a problem. I can easily ignore them doing git log --first-parent.

Also small suggestion regarding the name: what about updateAddonFromTemplate.py?

[nvdaes]

@CyrilleB79 wrote:

It's not clear to me if this tool is needed

Imo, it's a good addition:

1. It avoids the necessity of add the template as a remote, which may be tricky for some people.
2. Imagine that the template evolves and buildVars.py is replaced with something completely different, or that we want to replace .precommit.yaml with prek.toml, which seems to be easier to manage since it doesn't require to add each repo to precommit.ci. I think that this feature may manage this in a consistent way avoiding, from the start, resolving conflicts.
   I'm using it to update the template for all my add-ons now.

[CyrilleB79]

Sorry @nvdaes, I do not question the usefulness of tis tool. My question is if this tool is meant to address case 1 (first update of add-on from template), or case 2 (regular updates from template).

[nvdaes]

@CyrilleB79 wrote:

My question is if this tool is meant to address case 1 (first update of add-on from template), or case 2 (regular updates from template).

My understanding is that this covers both cases, but I may be wrong. Let's wait for Abdel's reply.

[abdel792]

Hi @CyrilleB79, thank you for sharing your thoughts! And thanks @nvdaes for jumping in with those excellent points.

To answer your question directly: as @nvdaes suspected, this tool is designed to address both cases.

1. For Case 1 (Initial Catch-up from very old versions): In older versions of the template, addon_info was defined as a standard dictionary, and pyproject.toml didn't even exist yet. In this scenario, the script does some heavy lifting: it migrates the legacy dictionary structure in buildVars.py into the modern AddonInfo object now in use, and automatically generates a brand new, fully populated pyproject.toml file to seamlessly align the add-on with modern template standards.

2. For Case 2 (Regular Updates / Maintenance):
   I completely understand that maintaining a templateMaster branch and doing standard git merges works smoothly for you. However, for many developers, dealing with Git remotes and tracking upstream configuration changes can be intimidating.
   Even for synchronized repositories, the script adds value by safely checking if new keys have been introduced in buildVars.py or if updates/new dependencies have been added to pyproject.toml. It merges these programmatically without triggering layout corruption or unexpected syntax conflicts. This ensures all other infrastructure components of the workspace stay perfectly aligned with the latest template evolution with a single command.

It's completely optional, of course! Your manual Git merge workflow remains fully supported and valid (and is still detailed in the documentation). This tool simply offers a streamlined alternative.

Regarding the name suggestion: updateAddonFromTemplate.py is a fantastic idea. It perfectly clarifies the script's intent without being misleading (unlike updateTemplate.py, which sounded like we were modifying the template itself). I will rename the script and update the documentation to reflect this right away!

[abdel792]

Just a quick update regarding the recent merge conflict in pyproject.toml.

It turns out it was caused by a recent upstream commit on the master branch by @LeonarddeR. After a few attempts, I managed to resolve it cleanly by first merging master to pull in Leonard's additions, and then reapplying the specific dependencies and configurations needed for this PR.

Everything is now sorted out and the branch is up to date!

[abdel792]

Hi @nvdaes,

I have reorganized the integration guide by merging your structural improvements with the proposed template updates in this PR.

To ensure we didn't lose any of your valuable contributions, I made sure to:

1. Restore your explanations regarding the automatic cleanup of placeholder metadata (specifically clarifying that "nvaccess" is only stripped from the authors list if they are not actually among the maintainers, preventing accidental copy-pasting from the template).
2. Maintain your preferred formatting methodology (using frequent line breaks after periods to keep the document highly readable and easy to diff).
3. Align the prerequisites, making sure Python versions, Git steps, and tool options (--squash, .addonmergeignore, etc.) are accurate and clean.

Please let me know if this reorganization looks good to you or if you would like any further adjustments!

[nvdaes]

@abdel792 , I've pushed a commit updating the documentation.
Summary:

• Added sh to commands.
• • Place the step to fetch the template manually in the alternative method (merge manually).
• Explain how to remove the backup folder (untracked) using git clean -f.
  Please review it and, if all is right for you, you may request a review from NV Access.

Thanks for this amazing script!

[abdel792]

Hi @nvdaes,

Thank you so much for updating the documentation!

The additions (adding sh, restructuring the manual merge steps, and adding the git clean -fd tip to remove the backup folder) look absolutely perfect and make the guide much clearer.

I have also just pushed an update to the PR to handle the repository URL:

• Repository URL handling:
  The update script now successfully extracts the addon_url from buildVars.py when generating the initial pyproject.toml file. If the URL is blank in buildVars.py (which is the case for some legacy add-ons), it will still write the Repository key with an empty string ("") to keep the structure consistent, rather than mistakenly importing the AddonTemplate's URL.

On a side note regarding the existing documentation and other suggestions:

• The "NV Access" author explanation:
  In the documentation, there is a section explaining the conditional removal of "NV Access" from the authors list. Historically, in the very early versions of our synchronization tool, the template's author was systematically added to all synchronized add-ons, which is why we had to introduce logic to prevent copying "NV Access" unless they were actual maintainers of that specific add-on. Since this is now fully resolved and handled seamlessly by the script, please feel free to rewrite, simplify, or completely remove that existing explanation from the document if you feel it is no longer necessary or adds too much noise.

• Script naming & suggestions:
  Please feel free to suggest any other modifications, additions, or documentation adjustments you'd like to see. For instance, if you think a different filename for the python module would be more appropriate—such as addonMergeTemplate.py or something similar instead of updateAddonFromTemplate.py—I would be more than happy to rename it!

If everything looks good to you with these latest changes, I'll go ahead and request a review from @seanbudd.

Thanks again for your amazing collaboration on this!

[nvdaes]

@abdel792 , the last commit about managing URL looks good to me. Thanks so much!

[abdel792]

Hi @nvdaes, @seanbudd, and everyone,

I have just pushed a new update to the PR to finalize the tool's naming and refine the documentation:

• Script Renaming & Tool Configuration:
  I have renamed the companion script from updateAddonFromTemplate.py to syncAddonWithTemplate.py. To me, "update" felt slightly misleading, as developers might think the script modifies their actual add-on source code or upgrades its functional features. Since the tool's true purpose is to synchronize the repository's underlying build infrastructure with the latest upstream template standards (without touching the add-on code itself), "sync" feels much more accurate and reassuring.
  Consequently, I have also updated the pyproject.toml configuration so that Ruff and Pyright exclusions correctly reference this new file name.

• Documentation Alignments:
  I updated the documentation guide to reflect the new script name (syncAddonWithTemplate.py) in all text and examples (including .addonmergeignore). I also standardized all invocation examples to use the safer, cross-platform syntax uv run python syncAddonWithTemplate.py.

• The "NV Access" Author Cleanup:
  As discussed, I have removed the historical explanation regarding the conditional removal of "NV Access" from the authors list. Since this cleanup is now fully resolved and handled seamlessly and invisibly by the script, keeping this section in the documentation was no longer necessary or helpful for the users.

---

@seanbudd, everything is fully updated, tested, and ready on our end. If you have no further suggestions or reviews to make, this PR is ready to be merged.

Thank you all for this amazing collaboration!

[abdel792]

Hi @nvdaes, @seanbudd, and everyone,

I have just pushed one final quality-of-life update to the script to align its CLI behavior with our documentation:

• Improved CLI Ergonomics (Positional Option):
  Instead of requiring the -ad or --addon-dir flag to target an external add-on directory, I have refactored the parameter into an optional positional argument in argparse using nargs="?".
  This officially enables the cleaner syntax already showcased in our documentation examples:
  uv run python syncAddonWithTemplate.py ../MyAddon

  Running the script without any arguments still defaults seamlessly to the current working directory, preserving the original behavior.

---

@seanbudd, this completes all refinements on our side. The implementation is now fully aligned, tested, and ready for your review whenever you are available.

Thank you again!

[abdel792]

Hi @seanbudd,

Thank you very much for your valuable feedback and excellent recommendations!

I have updated the PR to fully align with your suggestions:

• Refactored syncAddonWithTemplate.py to ensure proper separation of concerns by splitting the logic into dedicated functions.
• The unit test suite and its configuration have been integrated as requested.
• The documentation has been updated to explain how to run unit tests using pytest inside the uv environment.
• I have also added instructions in the documentation regarding the use of the uv run --with tomlkit option. This will be very helpful for users who want to execute the companion script without having to manually install tomlkit in their current environment.

Additionally, regarding the -ad / --addon-dir option: I decided not to use nargs or make the argument positional. From a pedagogical standpoint, keeping the explicit option keys (-ad and --addon-dir) is much clearer and helps users immediately understand what parameter they are passing.

Please let me know if you have any other recommendations or remarks—I would be more than happy to address them!

Thank you again for your guidance on improving this workflow!

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 13 comments.

[abdel792]

Hi @seanbudd and Copilot,

Thank you both for the detailed and constructive code review!

I have addressed all the feedback in the latest commit, including a few refinements following Copilot's suggestions:

1. Refactor & Python Script (syncAddonWithTemplate.py)

• Logging: Replaced print() statements with structured logging to support both console and file logging.
• CLI Arguments: Refactored argument parsing into its own dedicated helper function.
• Markdown Report: Converted the sync summary report output to Markdown format for better readability.
• TOML Handling & Indentation Fix:
  • Updated author/maintainer checks to use MutableSequence instead of list to seamlessly support tomlkit data structures.
  • Following initial feedback, global whitespace replacement was removed to prevent string corruption. However, this caused the tab indentation required by NVDA standards in authors/maintainers arrays to revert to spaces. This is now fixed using targeted block-scoped indentation formatting without affecting string literals.
• Robust os Import Detection:
  • Replaced the simple "os." in valExpression string lookup with recursive AST inspection (ast.Name nodes). This ensures that references inside comments or text strings do not trigger a false-positive import os insertion.
  • Adjusted AST merging to insert a single top-level import os statement in buildVars.py only when actual os usage is detected.
• Formatting: Cleaned up trailing EOF whitespace.

2. Documentation (docs/managementFromGit/updatingExistingAddons.md)

• Structure & Formatting: Removed duplicated prerequisite sections, fixed broken code block backticks (```), and postponed the unit testing section to the upcoming testing PR (Swap pre-commit for prek #42).

3. Testing Infrastructure

• As requested, I removed tests/__init__.py (and the corresponding documentation section) from this PR. I will introduce the full test suite and documentation in PR Swap pre-commit for prek #42.

Please let me know if any further adjustments are needed!

[abdel792]

Hi everyone,

As agreed with @seanbudd, the next PR will introduce a full unit test suite along with several configuration and documentation adjustments. Here is a detailed preview of what is scheduled:

1. Unit Test Suite (tests/test_syncAddonWithTemplate.py)

A comprehensive unit test suite based on Python's standard unittest framework and using isolated temporary workspace directories (tempfile) will be added. It will validate 4 core test cases to ensure tool stability across future updates:

• testFixTomlIndentation: Will verify that 4-space indentations within array blocks (authors/maintainers) are strictly converted to tabs while preserving formatting elsewhere.
• testFormatAuthorList: Will validate raw author string parsing ("Name <email>") into structured tomlkit inline table arrays.
• testMergeLegacyBuildvarsWithOfficialTemplate: Will ensure safe migration of legacy dictionary-based buildVars.py files into the modern AddonInfo AST structure.
• testMergeModernBuildvarsMissingSpeechDictionaries: Will confirm automatic injection of missing global variables (such as speechDictionaries) into modern configurations.

Note: For these tests to run without errors, the target module (syncAddonWithTemplate.py) must be located at the root of the project alongside the tests/ directory.

2. Scope Protection & Linter Exclusions

• PROTECTED_ELEMENTS update: The tests directory will be added to the protectedElements set in syncAddonWithTemplate.py to ensure local developer unit tests are never overwritten, modified, or removed during synchronization.
• pyproject.toml exclusions: The tests directory will be explicitly added to the exclude lists of both [tool.ruff] and [tool.pyright] to avoid unnecessary noise in static analysis reports.

3. Documentation Updates

The developer documentation (updatingExistingAddons.md) will be updated with a dedicated section explaining how to run the test suite locally using uv:

## Unit Testing the Add-on

Ensuring your add-on behavior remains consistent during development and following template updates is done through unit testing. Tests are stored in the `tests/` subdirectory and import the module files located at the project root. They can be validated quickly using `unittest` inside the virtual environment managed by `uv`.

To run the entire unit test suite with detailed progress, use the following command:

uv run python -m unittest -v

To run a specific test module individually during development, specify its path:

uv run python -m unittest -v tests/test_syncAddonWithTemplate.py