Feat/include template files in sends

[andrewleith]

Summary | Résumé

This pull request introduces a new mechanism to efficiently handle template file attachments for api and admin sending (both one-off and bulk).

It employs caching in redis to ensure the same set of attachments will be sent for a bulk job, even if users change the template during the send. This also reduces database load and file download duplication for bulk jobs.

The most important changes are:

Caching and Retrieval of Template File Attachments:

• Added a new function _cache_template_files_for_job in app/celery/tasks.py to pre-cache template file metadata in Redis for bulk email jobs, reducing redundant database queries and downloads during job processing. This function is called at the start of email job processing. [1] [2]
• Implemented _get_template_files_from_cache_or_db and _get_template_attachments in app/delivery/send_to_providers.py to fetch template file metadata from Redis (for bulk jobs) or the database (for one-off sends or cache misses), and to download the files from the document download service.

Integration with Email Sending Workflow:

• Modified send_email_to_provider in app/delivery/send_to_providers.py to merge template file attachments (fetched and downloaded using the new caching mechanism) with any personalized file attachments before sending emails.

Database Access Enhancements:

• Added dao_get_ready_files_by_template_id in app/dao/files_dao.py to retrieve only files that are scanned and ready for sending, and updated imports accordingly. [1] [2]
• Updated imports in both app/celery/tasks.py and app/delivery/send_to_providers.py to include the new DAO function and redis_store. [1] [2] [3] [4]

Personalization Logic Update:

• Updated the logic in send_email_to_provider to exclude template attachments from the list of personalized file attachments, ensuring that template and personalized files are handled separately and correctly.

Related Issues | Cartes liées

• Dev Implementation of UI Attachments notification-planning#3093

Test instructions | Instructions pour tester la modification

ADMIN

• One-off emails from a template with attachments have those files attached to the received email
• Bulk emails from a template with attachments have those files attached to the received email

API

• One-off emails sent via the API using a template with attachments have those files attached to the received email
• Bulk emails sent via the API using a template with attachments have those files attached to the received email

Bulk

• Files are only read once from DDAPI/DB during a bulk send

Release Instructions | Instructions pour le déploiement

None.

Reviewer checklist | Liste de vérification du réviseur

• This PR does not break existing functionality.
• This PR does not violate GCNotify's privacy policies.
• This PR does not raise new security concerns. Refer to our GC Notify Risk Register document on our Google drive.
• This PR does not significantly alter performance.
• Additional required documentation resulting of these changes is covered (such as the README, setup instructions, a related ADR or the technical documentation).

⚠ If boxes cannot be checked off before merging the PR, they should be moved to the "Release Instructions" section with appropriate steps required to verify before release. For example, changes to celery code may require tests on staging to verify that performance has not been affected.

[Copilot]

Pull request overview

This PR adds support for including template-level file attachments in email sends (API/admin; one-off and bulk), with Redis-based caching of template file metadata for bulk jobs to stabilize attachment sets and reduce repeated DB lookups.

Changes:

• Added Redis-backed retrieval for template file metadata and a download helper to turn those into provider attachments (app/delivery/send_to_providers.py).
• Added a Celery-side pre-cache step for bulk email jobs to store template file metadata in Redis (app/celery/tasks.py).
• Added a DAO method to fetch “ready” template files and new tests covering the new attachment flow (app/dao/files_dao.py, tests/app/delivery/test_template_attachments.py).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
tests/app/delivery/test_template_attachments.py	Adds unit/integration-style tests for caching, downloading, and merging template attachments into sends.
app/delivery/send_to_providers.py	Implements cache/DB retrieval + download of template files, and merges them into send_email_to_provider attachments while excluding template_attach entries from personalisation file processing.
app/dao/files_dao.py	Adds a DAO for retrieving “ready” template files used by both the Celery pre-cache and delivery layer.
app/celery/tasks.py	Adds a pre-cache step for bulk email jobs to store template file metadata in Redis.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[whabanks]

LGTM!