Feature/source package rf

.vscode/settings.json

@@ -2,7 +2,7 @@
     "maxNumberOfProblems": 100,
     "outputLevel": "debug",
     "editor.formatOnSave": false,
-    "python.defaultInterpreterPath": "/opt/poetry-venvs/packtly-builder-tooling-4uYNFs1t-py3.13/bin/python3",
+    "python.defaultInterpreterPath": "/opt/venv/bin/python3",
     "python.testing.autoTestDiscoverOnSaveEnabled": true,
     "python.testing.pytestEnabled": true,
     "python.testing.cwd": "${workspaceFolder}",

.vscode/tasks.json

@@ -3,12 +3,15 @@
     "tasks": [
         {
             "label": "packtly_builder_tooling",
-            "type": "vscode-just",
-            "task": "just",
+            "type": "shell",
+            "command": "just",
             "args": [
                 "--justfile=${workspaceFolder}/packtly-builder/tooling/justfile",
                 "${input:justRecipe}"
             ],
+            "options": {
+                "cwd": "${workspaceFolder}/packtly-builder/tooling"
+            },
             "problemMatcher": []
         }
     ],

CHANGELOG.md

@@ -16,16 +16,24 @@ RegEx for release version from file
 r"^\#\# \[\d{1,}[.]\d{1,}[.]\d{1,}\] \- \d{4}\-\d{2}-\d{2}$"
 -->
 
-## [1.2.0] - 2026-06-26
+## [1.2.0] - 2026-07-04
 ### Added
 - Source package build
 - Introduced automatic creation of .orig tarballs from the working tree when pristine-tar is not available
 - Added DebSourceBuilder to manage and automate upstream source archive generation.
 - Add of multi architecture support (x86, arm64 and armhf)
 - Add support for full build mode and enhance package existence checks
+- `deb_source`: cache parsed changelog; invalidate after `git checkout` in `reset_source_tree`
 
 ### Changed
 - Improved SIGINT / KeyboardInterrupt handling
+- `debuild`: pass `-sa` flag to always include orig tarball in `.changes` for source/full builds
+- `apt`: improve binary and source package existence detection with per-file `[EXISTS]`/`[MISSING]` logging
+- `AptManager`: reorganise class into logical sections (repository, installation, existence checks, private helpers)
+- `log`: add error logging for failed package upload
+
+### Fixed
+- Source package upload no longer fails when `.orig.tar.*` is absent from `.changes` on non-first revisions
 
 
 ## [1.1.0] - 2026-06-11

Readme.md

@@ -1,65 +1,107 @@
 # packtly-builder
 
-A containerized build system for creating, signing, and publishing Debian packages to an [Aptly](https://www.aptly.info/) repository.
+A containerized build system for **building, signing, and publishing Debian packages** to an [Aptly](https://www.aptly.info/) repository — reproducibly inside Podman containers, with no build tooling required on the host.
+
+---
+
+## Table of contents
+
+- [Overview](#overview)
+- [How it works](#how-it-works)
+- [Prerequisites](#prerequisites)
+- [Quick start](#quick-start)
+- [Container images](#container-images)
+- [`just` targets](#just-targets)
+- [The `packtly_builder_tooling` CLI](#the-packtly_builder_tooling-cli)
+- [Signing keys & credentials](#signing-keys--credentials)
+- [Development](#development)
+- [Versioning & releases](#versioning--releases)
+- [Related projects](#related-projects)
+
+---
 
 ## Overview
 
-packtly-builder provides:
+packtly-builder bundles three things:
 
-- **Container images** — a layered set of Podman images based on Debian Trixie for building and running Debian packages
-- **`packtly_builder_tooling`** — a Python CLI that orchestrates the full build-sign-publish pipeline
-- **CI/CD pipeline** — GitLab CI configuration covering image builds, tooling tests, deployment, and versioned releases
+- **Container images** — a layered set of Podman images based on Debian Trixie that carry the full Debian packaging toolchain (`debhelper`, `devscripts`, `gnupg2`, `git-buildpackage`, …).
+- **`packtly_builder_tooling`** — a Python CLI that orchestrates the build → sign → publish pipeline for a single source tree.
+- **CI/CD** — a GitLab CI pipeline that builds the images, runs the tooling test suite, and cuts versioned, multi-arch releases.
 
-### Container stages
+## How it works
 
-| Image | Purpose |
-|---|---|
-| `packtly-builder-base` | Debian Trixie with `debhelper`, `devscripts`, `gnupg2`, etc. |
-| `packtly-builder-builder` | Base + Poetry, `just`, tooling venv — used for CI builds |
-| `packtly-builder` (runtime) | Base + installed `packtly_builder_tooling` wheel |
-| `packtly-builder-devcontainer` | Builder image configured for VS Code devcontainer use |
+```mermaid
+flowchart LR
+    A[Debian source tree] --> B[debuild]
+    B --> C[debsign<br/>GPG signing]
+    C --> D{--upload?}
+    D -- yes --> E[Aptly repo]
+    D -- no --> F[Local artifacts]
+```
+
+The CLI builds the package with `debuild`, signs the resulting `.changes`/`.dsc` with a configured GPG key, and — when `--upload` is given and an Aptly host is reachable — publishes the artifacts, skipping anything already deployed upstream.
 
 ## Prerequisites
 
+On the **host** you only need a container runtime and a task runner:
+
 - [Podman](https://podman.io/) and [podman-compose](https://github.com/containers/podman-compose)
 - [just](https://github.com/casey/just)
+- [yq](https://github.com/mikefarah/yq) (used by some `just` targets)
+
+Everything else (the Debian toolchain, Python, Poetry) lives inside the images.
 
-## Getting started
+## Quick start
 
 ```bash
-# Build all images, run tests, and build the tooling wheel
+# Build the builder image, run tests, build the wheel, build the runtime image
 just all
 
-# Or step by step:
+# …or step by step:
 just build-builder        # Build the builder image
-just test-tooling-keys    # Verify GPG key setup
+just test-tooling-keys    # Verify the GPG key setup
 just test-tooling         # Run the Python test suite
-just build-tooling        # Build the Python wheel
+just build-tooling        # Build the packtly_builder_tooling wheel
 just build-runtime        # Build the final runtime image
 ```
 
-## Available targets
+List every available target at any time:
 
+```bash
+just            # equivalent to `just --list`
 ```
-just                      # List all targets
-```
+
+## Container images
+
+| Image | Built from | Purpose |
+|---|---|---|
+| `packtly-builder-base` | Debian Trixie | Debian packaging toolchain (`debhelper`, `devscripts`, `gnupg2`, …) |
+| `packtly-builder-builder` | base | + Poetry, `just`, Node, and the tooling venv — used for CI builds & tests |
+| `packtly-builder` (runtime) | base | + the installed `packtly_builder_tooling` wheel; entrypoint for real builds |
+| `packtly-builder-devcontainer` | builder | Builder image wired up for VS Code Dev Containers |
+
+Container build targets accept an optional **architecture** argument (`amd64` by default, `arm64` also supported), e.g. `just build-builder arm64`.
+
+## `just` targets
 
 ### Build
 
 | Target | Description |
 |---|---|
-| `build-base` | Build the base container image |
-| `build-builder` | Build the builder container image |
-| `build-runtime` | Build the runtime container image |
-| `build-devcontainer` | Build the devcontainer image |
+| `build-base [arch]` | Build the base image |
+| `build-builder [arch]` | Build the builder image |
+| `build-runtime [arch]` | Build the runtime image |
+| `build-devcontainer [arch]` | Build the devcontainer image |
+| `build-builder-multiarch` | Build the builder image for amd64 + arm64 and assemble a manifest |
+| `build-runtime-multiarch` | Build the runtime image for amd64 + arm64 and assemble a manifest |
 | `build-tooling` | Build the `packtly_builder_tooling` Python wheel |
 
-### test
+### Test
 
 | Target | Description |
 |---|---|
-| `test-tooling` | Run the full pytest suite inside the builder container |
-| `test-tooling-keys` | Verify GPG key availability before tests |
+| `test-tooling [arch]` | Run the full pytest suite inside the builder container |
+| `test-tooling-keys [arch]` | Verify GPG key availability before tests |
 
 ### Clean
 
@@ -70,54 +112,76 @@ just                      # List all targets
 | `clean-runtime` | Remove the runtime image |
 | `clean-devcontainer` | Remove the devcontainer image |
 | `clean-containers` | Remove all container images |
-| `clean` | Remove all images and built wheel artifacts |
+| `clean` | Remove all images **and** built wheel artifacts |
 
-### Utilities
+### Utilities & pipeline
 
 | Target | Description |
 |---|---|
 | `shell` | Open an interactive bash shell in the builder container |
-| `all` | Full pipeline: build-builder → test → build-tooling → build-runtime |
+| `all` | Full pipeline: `build-builder` → `test-tooling` → `build-tooling` → `build-runtime-multiarch` |
 
-## `packtly_builder_tooling` CLI
+## The `packtly_builder_tooling` CLI
 
-The CLI is the runtime entrypoint of the final container image. It builds, signs, and optionally uploads a Debian package.
+The CLI is the entrypoint of the runtime image. It builds, signs, and optionally uploads a single Debian package.
 
 ```
 packtly_builder_tooling <builddir> [options]
-
-Options:
-  --aptlyhost URL     Aptly REST API base URL (or set APTLYHOST env var)
-  --dist NAME         Aptly publish distribution (e.g. trixie-apollo)
-  --component NAME    Aptly component (e.g. main)
-  --upload            Upload the built package to Aptly after signing
-  --verbose           Enable debug logging
 ```
 
-GPG signing keys are expected at:
+| Option | Description |
+|---|---|
+| `builddir` | Path to the Debian build directory (positional, required) |
+| `--build-mode {binary,source,full}` | What to build: `binary` (default), `source`, or `full` (source + binary) |
+| `--no-build` | Skip the build step (sign/upload existing artifacts) |
+| `--aptlyhost URL` | Aptly REST API base URL (falls back to the `APTLYHOST` env var) |
+| `--dist NAME` | Aptly publish distribution (e.g. `trixie-apollo`) |
+| `--component NAME` | Aptly component (e.g. `main`) |
+| `--credentials-file PATH` | Aptly credentials file (default: `/run/secrets/aptly-credentials`) |
+| `--upload` | Upload the built package to Aptly after signing |
+| `--force-upload` | Upload even if the package already exists upstream |
+| `--log-file PATH` | Also write log output to this file |
+| `--verbose`, `-v` | Enable debug logging |
+
+### Build modes
+
+| Mode | `debuild` flag | Produces |
+|---|---|---|
+| `binary` | `-b` | Binary packages only (no `.orig` tarball required) |
+| `source` | `-S` | Source package only |
+| `full` | `-F` | Source **and** binary packages |
+
+## Signing keys & credentials
+
+GPG signing keys are read from these fixed paths inside the container:
+
 ```
-/opt/keys/gpg/repo_signing.key
-/opt/keys/gpg/repo_signing_private.key
-/opt/keys/gpg/repo_signing_private_pass
+/opt/keys/gpg/repo_signing.key            # public key
+/opt/keys/gpg/repo_signing_private.key    # private key
+/opt/keys/gpg/repo_signing_private_pass   # passphrase
 ```
-## Related
 
-- **[packtly-infra](https://github.com/packtly/packtly-infra)** — Infrastructure automation for deploying packtly: a self-hosted Debian package repository based on [aptly](https://www.aptly.info/) and [nginx](https://nginx.org/), running as a rootless Podman container managed by systemd Quadlets.
-
-## Versioning
-
-The release version is driven by `CHANGELOG.md` (Keep a Changelog format). The CI reads the latest `## [x.y.z]` entry and applies it as the container image label and Git tag at release time.
+Locally, the `just` targets mount `Keys/gpg/` into `/opt/keys/gpg`, so place your keys there before running builds. Aptly credentials are read from a simple `key = value` file (default `/run/secrets/aptly-credentials`) containing `username` and `password`.
 
 ## Development
 
-The repository includes a VS Code devcontainer configuration. Open the project in VS Code and select **Reopen in Container** to get a fully configured Debian build environment.
+The repository ships a VS Code Dev Container configuration. Open the project in VS Code and choose **Reopen in Container** for a fully configured Debian build environment.
 
-For local development without the devcontainer:
+For local development on the tooling without a devcontainer:
 
 ```bash
 cd packtly-builder/tooling
-just prepare   # Install Poetry dependencies
-just pytest    # Run tests
+just prepare   # Install Poetry dependencies (incl. dev tools)
+just test      # flake8 + mypy + pytest
+just pytest    # Run tests only
 just mypy      # Type-check
 just flake8    # Lint
 ```
+
+## Versioning & releases
+
+The release version is driven by `CHANGELOG.md` (Keep a Changelog format). At release time, CI reads the latest `## [x.y.z]` entry and applies it as the container image label and Git tag.
+
+## Related projects
+
+- **[packtly-infra](https://github.com/packtly/packtly-infra)** — Infrastructure automation for deploying packtly: a self-hosted Debian package repository based on [Aptly](https://www.aptly.info/) and [nginx](https://nginx.org/), running as a rootless Podman container managed by systemd Quadlets.

packtly-builder/containers/Containerfile

@@ -44,6 +44,7 @@ RUN apt-get update && \
   python3 \
   python3-pip \
   python3-apt \
+  python3-venv \
   dumb-init \
   locales \
   ca-certificates \
@@ -55,6 +56,10 @@ RUN apt-get update && \
   locale-gen && \
   rm -rf /var/lib/apt/lists/*
 
+# Optional CA certificates to be added to the system store
+COPY ${CONTEXT_ROOT}/conf/ca-certificates/ /usr/local/share/ca-certificates/
+RUN update-ca-certificates
+
 RUN mkdir -p /etc/gnupg && chmod 700 /etc/gnupg
 
 # GPG config
@@ -69,11 +74,13 @@ COPY ${CONTEXT_ROOT}/conf/rtprio.conf /etc/security/limits.d/
 FROM base AS build
 ARG CONTEXT_ROOT=.
 
-# Use a virtualenv in the build stage so Poetry is isolated from system packages
-# but include system site packages so Debian-only modules like python3-apt are importable.
-ENV POETRY_VIRTUALENVS_CREATE=true \
-  POETRY_VIRTUALENVS_OPTIONS_SYSTEM_SITE_PACKAGES=true \
-  POETRY_VIRTUALENVS_PATH=/opt/poetry-venvs
+# Fixed-path venv (survives the ephemeral build mount into the devcontainer) with
+# --system-site-packages for python3-apt; Poetry installs into it (create=false).
+ENV VIRTUAL_ENV=/opt/venv \
+  PATH=/opt/venv/bin:$PATH \
+  POETRY_VIRTUALENVS_CREATE=false
+
+RUN python3 -m venv --system-site-packages /opt/venv
 
 RUN mkdir -p /etc/apt/keyrings && \
   wget -qO- https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key \

packtly-builder/containers/conf/ca-certificates/.gitignore

@@ -0,0 +1 @@
+*.crt

packtly-builder/containers/conf/ca-certificates/.gitkeep

@@ -0,0 +1,3 @@
+# Drop corporate/proxy root CA files (*.crt, PEM format) into this folder.
+# They are copied into the image and installed via update-ca-certificates.
+# If empty, the build still works and no extra CAs are added.

packtly-builder/tooling/justfile

@@ -8,7 +8,7 @@ prepare:
     set -eu
 
     if ! command -v poetry >/dev/null 2>&1; then
-        echo "Please install poetry first: scripts/setup_venv" >&2
+        echo "Please install poetry first" >&2
         exit 1
     fi
 
@@ -34,10 +34,11 @@ prepare:
     else
         echo "No pip credentials found — skipping private registry configuration"
     fi
-
-    # Local project settings
-    poetry config virtualenvs.options.system-site-packages true --local
-
+    if [ -z "${VIRTUAL_ENV:-}" ]; then
+        echo "No virtualenv active (VIRTUAL_ENV is unset). Create/activate one before running 'just prepare' (e.g. python3 -m venv .venv && . .venv/bin/activate)." >&2
+        exit 1
+    fi
+    rm -f poetry.lock || true
     poetry install --no-interaction --with dev
 
 [private]

packtly-builder/tooling/packtly_builder_tooling/cli.py

@@ -68,11 +68,19 @@ def is_already_deployed(
     deb_files = [f for f in files if f.endswith(".deb")]
     if not deb_files:
         return False
-    if not all(
-        aptmanager.upstream_file_exists(Path(f), source_host=aptlyhost)
-        for f in deb_files
-    ):
+
+    upstream_exists = True
+    for f in deb_files:
+        upstream_exists = True
+        if aptmanager.upstream_file_exists(Path(f), source_host=aptlyhost):
+            logger.info("[EXISTS]   %s is already deployed upstream", Path(f).name)
+        else:
+            logger.warning("[MISSING]  %s is not deployed upstream", Path(f).name)
+            upstream_exists = False
+
+    if not upstream_exists:
         return False
+
     if build_mode == BuildMode.BINARY:
         return True
     dsc_files = [f for f in files if f.endswith(".dsc")]