Workspaces

.github/codeql/extensions/dw-models/codeql-pack.yml

@@ -0,0 +1,16 @@
+# A CodeQL model pack describing this project's own security barriers.
+#
+# CodeQL's dataflow library cannot see that dw/security.py validates a path
+# and *raises* rather than returning a rewritten one, so py/path-injection
+# reports every os.* call downstream of a validated path. These rows tell it
+# the barrier is there.
+#
+# Placed under .github/codeql/extensions/, this is picked up automatically by
+# code scanning default setup - no workflow changes, nothing to publish.
+name: dkackman/dw-models
+version: 0.0.1
+library: true
+extensionTargets:
+  codeql/python-all: "*"
+dataExtensions:
+  - models/**/*.yml

.github/codeql/extensions/dw-models/models/dw-security.model.yml

@@ -0,0 +1,23 @@
+extensions:
+  # validate_path() resolves a path with realpath and raises PathTraversalError
+  # unless it lands inside base_dir. Its return value is contained by
+  # construction, so taint stops there.
+  - addsTo:
+      pack: codeql/python-all
+      extensible: barrierModel
+    data:
+      - ["dw.security", "Member[validate_path].ReturnValue", "path-injection"]
+      - ["dw.security", "Member[validate_workflow_path].ReturnValue", "path-injection"]
+      - ["dw.security", "Member[validate_output_path].ReturnValue", "path-injection"]
+      - ["dw.security", "Member[validate_prompt_path].ReturnValue", "path-injection"]
+
+      # The name validators are regex whitelists that raise InvalidInputError:
+      # a workspace name is one path segment (^[\w][\w.-]*\Z - no separator,
+      # and '..' cannot match a leading \w), an asset or output reference is
+      # bounded, separator-limited segments of the same shape. Each returns the
+      # name it validated, so the return value is the safe one to use.
+      - ["dw.security", "Member[validate_workspace_name].ReturnValue", "path-injection"]
+      - ["dw.security", "Member[validate_asset_reference].ReturnValue", "path-injection"]
+      - ["dw.security", "Member[validate_output_reference].ReturnValue", "path-injection"]
+      - ["dw.security", "Member[validate_prompt_reference].ReturnValue", "path-injection"]
+      - ["dw.security", "Member[validate_variable_name].ReturnValue", "path-injection"]

.github/workflows/codeql.yml

@@ -13,12 +13,12 @@ name: "CodeQL"
 
 on:
   push:
-    branches: [ "main" ]
+    branches: ["master"]
   pull_request:
     # The branches below must be a subset of the branches above
-    branches: [ "main" ]
+    branches: ["master"]
   schedule:
-    - cron: '26 12 * * 6'
+    - cron: "26 12 * * 6"
 
 jobs:
   analyze:
@@ -32,43 +32,42 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        language: [ 'python' ]
+        language: ["python"]
         # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
         # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
 
     steps:
-    - name: Checkout repository
-      uses: actions/checkout@v7
+      - name: Checkout repository
+        uses: actions/checkout@v7
 
-    # Initializes the CodeQL tools for scanning.
-    - name: Initialize CodeQL
-      uses: github/codeql-action/init@v4
-      with:
-        languages: ${{ matrix.language }}
-        # If you wish to specify custom queries, you can do so here or in a config file.
-        # By default, queries listed here will override any specified in a config file.
-        # Prefix the list here with "+" to use these queries and those in the config file.
+      # Initializes the CodeQL tools for scanning.
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v4
+        with:
+          languages: ${{ matrix.language }}
+          # If you wish to specify custom queries, you can do so here or in a config file.
+          # By default, queries listed here will override any specified in a config file.
+          # Prefix the list here with "+" to use these queries and those in the config file.
 
-        # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
-        # queries: security-extended,security-and-quality
+          # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
+          # queries: security-extended,security-and-quality
 
+      # Autobuild attempts to build any compiled languages  (C/C++, C#, Go, or Java).
+      # If this step fails, then you should remove it and run the build manually (see below)
+      - name: Autobuild
+        uses: github/codeql-action/autobuild@v4
 
-    # Autobuild attempts to build any compiled languages  (C/C++, C#, Go, or Java).
-    # If this step fails, then you should remove it and run the build manually (see below)
-    - name: Autobuild
-      uses: github/codeql-action/autobuild@v4
+      # ℹ️ Command-line programs to run using the OS shell.
+      # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
 
-    # ℹ️ Command-line programs to run using the OS shell.
-    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
+      #   If the Autobuild fails above, remove it and uncomment the following three lines.
+      #   modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
 
-    #   If the Autobuild fails above, remove it and uncomment the following three lines.
-    #   modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
+      # - run: |
+      #   echo "Run, Build Application using script"
+      #   ./location_of_script_within_repo/buildscript.sh
 
-    # - run: |
-    #   echo "Run, Build Application using script"
-    #   ./location_of_script_within_repo/buildscript.sh
-
-    - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v4
-      with:
-        category: "/language:${{matrix.language}}"
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v4
+        with:
+          category: "/language:${{matrix.language}}"

.gitignore

@@ -153,5 +153,12 @@ venv
 # generated media - regenerable, archived outside the repo
 local_inputs/
 workflows/*/assets/
+# the workspace's asset library: input media, and where browser uploads land
+# when the checkout is the workspace (dw/workspace.py). Anchored at the root
+# so the SPA's own assets/ folders are not swept up. Generated output is
+# covered by the outputs/ rule above, run directories and manifests included
+/assets/
+# a run directory a test or a bare `dw.run -o output` leaves at the root
+/output/
 /*.mp4
 /*.wav

CLAUDE.md

@@ -51,6 +51,51 @@ The REPL (`dw/repl.py`) uses a **persistent worker subprocess** (`dw/worker.py`)
 
 **Critical**: Uses `multiprocessing.set_start_method("spawn")` for CUDA/MPS compatibility.
 
+### Workspaces on the server
+
+`dw.serve` can hold several workspaces under one root: the root's own
+`workflows/assets/outputs` are the `default` workspace, a named one is a
+subdirectory beside them (`named_workspace`, `create_workspace` in
+`dw/workspace.py`), and `prompts/` at the root is shared by all of them - there
+is one prompt library, because `prompt:` is shared by reference. Routes take an
+optional `workspace`; omitting it means the default, so pre-workspace calls are
+unchanged. A job carries its own `output_dir`, `asset_dir` and `workflow_dir`
+(`JobManager.submit`), so it stays in its workspace whatever the manager serves
+next; the worker activates the asset root per job (`activate_asset_dir`), which
+is the one root that could not stay process-wide. `jobs.sqlite` has a
+`workspace` column, backfilled to `default`. Reserved names: `workflows`,
+`prompts`, `assets`, `outputs`.
+
+### Workflow sources
+
+`dw/workflow_sources.py` is the server's workflow search path: the writable
+directory first (the workspace's `workflows/`), then any `--examples-dir`, each
+read-only. Reads (`listing`, `find_workflow`) span every root front-to-back so an
+earlier name shadows a later one; `PUT /api/workflows` always resolves through
+`writable_source`, so saving something opened from a read-only root writes a copy
+rather than overwriting it, and `DELETE` on a read-only root answers 403. A job
+carries the root it is confined to (`JobManager.submit(workflow_dir=...)`), so an
+examples workflow runs confined to the examples directory rather than to the
+writable one. Packaged `dw/workflows/` is off the path - it is what `builtin:`
+sub-workflow steps name, resolved in `dw/workflow.py`.
+
+### Workspaces
+
+`dw/workspace.py` resolves the one directory a run's content belongs to -
+`workflows/`, `prompts/`, `assets/`, `outputs/`. Order: `--workspace` >
+`DW_WORKSPACE` > the `workspace` setting > the working directory when it holds
+any of `workflows/`, `prompts/` or `outputs/` > `~/diffusers-workspace`. A
+checkout satisfies rule four, so every default lands where it did before
+workspaces existed. Resolution creates nothing; an entry point about to write
+calls `ensure()` (or creates the one folder it needs). `set_workspace` pins the
+root *and* how it was chosen into the environment, so a spawned worker does not
+read an inferred workspace back as one the user named - `get_prompt_dir` yields
+to its older discovery (`./prompts`, then the walk up from the workflow file)
+for an inferred workspace but not for an explicit one. `--workflow-dir`,
+`--output-dir` and `--prompt-dir` each still override one folder. See
+docs/WORKSPACES.md, and docs/proposals/workspaces.md for the later stages
+(workflow search path, run directories, `asset:`/`output:` references).
+
 ### Type System
 
 `arguments.py` + `type_helpers.py` handle dynamic type conversion during workflow loading:
@@ -60,6 +105,23 @@ The REPL (`dw/repl.py`) uses a **persistent worker subprocess** (`dw/worker.py`)
 - Values prefixed with `constant:` read a value declared in python rather than copying it
   into JSON: `"constant:diffusers.pipelines.ltx2.utils.DISTILLED_SIGMA_VALUES"`. Resolved
   in `realize_args`, validated by `validate_constant_name()`; anything callable is refused
+- Values prefixed with `asset:` resolve to the path of a file in the asset library:
+  `"asset:iris.png"` or `"asset:gyre/frames/web.mp4"`. Resolved in `realize_args` before
+  every other convention (`dw/assets.py`), rooted at the library rather than the workflow
+  file, confined to it, and then loaded by whatever would have loaded a path written
+  there. The library is `DW_ASSET_DIR` / `--asset-dir`, else the workspace's `assets/`
+  when a workspace was named, else `./assets` if it exists, else found by walking up
+  from the workflow file's directory
+- Values prefixed with `output:` resolve to the path of a file an earlier run wrote:
+  `"output:ltx2/Gyre/latest/still.png"`. The name is `<workflow identity>/<run id>/<file>`
+  under the output root, and `latest` in the run-id position picks the newest run that
+  holds the file (run ids sort by their UTC timestamp; a failed or fully-cached run holds
+  only a manifest and is skipped). Resolved in `realize_args` beside `asset:` (`dw/runs.py`),
+  against the output root `Workflow.run` activates, and confined to it
+- A generated file becomes a stable input with `POST /api/assets/keep` (gallery "Keep as
+  asset", MCP `keep_output`): it is hard-linked, else copied, from the workspace's outputs
+  into its assets under a chosen name, so later workflows reference `asset:name` rather
+  than a run id that pruning would break
 - Values prefixed with `prompt:` load a stored prompt's `text` from the prompt library:
   `"prompt:name"` or `"prompt:folder/name"`. Resolved in `realize_args` (`dw/prompts.py`),
   rooted at the library rather than the workflow file. The library is `DW_PROMPT_DIR` /
@@ -107,9 +169,19 @@ All entry points use `dw/security.py`. When adding features:
 - **Built-in workflows** need explicit argument mapping: `"prompt": "variable:prompt"`
 - **MPS differences from CUDA**: no autocast, no bitsandbytes, no flash_attn, no triton, no torch.compile. Model offloading has less benefit on unified memory, and `"offload": "sequential"` is downgraded to `"model"` with a warning there (`place_component`) — per-submodule streaming hands back no residency when the CPU and the accelerator share one pool. `exclude_from_cpu_offload` is sequential-only and does not survive the downgrade.
 - **`{}`-escaped strings** in JSON arguments: `"{nf4}"` stays as string `"nf4"`, without braces it would try to load as a type
-- **A stored prompt's `text` may not begin with a reference prefix** (`variable:`, `previous_result:`, `constant:`, `prompt:`) — the engine rejects it to prevent double resolution or iteration expansion
+- **A stored prompt's `text` may not begin with a reference prefix** (`variable:`, `previous_result:`, `constant:`, `asset:`, `output:`, `prompt:`) — the engine rejects it to prevent double resolution or iteration expansion
 - **Audio+video muxing**: pipelines that generate audio alongside video (LTX-2) have the two muxed into one `video/mp4` file with PyAV in `result.py`
-- **Step cache**: a process-wide singleton (`dw/step_cache.py`) consulted by every `Workflow.run`, including server jobs; entries are keyed by `(workflow id, step name)`; disabled entirely when the workflow sets no `seed`; a hit reports the earlier run's files with `reused: true` and writes nothing new; `memory clear` drops it
+- **Run directories**: each execution writes `<output_dir>/<workflow identity>/<run id>/`
+  with a `manifest.json` beside its files (`dw/runs.py`, `Workflow.effective_output_dir`).
+  Identity is the workflow's path under a `workflows/` tree, else its file name, else its
+  `id`; the run id is `<UTC timestamp>-<8 hex of the spec>`, with a `-N` counter if taken.
+  A sub-workflow inherits the parent's run directory and writes no manifest of its own.
+  `--output-layout flat` / `DW_OUTPUT_LAYOUT` / the `output_layout` setting restores the
+  old layout. The gallery groups a workflow's runs under one folder by stripping the run
+  id (`strip_run_id`)
+- **Step cache**: a process-wide singleton (`dw/step_cache.py`) consulted by every `Workflow.run`, including server jobs; entries are keyed by `(workflow id, step name)` and validated against the output
+  *root*, never the per-run directory - a run directory is new every execution and would
+  defeat the cache; disabled entirely when the workflow sets no `seed`; a hit reports the earlier run's files with `reused: true` and writes nothing new; `memory clear` drops it
 
 ## JSON Workflow Structure