fix: use config service for head.html, rewrite its CSP

[benpeter]

Description

Reapplies #270 (reverted in #275) + the CSP rewrite that makes it safe.

#270 got head.html from the config service instead of the preview host, so CSP-metas with 'nonce-aem' made it to the page and were enforced -> editor could not instrument the page loaded to the canvas

da-universal now rewrites the CSP when it builds the page

before

• the editor worked only because the pipeline already moved the CSP meta from /head.html to a response header that getAEMHtml throws away, so the page had no CSP at all
• a site behind Helix authentication refuses {ref}--{site}--{org}.aem.page/head.html without a site token, so the editor got a page with an empty head
• nonce logic added in fix: add nonce to quickedit for 404 cases #233 is dead since feat: Implement full proxy behaviour #240

now

• one read of the config service answers existence, head.html and the store
• create a nonce per request, replace 'nonce-aem' in the policy and on head.html' elements, put it on the scripts we inject (UE scaffold and quick-edit import map)
• policy stays a <meta> and is not moved to a response header, so frame-ancestors, sandbox and report-uri stay unused. (frame-ancestors 'self' prevent the page loading in the editor)
• require-trusted-types-for is dropped from the policy. neither a nonce nor the editor's DOM writes can satisfy it.
• a policy with no 'nonce-aem' is left as is
• isUE also recognises the configured local host to make the editor path reachable in local development

the worker names the upstream that failed

The worker answers 503 with Retry-After and an x-error naming the upstream that failed: preview host, content store, editor config or site lookup. The worker answers a bodyless 500 for a throw in its own tree work. Tests pin both.

• the worker names the upstream for three reads that previously produced a bodyless 500: the metadata sheet read while it builds the page, the document body read from the store, and the proxy read behind .css, .js, .json, font and .plain.html requests
• the worker puts a 5s deadline on the metadata sheet read, matching the site lookup, so a preview host holding a connection open no longer holds the request
• the worker resolves the site before it reads and parses the document, so a write to an unknown site costs one round trip instead of a full parse; both 415 checks still precede the 404
• the worker answers 415 for a document that parses without a body element, instead of throwing
• the worker logs whether HLX_CONFIG_SERVICE_TOKEN was set when the config service answers 401 or 403, separating a rotated token from an outage
• the worker logs the status that refused a site token exchange, separating it from a site needing no site token

local development

The stand-in in dev/ serves a head.html with the policy and the placeholders, holds one site per store kind, and answers 401 without a token, matching the config service. A local run therefore exercises the nonce rewrite, the trusted-types strip and both store branches.

tested

Unit — 495 tests. src/render/csp.js, src/routes/da-admin.js, src/storage/site.js, src/routes/cookie.js and src/utils/upstream.js at 100% statements; 94.17% over all files.

Live on stage — 55 checks over HTTP, 0 failures, site behind Helix auth for the run.

The preview host answered 401 for all three direct reads without a site token: /index.html, styles.css, component-definition.json. The worker answered 200 on the page it builds, in the same run, with all four href/src values declared by head.html present. head.html source before #270: preview host. After: config service.

One nonce per request on the policy and on all six head scripts, four of them injected by the editor. Second request gives a different value. 'nonce-aem' gone. require-trusted-types-for and move-to-http-header declared at source, absent from the served policy, meta kept. No CSP response header. Quick-edit import map injected first in the document, same nonce. Plain preview host not UE-instrumented, policy still rewritten.

200 without policy and nonce for a non-existing branch — the headHtml ?? '' path.

404 with its own body for an unknown site, and a bare .mp4 on it is a 404 with no HTML. POST refused in order: non-html extension, non-html part content-type and missing data part each 415 ahead of the site lookup, lookup itself 404, none with a Retry-After. All eleven TRUSTED_ORIGINS entries and near-misses landed on the correct side, including the experience-stage.adobe.com entry this PR adds. da:401 recovery shell served, and HEAD without authorization is a bodyless 401 instead of the shell.

In the editor — the canvas loads the page and the editor initialises.

scripts/aem.js and scripts/scripts.js have the minted nonce and execute. They then request the block CSS and JS, nav.plain.html, footer.plain.html and delayed.js — requests the browser only makes if it ran the two scripts, so it accepted the nonce. POST /configuration answers 200 with the page url. The editor reads component-definition.json, component-models.json and component-filters.json, and the component tree fills.

Console: no Refused to execute, no Refused to load, no violates. One notice:

The Content Security Policy directive 'frame-ancestors' is ignored when delivered via a <meta> element.

The notice states the behaviour the rewrite depends on. applyCsp leaves the policy in a <meta> instead of moving it to a response header, so the browser drops frame-ancestors and the editor frames the page.

Open — browser only: editor boot and component tree, CSP violations in console, frame-ancestors not enforced from a meta, nonce accepted instead of only present in the bytes. Two HTTP checks also open: 405 for a source-bus write, and 415 for a body-less document, the only check that runs past a successful lookup.

Quick-edit — https://{branch}--da-live--adobe.aem.page/canvas#/{org}/{site}/{path} points the canvas at stage-preview.da.live and signs in against stage IMS.

The canvas edits and saves. Console: no Refused to execute, no Refused to load, no Failed to resolve module. The import map resolves da-lit and da-y-wrapper from da.live. The bootstrap the worker appends to scripts/scripts.js runs and calls addImportmap; the browser keeps the map the worker injected first and drops the duplicate rules:

An import map rule for specifier 'da-lit' was removed, as it conflicted with an existing rule.

A save leaves no nonce in the stored document.

[benpeter]

live test on stage

Branch deployed to stage. Site behind Helix auth for the run.

H="Authorization: Bearer <stage IMS token>"
U=https://main--{site}--{org}.stage-ue.da.live
P=https://main--{site}--{org}.stage-preview.da.live
N=https://main--{nosuchsite}--{org}.stage-ue.da.live
B=https://nobuild--{site}--{org}.stage-ue.da.live

code(){ curl -so/dev/null -w'%{http_code}\n' "$@"; }
size(){ curl -so/dev/null -w'%{http_code} %{size_download}\n' "$@"; }
body(){ curl -s "$@"; }
head_(){ curl -sD- -o/dev/null "$@"; }
printf '<p>x</p>' > x.html
printf '<html><head></head><frameset cols="50%%,50%%"><frame src="/a"></frameset></html>' > f.html

53 pass, 0 fail, 1 tbd.

config service decides

code  -H"$H" $U/index                                        # 1  -> 200
code  -H"$H" $N/index                                        # 2  -> 404
body  -H"$H" $N/index | grep -c 'no site at this address'    # 3  -> 1
code  -I -H"$H" $N/index                                     # 4  -> 404
size  -H"$H" $N/probe.mp4                                    # 5  -> 404 0
code  -I -H"$H" $U/index                                     # 6  -> 200
body  -H"$H" $U/index | grep -c styles.css                   # 7  -> 1
code  -H"$H" $U/index                                        # 8  -> 200   site is behind Helix auth
code  -H"$H" $U/no-such-doc                                  # 9  -> 200
body  -H"$H" $U/no-such-doc | grep -c '<header'              # 10 -> 1
body  -H"$H" $U/no-such-doc | grep -c styles.css             # 11 -> 1
code  -I -H"$H" $U/no-such-doc                               # 12 -> 404

1. composed page 200
2. unknown site GET 404
3. unknown site GET names the site, not the store
4. unknown site HEAD 404
5. unknown site .mp4 GET is a bare 404, no HTML body
6. HEAD on the site 200
7. head.html content reached the page, 4 of 4 href/src
8. site behind Helix auth still renders a head
9. missing document GETs 200 with the starter template
10. starter template is what came back
11. missing-document page still carries the head
12. same path HEADs 404

head.html source before #270: preview host. After: config service. Checks 7 and 8 hold while the preview host answers 401 — see the resource route below.

write refusals

code    -H"$H" -X POST -F 'data=@x.html;type=text/html'        $N/x.json   # 13 -> 415
code    -H"$H" -X POST -F 'data=@x.html;type=application/json' $N/index    # 14 -> 415
code    -H"$H" -X POST -F 'payload=@x.html;type=text/html'     $N/index    # 15 -> 415
code    -H"$H" -X POST -F 'data=@f.html;type=text/html'        $N/index    # 16 -> 404
body    -H"$H" -X POST -F 'data=@f.html;type=text/html'        $N/index | grep -c 'nothing was written'   # 17 -> 1
head_   -H"$H" -X POST -F 'data=@f.html;type=text/html'        $N/index | grep -ci retry-after            # 18 -> 0

13. non-html extension 415, ahead of the lookup
14. non-html part content-type 415, ahead of the lookup
15. POST with no data part 415, ahead of the lookup
16. unknown site write 404, not sent to a store
17. refused write says nothing was written
18. settled write refusal carries no Retry-After

resource route

code -H"$H" $U/styles/styles.css              # 19 -> 401
code -H"$H" $U/component-definition.json      # 20 -> 401
code -H"$H" $N/styles/styles.css              # 21 -> 404
code -H"$H" $U/index.html                     # 22 -> 401

19. stylesheet answered by the preview host, not by a lookup
20. component-definition.json answered by the preview host, not by a lookup
21. stylesheet on an unknown site answered by the preview host, not by a lookup
22. /index.html is the preview host's copy, not the composed page

da:401 shell

code $U/index                            # 23 -> 401
body $U/index | grep -c 'da:401'         # 24 -> 1
body $U/index | grep -c cors.js          # 25 -> 1
body $U/index | grep -c authorbus        # 26 -> 1
size -I $U/index                         # 27 -> 401 0

23. no Authorization 401
24. 401 body carries the da:401 meta
25. 401 body carries cors.js
26. 401 body names the authorbus extension
27. no Authorization on HEAD is a bodyless 401, not the shell

trusted origins

code -H"$H" -H'Origin: https://evil.example' $U/gimme_cookie   # 28 -> 403

for O in https://da.live https://experience.adobe.com https://experience-stage.adobe.com \
         https://localhost.corp.adobe.com:8080 http://localhost:3000 \
         https://main--da-live--adobe.aem.page
do code -H"Origin: $O" $U/gimme_cookie; done                   # 29,31 -> 401 x6, trusted

for O in https://experience-stage.adobe.com.evil.example https://experience-stage.adobe.com:443 \
         https://my-branch--da-live--adobe.aem.page https://main--da-live--adobe.aem.dev \
         http://localhost.evil.example
do code -H"Origin: $O" $U/gimme_cookie; done                   # 29 -> 403 x5, untrusted

body -H"Origin: https://da.live" $U/gimme_cookie                # 30 -> bare 401 body
body -H'Origin: https://evil.example' $U/gimme_cookie           # 30 -> bare 403 body

28. gimme_cookie refuses an untrusted origin 403
29. all 11 TRUSTED_ORIGINS entries and near-misses on the correct side
30. bodies are the bare 401 and the bare 403
31. experience-stage.adobe.com trusted, the entry this PR adds

A trusted origin reaches the auth check and answers 401 without a bearer. An untrusted one answers 403 before the auth check.

policy rewrite

head_ -H"$H" $U/index | grep -ci '^content-security-policy'                  # 32 -> 0
body  -H"$H" $U/index | grep -ci 'http-equiv="Content-Security-Policy"'      # 33 -> 1
body  -H"$H" $U/index | grep -c "nonce-aem"                                  # 34 -> 0
body  -H"$H" $U/index | grep -o "nonce-[A-Za-z0-9+/]\{24\}" | head -1        # 35,36 -> nonce-<24 chars>
body  -H"$H" $U/index | grep -c require-trusted-types-for                    # 37 -> 0
body  -H"$H" $U/index | grep -c move-to-http-header                          # 38 -> 0

32. no CSP response header on the composed page
33. CSP meta present in head
34. placeholder 'nonce-aem' gone from the policy
35. policy carries a minted nonce
36. nonce is 24 base64 chars
37. require-trusted-types-for declared at source, gone from the served policy
38. move-*-header attribute gone, meta kept — 33 shows the meta survives

nonce reaches the injected scripts

body -H"$H" $U/index | grep -cE 'cors\.js|component-(definition|models|filters)\.json'   # 39 -> 4
body -H"$H" $U/index | grep -c 'urn:adobe:aue:system:ab'                                 # 40 -> 1
body -H"$H" $U/index | grep -o 'nonce="[A-Za-z0-9+/]\{24\}"' | sort -u                   # 41,42 -> one line
body -H"$H" $U/index | grep -c 'nonce="aem"'                                             # 43 -> 0
body -H"$H" $U/index | grep -o "nonce-[A-Za-z0-9+/]\{24\}" | head -1                     # 44 -> run twice, two values

39. all four injected editor scripts present
40. UE instrumentation applied
41. all four injected scripts carry the script-src nonce
42. no head script carries a stale nonce
43. no head script left with nonce="aem"
44. nonce differs between two requests

One line out of 41 means the policy and all six head scripts share one value: aem.js, scripts.js, cors.js, three component-*.json.

quick-edit

body -H"$H" "$P/index?quick-edit=on" | grep -c 'type="importmap"'                          # 45 -> 1
body -H"$H" "$P/index?quick-edit=on" | grep -o '<script[^>]*>' | head -1                   # 46 -> the importmap tag
body -H"$H" "$P/index?quick-edit=on" | grep -c da-y-wrapper                                # 47 -> 1
body -H"$H" "$P/index?quick-edit=on" | grep -o 'nonce="[A-Za-z0-9+/]\{24\}"' | sort -u     # 48 -> one line

45. import map injected
46. import map first in the document
47. quick-edit imports in the map: da-lit, da-y-wrapper
48. import map carries the policy nonce

preview host

body -H"$H" $P/index | grep -c 'urn:adobe:aue:system:ab'                 # 49 -> 0
body -H"$H" $P/index | grep -o "nonce-[A-Za-z0-9+/]\{24\}" | head -1     # 50 -> nonce-<24 chars>

49. plain preview not UE-instrumented
50. preview page policy also rewritten

branch that does not exist

code -H"$H" $B/index                                     # 51 -> 200
body -H"$H" $B/index | grep -c Content-Security-Policy   # 52 -> 0
body -H"$H" $B/index | grep -c 'nonce='                  # 53 -> 0

51. 200
52. no policy in <head>
53. no nonce on head scripts

tbd

• 405 for source-bus write
• 415 for body-less document
• browser: editor boot and component tree, CSP violations in console, frame-ancestors not enforced from a meta, nonce accepted

[benpeter]

two more:

sourcePost reads no Authorization header before the lookup, so neither check needs a bearer.

S=https://main--{sourcebus-site}--{org}.stage-ue.da.live
U=https://main--{site}--{org}.stage-ue.da.live
printf '<html><head></head><frameset cols="50%%,50%%"><frame src="/a"></frameset></html>' > f.html

curl -sD- -o/dev/null -X POST -F 'data=@f.html;type=text/html' $S/x   # 54 -> 405, allow: GET, HEAD, OPTIONS
curl -s              -X POST -F 'data=@f.html;type=text/html' $S/x   # 54 -> "This site is on the source bus, which
                                                                     #        this proxy only reads. Nothing was
                                                                     #        written, and retrying will not help."
curl -so/dev/null -w'%{http_code}\n' -X POST -F 'data=@f.html;type=text/html' $U/x   # 55 -> 415

54. source-bus write 405 with an Allow header, refused before getStore
55. body-less document 415, the one check that runs past the lookup

55 pass, 0 fail.

The config service classifies the store the same way on stage as on production, so a source-bus site under any environment covers 54. The 405 lands before getStore, so the request reaches no store.

[benpeter]

editor test on stage

The canvas host decides which worker is tested. Open the editor on experience.adobe.com, point it at a stage canvas host, and mint the cookie with a stage token — the stage worker binds to da-admin-stage.

1. open the editor, stage canvas host

https://experience.adobe.com/#/@{org-slug}/aem/editor/canvas/main--{site}--{org}.stage-ue.da.live/index

2. get a stage token

Second tab, https://experience-stage.adobe.com. Copy an Authorization value from any request. Check it:

fetch('https://ims-na1-stg1.adobelogin.com/ims/profile/v1',
      { headers: { Authorization: 'Bearer STAGE_TOKEN' } }).then(r => console.log(r.status))
// 200

3. mint the cookie, in the editor tab console, then reload

fetch('https://main--{site}--{org}.stage-ue.da.live/gimme_cookie', {
  headers: { Authorization: 'Bearer STAGE_TOKEN' },
  credentials: 'include'
}).then(r => r.text()).then(console.log)
// cookie set

---

result

Canvas loads the page. aem.js and scripts.js have the minted nonce and execute. They then request the block CSS and JS, nav.plain.html, footer.plain.html and delayed.js — requests the browser only makes if it ran the two scripts, so it accepted the nonce.

POST /configuration answers 200 with the page url. Editor reads component-definition.json, component-models.json, component-filters.json. Component tree fills.

Console: no Refused to execute, no Refused to load, no violates. One notice:

The Content Security Policy directive 'frame-ancestors' is ignored when delivered via a <meta> element.

The notice confirms that applyCsp leaves the policy in a <meta> instead of moving it to a response header, so the browser drops frame-ancestors and the editor frames the page.

[benpeter]

quick-edit test on stage

Open da-live from aem.page, so canvas uses stage-preview.da.live, da-live signs in via stage IMS.

1. open the canvas

https://{branch}--da-live--adobe.aem.page/canvas#/{org}/{site}/{path}

Sign in. Canvas edits and saves.

2. confirm the host runs this branch

printf '<html><head></head><frameset cols="50%,50%"><frame src="/a"></frameset></html>' > f.html
curl -sw'|%{http_code}' -X POST -F 'data=@f.html;type=text/html' \
  https://main--{nosuchsite}--{org}.stage-preview.da.live/index

404 and There is no site at this address come from this branch. main would answer 500

3. check the stored document after a save

fetch('https://main--{site}--{org}.stage-preview.da.live/{path}?quick-edit=on',
      { credentials: 'include' })
  .then(r => r.text())
  .then(t => console.log('nonce in body:', /nonce=/.test(t.split('</head>')[1] || '')))
// false

result

Console: no Refused to execute, no Refused to load, no Failed to resolve module.
import map resolves da-lit and da-y-wrapper from da.live.

bootstrap appended to scripts/scripts.js runs and calls addImportmap. the browser keeps the first injected map and drops duplicate rules:

An import map rule for specifier 'da-lit' was removed, as it conflicted with an existing rule.

no nonce in the stored document after save

[mhaack]

There is already a get503 and a post503 helper can't these be used?

[benpeter]

get503 and post503 have a body and here we're sending a 503 for a resource that couldn't be fetched. I don't think sending a body makes sense here.
resource503 and head503 do the same, I'll consolidate that

[mhaack]

Parts of them have been here before, but should we maybe align on either sending only HTML messages or only text messages?

[benpeter]

that would need changes in UE though.
for GETs the message is what users see in the browser instead of whatever failed, so HTML seems right
for POSTs, UE embeds the message in problem+json so it needs to be plain-text. or at least if it's HTML we'd show HTML markup to the human in front of the screen

[mhaack]

Ok aggreed

[benpeter]

live test on production

H="Authorization: Bearer <prod IMS token>"
U=https://main--{site}--{org}.ue.da.live             # legacy DA site
S=https://main--{sourcebussite}--{org}.ue.da.live    # source-bus site
N=https://main--{nosuchsite}--{org}.ue.da.live       # unknown to the config service
B=https://nobuild--{site}--{org}.ue.da.live          # ref never previewed
A=https://main--{site}--{org}.aem.page               # the preview host
P=/da-ue-livetest-276

code(){ curl -so/dev/null -w'%{http_code}\n' "$@"; }
size(){ curl -so/dev/null -w'%{http_code} %{size_download}\n' "$@"; }
body(){ curl -s "$@"; }
head_(){ curl -sD- -o/dev/null "$@"; }
printf '<body><header></header><main><div><p>livetest</p></div></main><footer></footer></body>' > ok.html
printf '<html><head></head><frameset cols="50%%,50%%"><frame src="/a"></frameset></html>' > f.html

33 pass, 0 fail.

the deployed worker runs this change

body -X POST -F 'data=@f.html;type=text/html' $N/index | grep -c 'no site at this address'   # 1 -> 1
for h in ue stage-ue preview stage-preview; do
  body -X POST -F 'data=@f.html;type=text/html' https://main--{nosuchsite}--{org}.$h.da.live/index
done                                                                                          # 2 -> same body x4

1. the refusal is this change's, not Unable to retrieve AEM branch
2. all four tiers run it

config service decides

code  -H"$H" $U/index                                    # 3  -> 200
body  -H"$H" $U/index | grep -c styles.css               # 4  -> 1
code  -H"$H" $N/index                                    # 5  -> 404
body  -H"$H" $N/index | grep -c '404: Site not found'    # 6  -> 1
code  -H"$H" $B/index                                    # 7  -> 200
body  -H"$H" $B/index | grep -c 'nonce-'                 # 8  -> 0

3. composed page 200
4. head.html content reached the page
5. unknown site 404
6. and the body names the site, not the store
7. a ref that was never previewed still composes
8. it carries no head.html, so no policy and no nonces

CSP

body  -H"$H" $U/index | grep -o 'nonce="[^"]*"' | sort -u | wc -l           # 9  -> 1
body  -H"$H" $U/index | grep -o '<meta[^>]*Content-Security-Policy[^>]*>'   # 10 -> same nonce as 9
body  -H"$H" $U/index | grep -c 'nonce-aem'                                 # 11 -> 0
head_ -H"$H" $U/index | grep -ic '^content-security-policy'                 # 12 -> 0
head_ $A/head.html    | grep -ic '^content-security-policy'                 # 13 -> 1

9. one nonce for the page, on all six script tags
10. the meta carries that same nonce
11. no nonce-aem anywhere
12. no CSP response header on the composed page
13. the preview host still sends the policy as a header

writes

code  -H"$H" -X POST -F 'data=@f.html;type=text/html'  $N/x.json                # 14 -> 415
code  -H"$H" -X POST -F 'data=@f.html;type=text/html'  $N/x.html                # 15 -> 404
code  -H"$H" -X POST -F 'data=@f.html;type=text/html'  $S$P                     # 16 -> 405
head_        -X POST -F 'data=@f.html;type=text/html'  $S$P | grep -i '^allow:' # 17 -> allow: GET, HEAD, OPTIONS
code  -I -H"$H" $S$P                                                            # 18 -> 404
code  -H"$H" -X POST -F 'data=@ok.html;type=text/html' $U$P.html                # 19 -> 201
body  -H"$H" $U$P | grep -c livetest                                            # 20 -> 1
code  -H"$H" https://admin.da.live/source/{org}/{site}$P.html                    # 21 -> 200
code  -H"$H" -X DELETE https://admin.da.live/source/{org}/{site}$P.html          # 22 -> 204

14. .json refused 415, before any lookup
15. unknown site refused 404, no store asked
16. source-bus site refused 405
17. with the methods it does take
18. nothing was written there
19. legacy site write 201
20. read back through the worker
21. and from DA directly
22. scratch document removed

auth

code  -H"$H" -H'Origin: https://experience.adobe.com' $U/gimme_cookie   # 23 -> 200
head_ -H"$H" -H'Origin: https://experience.adobe.com' $U/gimme_cookie \
      | grep -io 'set-cookie: [a-z_]*'                                  # 24 -> auth_token
code  $U/gimme_cookie                                                   # 25 -> 403
code  -H'Origin: https://experience.adobe.com' $U/gimme_cookie          # 26 -> 401
code  $U/index                                                          # 27 -> 401
body  $U/index | grep -c 'da:401'                                       # 28 -> 1
body  $U/index | grep -c cors.js                                        # 29 -> 1
body  $U/index | grep -c authorbus                                      # 30 -> 1
size  -I $U/index                                                       # 31 -> 401 0

23. trusted origin with a token 200
24. one cookie, auth_token. the site needs no site token
25. no Origin 403
26. trusted origin, no token 401
27. no Authorization on GET 401
28. the 401 body carries the da:401 meta
29. and cors.js
30. and the authorbus extension
31. no Authorization on HEAD is a bodyless 401

preview host passthrough

code -H"$H" $U/index.html   # 32 -> 404
code        $A/index.html   # 33 -> 404

32. .html proxies to the preview host
33. which answers the same, so the 404 is the site's state and not the worker's