Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions fork-ci/RELEASE-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,27 @@ nothing.

## After ci/v14

### pr-queue and head-check: no attributes from HEAD; local git calls run no job program (#450)

- Every git call of the queue, the gate and the head check reads attributes from the empty tree
(`GIT_ATTR_SOURCE=4b825dc642cb6eb9a060e54bf8d69288fbee4904`). In a bare repository git 2.43 to 2.45 read
`HEAD:.gitattributes`, and git 2.43+ reads `attr.tree`: a cache whose `HEAD` an earlier job pointed at a commit
holding `* merge=union` made the queue's same-change rule union-merge a conflict.
- The queue's cache must have the `HEAD` the queue writes (`ref: refs/heads/master`, a branch it never makes);
a cache with another `HEAD`, a symlink, none, or that branch made is rebuilt.
- Only the queue's two fetches keep the job's `GIT_EXEC_PATH` (from GitHub) or `pr_queue.GIT_RUN` (another remote);
its merge-tree, cat-file and rev-parse calls in the cache run none of them.
- `head_check.py --remote` without `--bare` is refused (exit 2): checkout mode fetches from the checkout's origin.
- git runs with `LC_ALL=C`, so its messages are matched in one language.
- A fetch from GitHub (https) drops `GIT_TRACE*` and `GIT_CURL_VERBOSE`: the trace would write the login header, and
the gate puts git's error into its public summary.
- A fetch that failed for want of a login says so and names the token (`PR_QUEUE_READ_TOKEN` for the queue, `GH_TOKEN`
for the gate).

**Upgrade:** the queue, the gate and the head check need git 2.43 or newer (`GIT_ATTR_SOURCE` came in 2.41, and
`merge-tree` with it crashed before 2.43); an older git is refused (exit 2), never run without it. A kept queue cache
is rebuilt once if its `HEAD` is not the queue's; nothing else to do.

### pr-queue and head-check: GIT_ATTR_SOURCE and the job's git programs (#442)

- Every git call of the queue and the head check also drops `GIT_ATTR_SOURCE` and `GIT_EXTERNAL_DIFF`. With
Expand Down
8 changes: 5 additions & 3 deletions fork-ci/actions/head-check/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,14 @@ It refuses to start (exit 2, before it reads GitHub, every reason at once) on
a config error (also a base commit without the config), a missing pull
request number (not a `pull_request` event), a lens name in `required-lenses`
that names no lens, a `review` block whose path rules cannot judge the head
(above), a git older than 2.38, a checkout that is not GitHub's
(above), a git older than 2.43, a checkout that is not GitHub's
two-parent merge commit, a checkout that is shallow, or a checkout holding a
file that changes what git's objects or merges say (`info/grafts`, `shallow`,
alternates, `info/attributes`). A refusal is a red required check too. git
runs with no hooks, no fsmonitor, no replace refs and no attributes: the pull
request's `.gitattributes` cannot turn a conflict into a clean merge.
runs with no hooks, no fsmonitor, no replace refs and no attributes (they are
read from the empty tree): neither the pull request's `.gitattributes` nor a
bare repository's `HEAD` or `attr.tree` can turn a conflict into a clean merge.
`--remote` (where a missing read commit is fetched from) needs `--bare`.

It reads only and posts nothing. The one write is a fetch, into the job's own
checkout, of a read commit the head dropped (a rebase or force-push; GitHub
Expand Down
2 changes: 1 addition & 1 deletion fork-ci/actions/head-check/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: pr-queue head check
description: >-
Fails a pull request head that is not the change its review read: the merge queue's head_check (pr-queue), with the
queue's own lens rules and config. Run it in a job of its own, right after the checkout of GitHub's merge commit
with fetch-depth 0, on pull_request events. Reads only and posts nothing. Needs git 2.38+, gh and python3.
with fetch-depth 0, on pull_request events. Reads only and posts nothing. Needs git 2.43+, gh and python3.
inputs:
config:
description: The merge queue's JSON config (pr-queue README, "Config"), a path in the repository, read from the base commit.
Expand Down
8 changes: 5 additions & 3 deletions fork-ci/actions/head-check/head_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
never pass, when anything is unknown: GitHub unreadable after ATTEMPTS (the reviews' flags and inline comments too), a
malformed answer, a checkout whose merge commit merges another head than the one GitHub reports, a read commit that is
not in the checkout and cannot be fetched.
Refuse to start (exit 2, before GitHub is read) on a config or argument error, a git older than 2.38, and a checkout
Refuse to start (exit 2, before GitHub is read) on a config or argument error, a git older than 2.43, and a checkout
that is not GitHub's two-parent merge commit, is shallow or is tampered with. Reads only (a fetch of a missing read
commit into the job's own checkout aside) and posts nothing: a re-run on the same head gives the same answer, and a
run stopped half-way leaves nothing behind.
Expand Down Expand Up @@ -375,7 +375,7 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print):
p.add_argument('--bare', action='store_true', help='read a bare repository (head_gate.py) instead of a checkout')
p.add_argument('--base', default='', help='with --bare: the base tip the config is read from')
p.add_argument('--head', default='', help='with --bare: the head GitHub named when the base was read')
p.add_argument('--remote', default='origin', help='with --bare: where a missing read commit is fetched from')
p.add_argument('--remote', help='with --bare: where a missing read commit is fetched from (default origin)')
a = p.parse_args(argv)
required, problems = argument_problems(a.repo, a.pr, a.required_lenses)
bare = None
Expand All @@ -385,7 +385,9 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print):
if not SHA.fullmatch(v)]
elif a.base or a.head:
problems.append('--base and --head need --bare (a checkout names its own through its merge commit)')
checkout = Checkout(a.path, git, bare=bare, remote=a.remote)
if a.remote is not None and not a.bare:
problems.append("--remote needs --bare (a checkout fetches from its own origin)")
checkout = Checkout(a.path, git, bare=bare, remote=a.remote or 'origin')
if git is None:
problems += filter(None, [pr_queue.git_problem(), checkout.problem()])
code, ok, reason, rows = decide(a.repo, a.pr, a.config, checkout, required, a.pass_drafts, gh, sleep, out, problems)
Expand Down
50 changes: 50 additions & 0 deletions fork-ci/actions/head-check/head_check.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -588,6 +588,15 @@ def test_a_base_or_head_not_fetched_is_refused(self): # refused (2) before Git
self.assertEqual((code, calls), (H.REFUSED, []), out)
self.assertIn(says, '\n'.join(out))

def test_remote_needs_the_bare_mode(self): # #450 item 3: checkout mode fetches from the checkout's own origin
path = self.w.checkout(self.m1, self.read)
for _ in range(2):
out = []
code = H.cli(['--repo', REPO, '--pr', '7', '--config', CONFIG_PATH, '--path', path, '--remote',
'https://github.com/o/r.git'], env={}, gh=fake_gh((0, json.dumps(pr([])))), out=out.append)
self.assertEqual(code, H.REFUSED)
self.assertIn('--remote needs --bare', out[0])

def test_base_and_head_need_the_bare_mode(self): # a checkout names its own: never two answers to one question
path = self.w.checkout(self.m1, self.read)
out = []
Expand Down Expand Up @@ -1204,6 +1213,27 @@ def test_a_conflict_stays_a_conflict_whatever_attribute_tree_the_job_names(self)
self.assertEqual(code, H.REFUSED, out)
self.assertIn(f'the head {head[:12]} does not merge cleanly into the base {b1[:12]}', out)

def test_no_attribute_source_of_the_repository_merges_a_conflict(self): # #450 item 1
# A bare repository the caller cloned: its HEAD names a branch (git 2.43 to 2.45 read HEAD's .gitattributes in a
# bare repository), and its config may name attr.tree (2.43+). Neither may union-merge a conflict.
b0 = self.base('shadow')
head = self.head(b0, 'shadow', **{'docs/a.md': 'head\n'})
b1 = self.base('shadow', parent=b0, **{'docs/a.md': 'base\n'})
evil = self.w.commit({'.gitattributes': '* merge=union\n'})
gate = self.w.checkout(b1, head)
World.git(self.w.dir, '--git-dir', gate, 'fetch', '-q', self.w.remote, f'{evil}:refs/heads/evil')
with open(os.path.join(gate, 'HEAD'), 'w') as f:
f.write('ref: refs/heads/evil\n')
World.git(self.w.dir, '--git-dir', gate, 'config', 'attr.tree', 'refs/heads/evil')
union = subprocess.run(['git', '--git-dir', gate, 'merge-tree', '--write-tree', b1, head], capture_output=True,
env={**os.environ, **GIT_ENV})
self.assertEqual(union.returncode, 0, 'no union merge through attr.tree on this git: the test reads nothing')
comments = [self.p.lens(n, head) for n in EVERY_LENS]
for _ in range(2): # runs again: the same refusal, in this repository (self.check would make a new one)
code, out = run_cli(gate, fake_gh((0, json.dumps(pr(comments, head=head)))))
self.assertEqual(code, H.REFUSED, out)
self.assertIn(f'the head {head[:12]} does not merge cleanly into the base {b1[:12]}', out)

def test_the_merge_follows_a_directory_rename_as_the_queue_does(self): # #413 item 4: merge.directoryRenames
# pr_queue.holds merges with merge.directoryRenames=true, so the check does too: the base adds the config into a
# folder the head renamed, the merge moves it along, and the config's path is gone (git's default, conflict,
Expand Down Expand Up @@ -1264,6 +1294,26 @@ def test_the_gate_runs_no_program_of_the_job_but_gits_own_for_its_fetch(self):
self.assertEqual(envs[0], want)
self.assertTrue(all(e == {} for e in envs[1:]), envs) # cat-file -e after a fetch: local

def test_every_call_reads_no_attributes_and_speaks_one_language(self): # #450 items 1 and 4
github = f'https://github.com/{REPO}.git'
seen = []

def run(argv, **kw):
seen.append(kw['env'])
return types.SimpleNamespace(returncode=0, stdout='', stderr='')
work = tempfile.mkdtemp(prefix='head-check-env-')
self.addCleanup(shutil.rmtree, work, True)
job = {'LC_ALL': 'de_DE.ISO-8859-1', 'LANGUAGE': 'de', 'GIT_ATTR_SOURCE': 'refs/heads/evil'}
with unittest.mock.patch.object(subprocess, 'run', run), unittest.mock.patch.dict(os.environ, job):
head_gate.bare_repository(work, 'gate')
head_gate.fetch(H.Checkout(work, bare=(C, A), remote=github), C, A)
H.Checkout(work, bare=(C, A), remote=github).git('merge-tree', C, A)
H.Checkout(work).git('merge-tree', C, A)
H.Checkout(work).fetch(A)
self.assertEqual(len(seen), 6)
for env in seen:
self.assertEqual((env['LC_ALL'], env['GIT_ATTR_SOURCE']), ('C', '4b825dc642cb6eb9a060e54bf8d69288fbee4904'))

def test_the_checkout_mode_keeps_them_as_before(self): # old callers: its origin is the caller's
for name, call in (('merge-base', lambda d: H.Checkout(d).git('merge-base', C, A)),
('fetch', lambda d: H.Checkout(d).fetch(A))):
Expand Down
2 changes: 1 addition & 1 deletion fork-ci/actions/head-check/head_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -664,7 +664,7 @@ def fetch(checkout, base, head):
"""The base tip and the head, by sha, with their history, into refs only the gate names. None or why not."""
r = checkout.git('-c', 'gc.auto=0', 'fetch', '-q', '--no-tags', '--no-recurse-submodules', '--no-write-fetch-head',
checkout.remote, f'{base}:refs/gate/base', f'{head}:refs/gate/head', fetch=True)
return None if r.returncode == 0 else f'git fetch failed: {r.stderr.strip()[:300]}'
return None if r.returncode == 0 else pr_queue.fetch_failed(r.stderr, 'GH_TOKEN (the gate\'s read token)')


def empty_lenses_problem(checkout, config_path):
Expand Down
13 changes: 13 additions & 0 deletions fork-ci/actions/head-check/head_gate.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -968,6 +968,19 @@ def test_a_fetch_that_fails_is_a_failed_verdict(self): # GitHub dropped a comm
self.assertEqual((code, v['conclusion']), (0, 'failure'), out)
self.assertIn('could not fetch the base and the head', v['title'])

def test_a_fetch_with_no_login_says_so_by_name(self): # #450 item 7: a private repository and no GH_TOKEN
class NoLogin:
remote = f'https://github.com/{REPO}.git'

def git(self, *a, **kw):
return types.SimpleNamespace(returncode=128, stdout='', stderr=(
"fatal: could not read Username for 'https://github.com': terminal prompts disabled\n"))
for _ in range(2):
why = G.fetch(NoLogin(), 'a' * 40, 'b' * 40)
self.assertIn('git fetch failed', why)
self.assertIn('git found no login for GitHub', why)
self.assertIn('GH_TOKEN', why)

def test_a_crash_in_the_rules_is_a_failed_verdict_that_says_so(self): # unknown never passes
def boom(*a, **kw):
raise RuntimeError('broken')
Expand Down
19 changes: 11 additions & 8 deletions fork-ci/actions/pr-queue/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -382,12 +382,14 @@ queue's own bare repository (`--cache`) and gets its login from
A token is in git's environment and `STATE` decides what is re-armed, so run
the queue on a runner (or runner group) that runs no `pull_request` job and no
other repository's job, as a user of its own. The `--state-file` and `--cache`
folders belong to that user alone. git runs with no hooks, no fsmonitor and no
replace refs, and a cache that is not that user's alone (mode `0700`), has a
config the queue did not write, or holds a file that changes what its objects
say or how a merge reads them (`info/grafts`, `shallow`,
`objects/info/alternates`, `objects/info/http-alternates` or
`info/attributes`) is rebuilt, never used. A job of the same user
folders belong to that user alone. git runs with no hooks, no fsmonitor, no
replace refs and no attributes (`GIT_ATTR_SOURCE` is the empty tree, so no
`.gitattributes` of any commit, the cache's `HEAD` or `attr.tree` changes how
a merge reads a file), and a cache that is not that user's alone (mode
`0700`), has a config or a `HEAD` the queue did not write, has a branch at
`HEAD`, or holds a file that changes what its objects say or how a merge reads
them (`info/grafts`, `shallow`, `objects/info/alternates`,
`objects/info/http-alternates` or `info/attributes`) is rebuilt, never used. A job of the same user
can still change the cache between that check and git's next read: only the
runner rule above stops that.

Expand All @@ -401,7 +403,8 @@ runner rule above stops that.
It refuses to start on a GitHub-hosted runner (its disk, and with it the plan,
goes with the job), with a `--repo` that is not `owner/name`, a relative path,
a path inside the job's workspace (the next checkout empties it), a config
error, or a git older than 2.38 (`merge-tree --write-tree`).
error, or a git older than 2.43 (`merge-tree --write-tree` that honours
`GIT_ATTR_SOURCE`).

`--state-file` (`STATE`) holds the plan of the run that turned auto-merge off:
`{"repo", "pr", "old", "base", "read"}` (and `"told"` once the pull request was
Expand Down Expand Up @@ -474,7 +477,7 @@ the re-arm still needs exactly the change the lenses read.

The queue needs:

- a self-hosted runner whose disk outlives the job, with git 2.38+, `gh` and
- a self-hosted runner whose disk outlives the job, with git 2.43+, `gh` and
`python3`: the plan in `--state-file` and the cache must be there for the
next run. That runner (or runner group) runs no `pull_request` job and no
other repository's job, as a user of its own, and the state and cache
Expand Down
2 changes: 1 addition & 1 deletion fork-ci/actions/pr-queue/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: pr-queue
description: >-
A merge queue for a repository whose ruleset requires pull requests to be up to date: each run brings one waiting
pull request up to date and turns auto-merge back on only for the head the review read (README.md). Needs a
self-hosted runner whose disk outlives the job, with git 2.38+, gh and python3.
self-hosted runner whose disk outlives the job, with git 2.43+, gh and python3.
inputs:
repo:
description: The repository, owner/name.
Expand Down
Loading
Loading