diff --git a/fork-ci/RELEASE-NOTES.md b/fork-ci/RELEASE-NOTES.md index 3cac29249c3b1..43c5804d41847 100644 --- a/fork-ci/RELEASE-NOTES.md +++ b/fork-ci/RELEASE-NOTES.md @@ -13,6 +13,34 @@ nothing. ## After ci/v14 +### pr-queue and head-check: the cache's objects are checked before each merge check (#457) + +- Before its merge check the queue runs `git fsck` on its cache. git checks an object's hash on only some reads, + never of a tree or blob `merge-tree` reads: a tree or blob an earlier job stored under one side's name made a real + conflict merge clean. In an update the check runs in the background, while GitHub updates the branch. A cache + `git fsck` finds broken, or one a fetch fails on for a reason that is not the network or the login, is downloaded + from scratch once and checked again; one still broken is not merged in (the run keeps `STATE`, and the PR comment + says the cache fails `git fsck`). Old formats `git fetch` takes (time zone `+99999`, a tree out of order, an odd + `.gitmodules`) are ignored; a hash mismatch, a missing object or a corrupt pack never is. A cache that fails the same + way as right after its download from scratch is not downloaded again: a person must look. A fetch's `git gc` runs + in the foreground. `git fsck` reads every object on one core: about 28 seconds for 270,000 objects, at most once + per run. +- The queue's cache and the gate's repository are made with `GIT_DEFAULT_HASH=sha1` and + `GIT_DEFAULT_REF_FORMAT=files` (every git call gets both), and `HEAD` is written by the queue whatever branch git's + `init` names (git 3.0 plans `main`). A git that still makes another format (an `[extensions]` section) is refused + with the extension named: no cache, nothing fetched or merged; in the gate the head's check run fails. +- git calls that reach no remote (the queue's, the gate's `merge-tree`, and `head_check.py` in bare mode) drop + `GIT_TRACE*` and `GIT_CURL_VERBOSE`, as the fetches from GitHub do; `HTTPS://` counts as https in any case. The + head check's checkout mode keeps them, as before (a partial clone may fetch from `origin`). +- A plain `http://` remote, a `transport::address` remote and one that starts with `-` are refused: GitHub redirects + http to https, where git hands the token over, a remote helper runs any program, and git reads `-...` as an + option. `head_check.py --bare --remote ''` is refused (exit 2) instead of becoming `origin`. +- The gate checks git (2.43 or newer) and its remote before it makes its repository: either problem fails the head's + check run ("not judged: ..."), with nothing made or fetched. + +**Upgrade:** nothing to do. A kept cache is checked on its next merge check (it was made by the same code, so it +passes); a broken one is downloaded from scratch once. + ### pr-queue and head-check: no attributes from HEAD; local git calls run no job program (#450) - Every git call of the queue, the gate and the head check reads attributes from the empty tree @@ -31,8 +59,10 @@ nothing. for the gate). **Upgrade:** the queue, the gate and the head check need git 2.43 or newer (`GIT_ATTR_SOURCE` came in 2.41, and -`merge-tree` with it crashed before 2.43); an older git is refused (exit 2), never run without it. A kept queue cache -is rebuilt once if its `HEAD` is not the queue's; nothing else to do. +`merge-tree` with it crashed before 2.43); an older git never merges or judges anything. The queue refuses to start +(exit 2; it reads its config with git first and decides nothing on it), the head check refuses (exit 2), and the gate +fails the head's check run before it makes its repository (#457). A kept queue cache is rebuilt once if its `HEAD` is +not the queue's; nothing else to do. ### pr-queue and head-check: GIT_ATTR_SOURCE and the job's git programs (#442) diff --git a/fork-ci/actions/head-check/README.md b/fork-ci/actions/head-check/README.md index 5b8700a51ec67..9db61e3bf65c8 100644 --- a/fork-ci/actions/head-check/README.md +++ b/fork-ci/actions/head-check/README.md @@ -76,7 +76,10 @@ alternates, `info/attributes`). A refusal is a red required check too. git runs with no hooks, no fsmonitor, no replace refs and no attributes (they are read from the empty tree): neither the pull request's `.gitattributes` nor a bare repository's `HEAD` or `attr.tree` can turn a conflict into a clean merge. -`--remote` (where a missing read commit is fetched from) needs `--bare`. +`--remote` (where a missing read commit is fetched from) needs `--bare`, and +is refused when it is empty, plain `http://` (GitHub would redirect it to +https, and git would hand the token over), a `transport::address` remote +helper, or starts with `-` (git would read it as an option). It reads only and posts nothing. The one write is a fetch, into the job's own checkout, of a read commit the head dropped (a rebase or force-push; GitHub @@ -220,7 +223,13 @@ How it works: from the caller's tree. Nothing of the pull request is checked out: the head and the base tip are read from the API and fetched by sha into a new bare repository, and `head_check.decide` (the same rule code as the action, - in bare mode) judges them. The config is read from the base tip. + in bare mode) judges them. The config is read from the base tip. Before it + makes that repository the gate checks its git (2.43 or newer) and its + remote (https, a path or a remote name): either problem fails the head's + check run ("not judged: ..."), with no repository made and nothing fetched. + The repository is made as the queue makes its cache (SHA-1 objects and files + refs, whatever git's defaults): a git that makes another format fails the + head too, never judged in a repository its config misnames. - **A check run belongs to a commit, not to a pull request.** The newest check run of the name on a head counts for every pull request with that head. So the gate judges a head, never one pull request alone: diff --git a/fork-ci/actions/head-check/head_check.py b/fork-ci/actions/head-check/head_check.py index cfceb17b45db6..4da69ceba5a0b 100644 --- a/fork-ci/actions/head-check/head_check.py +++ b/fork-ci/actions/head-check/head_check.py @@ -375,7 +375,8 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print): p.add_argument('--bare', action='store_true', help='read a bare repository (head_gate.py) instead of a checkout') p.add_argument('--base', default='', help='with --bare: the base tip the config is read from') p.add_argument('--head', default='', help='with --bare: the head GitHub named when the base was read') - p.add_argument('--remote', help='with --bare: where a missing read commit is fetched from (default origin)') + p.add_argument('--remote', help='with --bare: where a missing read commit is fetched from (default origin): a ' + 'remote name, a path or a URL; not plain http, not empty') a = p.parse_args(argv) required, problems = argument_problems(a.repo, a.pr, a.required_lenses) bare = None @@ -387,7 +388,9 @@ def cli(argv=None, env=None, gh=gh_run, sleep=time.sleep, git=None, out=print): problems.append('--base and --head need --bare (a checkout names its own through its merge commit)') if a.remote is not None and not a.bare: problems.append("--remote needs --bare (a checkout fetches from its own origin)") - checkout = Checkout(a.path, git, bare=bare, remote=a.remote or 'origin') + elif a.remote is not None and (why := pr_queue.remote_problem(a.remote)): # an empty one is no "origin" + problems.append(f'--remote: {why}') + checkout = Checkout(a.path, git, bare=bare, remote='origin' if a.remote is None else a.remote) if git is None: problems += filter(None, [pr_queue.git_problem(), checkout.problem()]) code, ok, reason, rows = decide(a.repo, a.pr, a.config, checkout, required, a.pass_drafts, gh, sleep, out, problems) diff --git a/fork-ci/actions/head-check/head_check.test.py b/fork-ci/actions/head-check/head_check.test.py index a1736303342f8..038fd5dfa0bf1 100644 --- a/fork-ci/actions/head-check/head_check.test.py +++ b/fork-ci/actions/head-check/head_check.test.py @@ -597,6 +597,19 @@ def test_remote_needs_the_bare_mode(self): # #450 item 3: checkout mode fetche self.assertEqual(code, H.REFUSED) self.assertIn('--remote needs --bare', out[0]) + def test_a_remote_that_is_no_url_or_path_is_refused(self): # #457 item 4: never "origin" for an empty one + path = self.w.checkout(self.m1, self.read) + for remote, says in (('', 'the remote is empty'), ('http://github.com/o/r.git', 'plain http'), + ('ext::sh -c id', 'remote helper'), ('--upload-pack=touch x', 'starts with -')): + with self.subTest(remote): + for _ in range(2): + calls, out = [], [] + code = H.cli(['--repo', REPO, '--pr', '7', '--config', CONFIG_PATH, '--bare', '--path', path, + '--base', self.m1, '--head', self.read, '--remote', remote], env={}, + gh=fake_gh((0, json.dumps(pr([])))), out=out.append) + self.assertEqual((code, calls), (H.REFUSED, []), out) + self.assertIn(says, '\n'.join(out)) + def test_base_and_head_need_the_bare_mode(self): # a checkout names its own: never two answers to one question path = self.w.checkout(self.m1, self.read) out = [] @@ -1314,6 +1327,29 @@ def run(argv, **kw): for env in seen: self.assertEqual((env['LC_ALL'], env['GIT_ATTR_SOURCE']), ('C', '4b825dc642cb6eb9a060e54bf8d69288fbee4904')) + def test_the_gates_local_calls_and_github_fetch_write_no_trace(self): # #457 item 4: merge-tree's stderr is + github = f'https://github.com/{REPO}.git' # put into the public summary + trace = {'GIT_TRACE': '1', 'GIT_TRACE_PACKET': '1', 'GIT_TRACE2_EVENT': '1', 'GIT_CURL_VERBOSE': '1', + 'GIT_TRACE_REDACT': '0'} + for name, call, kept in ( + ('git init', lambda d: head_gate.bare_repository(d, 'gate'), False), + ('merge-tree', lambda d: H.Checkout(d, bare=(C, A), remote=github).git('merge-tree', C, A), False), + ('the gate fetch', lambda d: head_gate.fetch(H.Checkout(d, bare=(C, A), remote=github), C, A), False), + ('the checkout mode', lambda d: H.Checkout(d).git('merge-base', C, A), True)): # its own origin + with self.subTest(name): + seen = [] + + def run(argv, **kw): + seen.append(kw['env']) + return types.SimpleNamespace(returncode=0, stdout='', stderr='') + work = tempfile.mkdtemp(prefix='head-check-env-') + self.addCleanup(shutil.rmtree, work, True) + with unittest.mock.patch.object(subprocess, 'run', run), unittest.mock.patch.dict(os.environ, trace): + call(work) + self.assertTrue(seen) + got = {k: seen[0][k] for k in trace if k in seen[0]} + self.assertEqual(got, trace if kept else {}) + def test_the_checkout_mode_keeps_them_as_before(self): # old callers: its origin is the caller's for name, call in (('merge-base', lambda d: H.Checkout(d).git('merge-base', C, A)), ('fetch', lambda d: H.Checkout(d).fetch(A))): diff --git a/fork-ci/actions/head-check/head_gate.py b/fork-ci/actions/head-check/head_gate.py index 19472906d534c..b6247f5dd2df9 100644 --- a/fork-ci/actions/head-check/head_gate.py +++ b/fork-ci/actions/head-check/head_gate.py @@ -645,19 +645,15 @@ def finish(repo, sha, run, verdict_path, gh, out, env, sleep=time.sleep): # --- judge ---------------------------------------------------------------------------------------------------------- def bare_repository(work, name): - """(path, None): a new, empty bare repository in `work` with the queue's own config (pr_queue.CACHE_CONFIG: no - remote, no hook path, no helper), never one an earlier run left; or (None, why it could not be made).""" + """(path, None): a new, empty bare repository in `work` made as the queue makes its cache (pr_queue.init_bare: + CACHE_CONFIG, no remote, no hook path, no helper, SHA-1 objects and files refs), never one an earlier run left; or + (None, why it could not be made).""" try: path = tempfile.mkdtemp(prefix=name + '-', dir=work) except OSError as e: return None, f'could not make a folder in {work}: {e}' - r = subprocess.run(['git', *pr_queue.GIT_SAFE, 'init', '-q', '--bare', '--template=', path], text=True, - capture_output=True, env=pr_queue.git_env(os.environ)) - if r.returncode != 0: - return None, f'git init failed: {r.stderr.strip()[:300]}' - with open(os.path.join(path, 'config'), 'w', encoding='utf-8') as f: - f.write(pr_queue.CACHE_CONFIG) - return path, None + problem = pr_queue.init_bare(path) + return (None, problem) if problem else (path, None) def fetch(checkout, base, head): @@ -712,6 +708,8 @@ def judge(a, gh, out, env, decide=head_check.decide, sleep=time.sleep): state, why, _, base, numbers = preflight(a, gh) if state != 'go': ok, reason = False, f'not judged: {why}' + elif problem := pr_queue.git_problem() or pr_queue.remote_problem(a.remote): # before git runs at all + ok, reason = False, f'not judged: {problem}' else: path, problem = bare_repository(a.work, f'head-{a.sha[:12]}') checkout = head_check.Checkout(path or a.work, bare=(base, a.sha), remote=a.remote) @@ -722,7 +720,7 @@ def judge(a, gh, out, env, decide=head_check.decide, sleep=time.sleep): and (problem := empty_lenses_problem(checkout, a.config)): ok, reason = False, f'not judged: {problem}' else: - shared = list(filter(None, [pr_queue.git_problem(), checkout.problem()])) + shared = list(filter(None, [checkout.problem()])) required, _ = head_check.required_names(a.required_lenses) for n in numbers: # every one: the strictest verdict is the head's code, good, why, rows = decide(a.repo, str(n), a.config, checkout, required, False, gh, sleep, out, diff --git a/fork-ci/actions/head-check/head_gate.test.py b/fork-ci/actions/head-check/head_gate.test.py index 29372c8ebea8a..12525ec0164d6 100644 --- a/fork-ci/actions/head-check/head_gate.test.py +++ b/fork-ci/actions/head-check/head_gate.test.py @@ -18,7 +18,7 @@ T = importlib.util.module_from_spec(_spec) _spec.loader.exec_module(T) # first: its guard puts a fake gh on PATH, drops tokens and isolates git -import argparse, base64, io, json, shutil, tempfile, types, unittest, unittest.mock # noqa: E402 +import argparse, base64, io, json, shutil, subprocess, tempfile, types, unittest, unittest.mock # noqa: E402 sys.path.insert(0, HERE) import head_check as H # noqa: E402 @@ -968,6 +968,50 @@ def test_a_fetch_that_fails_is_a_failed_verdict(self): # GitHub dropped a comm self.assertEqual((code, v['conclusion']), (0, 'failure'), out) self.assertIn('could not fetch the base and the head', v['title']) + def test_an_old_git_fails_the_head_before_anything_is_made_or_fetched(self): # #457 item 4: as the notes say + view = T.pr([self.p.lens('correctness', self.read)], head=self.read) + old = 'git 2.43 or newer is needed (GIT_ATTR_SOURCE); found: git 2.42' + with unittest.mock.patch.object(G.pr_queue, 'git_problem', lambda *a: old), \ + unittest.mock.patch.object(G, 'fetch', side_effect=AssertionError('fetched with an old git')): + for _ in range(2): # runs again: the same failure + code, out, _ = self.judge(self.routes(self.read), view) + v = self.verdict() + self.assertEqual((code, v['conclusion']), (0, 'failure'), out) + self.assertIn('not judged: git 2.43 or newer', v['title']) + self.assertEqual(os.listdir(self.work), [], 'no repository made') + + def test_a_remote_that_is_no_url_or_path_fails_the_head(self): # #457 item 4: http hands the token over + view = T.pr([self.p.lens('correctness', self.read)], head=self.read) + for remote, says in (('http://github.com/o/r.git', 'plain http'), ('ext::sh -c id', 'remote helper')): + with self.subTest(remote), \ + unittest.mock.patch.object(G, 'fetch', side_effect=AssertionError('fetched')): + code, out, _ = self.judge(self.routes(self.read), view, remote=remote) + v = self.verdict() + self.assertEqual((code, v['conclusion']), (0, 'failure'), out) + self.assertIn('not judged', v['title']) + self.assertIn(says, v['title'] + v['summary']) + self.assertEqual(os.listdir(self.work), []) + + def test_a_git_that_makes_another_format_fails_the_head(self): # #457 item 3: git 3.0 defaults ignored + view = T.pr([self.p.lens('correctness', self.read)], head=self.read) + real = subprocess.run + + def run(cmd, **kw): + r = real(cmd, **kw) + if 'init' in cmd and r.returncode == 0: + with open(os.path.join(cmd[-1], 'config'), 'a') as f: + f.write('[extensions]\n\trefStorage = reftable\n') + return r + with unittest.mock.patch.object(subprocess, 'run', run): + for _ in range(2): + code, out, _ = self.judge(self.routes(self.read), view) + v = self.verdict() + self.assertEqual((code, v['conclusion']), (0, 'failure'), out) + self.assertIn('git init made a repository with extensions.refStorage=reftable', + v['title'] + v['summary']) + code, out, _ = self.judge(self.routes(self.read), view) # a git that keeps the formats: judged again + self.assertEqual(self.verdict()['conclusion'], 'success', out) + def test_a_fetch_with_no_login_says_so_by_name(self): # #450 item 7: a private repository and no GH_TOKEN class NoLogin: remote = f'https://github.com/{REPO}.git' diff --git a/fork-ci/actions/pr-queue/README.md b/fork-ci/actions/pr-queue/README.md index 102bc712d221f..a5bd6b2805595 100644 --- a/fork-ci/actions/pr-queue/README.md +++ b/fork-ci/actions/pr-queue/README.md @@ -389,9 +389,31 @@ a merge reads a file), and a cache that is not that user's alone (mode `0700`), has a config or a `HEAD` the queue did not write, has a branch at `HEAD`, or holds a file that changes what its objects say or how a merge reads them (`info/grafts`, `shallow`, `objects/info/alternates`, -`objects/info/http-alternates` or `info/attributes`) is rebuilt, never used. A job of the same user -can still change the cache between that check and git's next read: only the -runner rule above stops that. +`objects/info/http-alternates` or `info/attributes`) is rebuilt, never used. +git checks an object's hash on only some reads (never a tree or a blob +`merge-tree` reads), so the queue runs `git fsck` on the cache before its merge +check: an object stored under another object's name (a tree or a blob that +makes a conflict merge clean) is found there. In an update it runs in the +background right after the first fetch, while GitHub updates the branch, and +the queue waits for it before it fetches the new head; in a run that finishes +a stopped one it runs right after the fetch. Old formats that `git fetch` +takes (a commit with time zone `+99999`, a tree out of order, an odd +`.gitmodules`) are no problem (`FSCK_FORMAT_IDS`); an object that does not +hash to its name, a missing or unreadable one, or a corrupt pack is. A cache +`git fsck` finds broken, or one a fetch fails on for a reason that is not the +network or the login, is downloaded from scratch once and checked again; one +still broken after that is not merged in (STATE stays, and the PR comment says +the cache fails `git fsck`). If a later run's `git fsck` gives the same lines +as right after that download, the history GitHub serves is what fails: the +queue does not download it again, and a person must look. A fetch's automatic +`git gc` runs in the foreground, so no repack races the check. `git fsck` +reads every object on one core: about 30 seconds for 270,000 objects, at most +once per run (the queue makes at most one merge check per run). The cache is +made with SHA-1 objects and files refs whatever git's defaults +(`GIT_DEFAULT_HASH`, `GIT_DEFAULT_REF_FORMAT`; git 3.0 plans others); a git +that makes another format is refused, never written over. A job of the same +user can still change the cache between that check and git's next read: only +the runner rule above stops that. | Exit | Meaning | | ---- | ----------------------------------------------------------------------------------------------------------- | diff --git a/fork-ci/actions/pr-queue/pr_queue.py b/fork-ci/actions/pr-queue/pr_queue.py index 4d9074533a767..8a9879ed97cd6 100755 --- a/fork-ci/actions/pr-queue/pr_queue.py +++ b/fork-ci/actions/pr-queue/pr_queue.py @@ -86,6 +86,11 @@ # so git_problem refuses it: GIT_MIN). Kept apart from GIT_ENV, which other actions share (git_clean.test.py). EMPTY_TREE = '4b825dc642cb6eb9a060e54bf8d69288fbee4904' GIT_ATTRS = {'GIT_ATTR_SOURCE': EMPTY_TREE} +# What a new repository is made with, whatever the job's environment or git's own default says (git 3.0 plans SHA-256 +# and reftable): the queue's cache and the gate's repository hold SHA-1 objects and files refs, as GitHub's commits, +# EMPTY_TREE and CACHE_CONFIG (format 0, no extensions) are. git reads GIT_DEFAULT_REF_FORMAT from 2.45 (an older one +# makes files refs anyway); init_bare refuses a repository a git made otherwise. +GIT_FORMATS = {'GIT_DEFAULT_HASH': 'sha1', 'GIT_DEFAULT_REF_FORMAT': 'files'} # One language for git's messages: the queue and the gate match them (GONE_RE, NO_LOGIN_RE) and read them as UTF-8. GIT_LANG = {'LC_ALL': 'C'} # Never passed on to git (git_env): variables of the job that point git at another repository, object store, index, @@ -106,8 +111,9 @@ # run none of them, GIT_EXEC_PATH included. GIT_RUN = ('GIT_EXEC_PATH', 'GIT_SSH', 'GIT_SSH_COMMAND', 'GIT_PROXY_COMMAND', 'GIT_ASKPASS', 'SSH_ASKPASS', 'GIT_ALLOW_PROTOCOL') -# Traces of an https fetch: they write its headers (the login) and its talk with GitHub to stderr or a file, and the -# gate puts git's stderr into a public summary. Dropped there; a fetch from another remote carries no token of ours. +# Traces: they write an https fetch's headers (the login) and its talk with GitHub to stderr or a file, and the gate +# puts git's stderr into a public summary (a merge-tree's too). Dropped from every call but a fetch from another +# remote, which carries no token of ours (remote_problem refuses plain http, where GitHub's redirect would ask for it). GIT_TRACE_PREFIXES = ('GIT_TRACE', 'GIT_CURL_VERBOSE') GIT_READ_SECONDS = 60 # each git call of the config read # The cache's HEAD, as the queue writes it (what `git init --template=` writes): a branch that is never made, so HEAD @@ -115,6 +121,31 @@ CACHE_HEAD = 'ref: refs/heads/master\n' # git's words for a fetch that needed a login it did not have (none given, or one GitHub refused) NO_LOGIN_RE = re.compile(r'could not read (Username|Password)|terminal prompts disabled|Authentication failed', re.I) +# git's words for a fetch the network failed (curl's errors, a transfer cut off). Any other failed fetch may be the +# cache's fault (an object git cannot read under a name the fetch needs), so the cache is checked (fetch_checked). +NETWORK_RE = re.compile(r'unable to access|RPC failed|Could not resolve host', re.I) +# git fsck's message ids for an object whose hash is right but whose content is in an old or odd format: a commit +# with time zone +99999, a zero-padded date or a missing space; a tree with entries out of order or twice; a +# .gitmodules or .gitattributes git would not check out. git fetch takes them (transfer.fsckObjects is off) and old +# imports hold them, but plain git fsck calls them errors: a cache with such history would fail every check and be +# downloaded again on every run. objects_problem ignores them. What the check is for stays an error, since none of +# it is a message id: an object that does not hash to its name, a missing or unreadable object, a broken link, a +# corrupt pack or index. So do the ids for a commit git cannot parse (bad parent or tree line, no tree), an object it +# could not read, and the tags the queue never fetches (--no-tags). Each id is one git 2.43 (GIT_MIN) knows: git +# refuses to run with an unknown one. +FSCK_FORMAT_IDS = ('badDate', 'badDateOverflow', 'badEmail', 'badName', 'badTimezone', 'missingAuthor', + 'missingCommitter', 'missingEmail', 'missingNameBeforeEmail', 'missingSpaceBeforeDate', + 'missingSpaceBeforeEmail', 'multipleAuthors', 'zeroPaddedDate', 'duplicateEntries', 'treeNotSorted', + 'gitmodulesBlob', 'gitmodulesLarge', 'gitmodulesName', 'gitmodulesPath', 'gitmodulesSymlink', + 'gitmodulesUpdate', 'gitmodulesUrl', 'gitattributesBlob', 'gitattributesLarge', + 'gitattributesLineLength') +FSCK_IGNORE = tuple(a for i in FSCK_FORMAT_IDS for a in ('-c', f'fsck.{i}=ignore')) +# A fetch's automatic gc runs before the fetch returns, never detached: a repack in the background deletes packs and +# loose objects while the next git fsck reads them (a false "broken", and a download from scratch). +NO_DETACH = ('-c', 'gc.autoDetach=false', '-c', 'maintenance.autoDetach=false') +# In the cache: git fsck's lines right after a download from scratch that it still found broken. The same lines on a +# later run are GitHub's own history, not an earlier job's object: that run does not download it all again. +FSCK_KNOWN = 'queue-fsck-known' # The cache's whole config, as the queue writes it: a cache with any other config (a planted `url.*.insteadOf`, helper # or hook path) is rebuilt, never run. CACHE_CONFIG = '[core]\n\trepositoryformatversion = 0\n\tfilemode = true\n\tbare = true\n' @@ -335,17 +366,65 @@ def no_token_env(): def git_env(env, remote=None): - """`env` for a git call: without GIT_DROP, with GIT_ENV, GIT_ATTRS, GIT_LANG and no prompt. `remote`: None for a - call that reaches no remote, else the URL or remote name it fetches from, which decides what of GIT_RUN it keeps - (an https fetch also drops the traces, GIT_TRACE_PREFIXES). Tokens are the caller's to drop.""" + """`env` for a git call: without GIT_DROP, with GIT_ENV, GIT_ATTRS, GIT_FORMATS, GIT_LANG and no prompt. `remote`: + None for a call that reaches no remote, else the URL or remote name it fetches from, which decides what of GIT_RUN + it keeps. Only a fetch from a remote that is not https keeps the traces (GIT_TRACE_PREFIXES). Tokens are the + caller's to drop.""" drop, prefixes = set(GIT_DROP), GIT_DROP_PREFIXES if remote is None: drop.update(GIT_RUN) - elif remote.startswith('https://'): + prefixes += GIT_TRACE_PREFIXES + elif remote.lower().startswith('https://'): # git reads the scheme in any case: HTTPS:// is https too drop.update(set(GIT_RUN) - {'GIT_EXEC_PATH'}) prefixes += GIT_TRACE_PREFIXES return {**{k: v for k, v in env.items() if k not in drop and not k.startswith(prefixes)}, - 'GIT_TERMINAL_PROMPT': '0', **GIT_LANG, **GIT_ENV, **GIT_ATTRS} + 'GIT_TERMINAL_PROMPT': '0', **GIT_LANG, **GIT_ENV, **GIT_ATTRS, **GIT_FORMATS} + + +def remote_problem(remote): + """Why a fetch that carries the run's token (GIT_AUTH: gh's login for https://github.com) must not go to + `remote`, or None. Plain http keeps the job's traces (git_env), and GitHub redirects it to https, where git then + asks the helper for the token; a `transport::address` remote runs whichever remote helper it names on any address, + https included. An ssh URL, a path or a file:// URL gets no token.""" + if not remote: + return 'the remote is empty' + if remote.startswith('-'): + return f'the remote {remote[:100]} starts with -: git would read it as an option' + if remote.lower().startswith('http://'): + return (f'the remote {remote[:100]} is plain http: GitHub would redirect it to https and git would hand the ' + 'token over with the traces kept. Use https') + if re.match(r'[A-Za-z][A-Za-z0-9+.-]*::', remote): + return f'the remote {remote[:100]} names a remote helper (transport::address): give a URL or a path' + return None + + +def init_bare(path): + """Makes the empty folder `path` a bare repository as the queue's cache and the gate's repository are: + `git init --bare --template=` (no hooks, no sample files), then CACHE_CONFIG and CACHE_HEAD written over what git + wrote, never left to git's defaults (git 3.0 plans `main`). None, or why it could not be made. A repository git made + in another format than SHA-1 objects with files refs (format 0, no extensions: a git that ignored GIT_FORMATS) is + refused, never written over: CACHE_CONFIG would then name a format its refs or objects are not in.""" + r = subprocess.run(['git', *GIT_SAFE, 'init', '-q', '--bare', '--template=', path], text=True, + capture_output=True, env=git_env(no_token_env())) + if r.returncode != 0: + return f'git init failed: {r.stderr.strip()[:300]}' + try: + with open(os.path.join(path, 'config'), encoding='utf-8', errors='replace') as f: + made = f.read() + version = re.search(r'^\s*repositoryformatversion\s*=\s*(\S*)', made, re.M | re.I) + section = re.search(r'^\s*\[\s*extensions\s*\]([^[]*)', made, re.M | re.I) + if section or (version and version.group(1) != '0'): + said = ', '.join(f'extensions.{k}={v}' for k, v in + re.findall(r'^\s*([\w.-]+)\s*=\s*(\S*)', section.group(1) if section else '', re.M)) + said = said or (f'format version {version.group(1)}' if version else 'an [extensions] section') + return (f'git init made a repository with {said[:200]}: the ' + 'queue reads only SHA-1 objects and files refs (format 0, no extensions), and this git ignored ' + 'GIT_DEFAULT_HASH=sha1 and GIT_DEFAULT_REF_FORMAT=files') + write_text(os.path.join(path, 'config'), CACHE_CONFIG) + write_text(os.path.join(path, 'HEAD'), CACHE_HEAD) + except OSError as e: + return f'could not read or write its config: {e}' + return None def fetch_failed(stderr, login): @@ -446,6 +525,8 @@ def __init__(self, repo, cfg, state, cache, lock, sleep=time.sleep, read_token=N self.repo, self.cfg, self.state, self.cache, self.lock, self.sleep = repo, cfg, state, cache, lock, sleep self.read_token, self.write_token, self.said = read_token, write_token, set() self.remote = f'https://github.com/{repo}.git' + self.later, self.fetch_error, self.fsck_said = None, '', '' # check_later, fetch, objects_problem + self.unknown = 'the queue could not fetch from GitHub' # why fetch_checked knows nothing (PR comment words) self.review = lens.Review(cfg.bot_logins, cfg.override_prefix, cfg.override_association, cfg.lens_author_associations) @@ -514,14 +595,20 @@ def say_once(self, key, text): self.said.add(key) print(text) - def git(self, *a, fetch=False): + def git(self, *a, fetch=False, errors='strict'): """git in the cache, with the run's read token, and with no token at all when there is none (never gh's own login of the machine). `fetch`: the call fetches from the remote (only Queue.fetch's two), so it keeps what of - the job's GIT_RUN the remote may need; every other call runs none of it (git_env).""" + the job's GIT_RUN the remote may need; every other call runs none of it (git_env). `errors`: how git's output + is decoded (UTF-8).""" + argv, env = self.git_call(*a, fetch=fetch) + return subprocess.run(argv, text=True, encoding='utf-8', errors=errors, capture_output=True, env=env) + + def git_call(self, *a, fetch=False): + """The argv and environment of Queue.git's call.""" token = self.read_token env = self.gh_env(token) if token else no_token_env() - return subprocess.run(['git', *GIT_SAFE, *(GIT_AUTH if token else GIT_AUTH[:2]), '--git-dir', self.cache, *a], - text=True, capture_output=True, env=git_env(env, self.remote if fetch else None)) + return (['git', *GIT_SAFE, *(GIT_AUTH if token else GIT_AUTH[:2]), '--git-dir', self.cache, *a], + git_env(env, self.remote if fetch else None)) # --- step 1 and 4: the queue's own git repository ------------------------------------------------------------- def cache_problem(self): @@ -582,20 +669,70 @@ def own_cache(self): except OSError as e: print(f'could not make {new}: {e}') return False - r = subprocess.run(['git', *GIT_SAFE, 'init', '-q', '--bare', '--template=', new], text=True, - capture_output=True, env=git_env(no_token_env())) - if r.returncode != 0: - print(f'could not make {self.cache}: {r.stderr.strip()[:300]}') + problem = init_bare(new) + if problem: + print(f'could not make {self.cache}: {problem}') return False try: - write_text(os.path.join(new, 'config'), CACHE_CONFIG) - write_text(os.path.join(new, 'HEAD'), CACHE_HEAD) # what git init wrote; said here, not left to its default os.rename(new, self.cache) except OSError as e: # something made the cache between the two steps: the next run sets it aside print(f'could not move the new repository to {self.cache}: {e}') return False return True + def objects_problem(self, r=None): + """Why the cache's objects are not what GitHub sent, or None. git checks an object's hash on only some reads + (rev-parse, merge-base and merge-tree of a commit; never a tree or blob merge-tree reads, nor a commit rev-list + walks), and the cache is kept between runs: a tree or blob an earlier job wrote under the name of one side's + (a loose object where the real one is loose too, as a small fetch leaves it, or a pack of its own) makes a real + conflict merge clean. git fsck checks the hash of every object, loose and packed, and that each ref's history + is complete; an old format GitHub serves is no problem (FSCK_FORMAT_IDS). It reads every object, on one core: + about 30 s for 270,000 objects. `r`: the answer of a git fsck already run (check_later). Keeps git fsck's + lines in self.fsck_said (fetch_checked compares them).""" + if r is None: + r = self.git(*FSCK_IGNORE, 'fsck', '--no-dangling', '--no-reflogs', '--no-progress', errors='replace') + if r.returncode == 0: + self.fsck_said = '' + return None + said = [line for line in (r.stderr + r.stdout).splitlines() if line.strip()] + self.fsck_said = '\n'.join(sorted(set(said))) + return f'has objects git fsck finds broken or missing ({said[0][:200] if said else f"exit {r.returncode}"})' + + def check_later(self): + """objects_problem's git fsck, started in the background (update(), right after its fetch): it reads the cache + while the disarm, the branch update and the wait for the new head run, none of which touches the cache. The + next fetch_checked waits for its answer before it fetches, and update() stops it if the run ends first.""" + argv, env = self.git_call(*FSCK_IGNORE, 'fsck', '--no-dangling', '--no-reflogs', '--no-progress') + try: + self.later = subprocess.Popen(argv, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + encoding='utf-8', errors='replace') + except OSError as e: # checked in the foreground then + print(f'could not start git fsck in the background: {e}') + + def checked_before(self): + """The background check's answer: None when none ran, '' for a healthy cache, else the problem.""" + p, self.later = self.later, None + if p is None: + return None + out, err = p.communicate() + return self.objects_problem(subprocess.CompletedProcess(p.args, p.returncode, out, err)) or '' + + def stop_check(self): + """The background check, if one still runs, stopped and reaped.""" + p, self.later = self.later, None + if p is not None: + p.kill() + p.communicate() + + def known_broken(self): + """git fsck's lines the cache gave right after its download from scratch (FSCK_KNOWN), or None.""" + try: + with open(os.path.join(self.cache, FSCK_KNOWN), encoding='utf-8', errors='replace', + opener=lambda p, f: os.open(p, f | os.O_NOFOLLOW)) as f: + return f.read() + except OSError: + return None + def has(self, sha): return bool(SHA.fullmatch(sha or '')) and self.git('cat-file', '-e', sha + '^{commit}').returncode == 0 @@ -603,23 +740,69 @@ def fetch(self, base, n, *shas): """The base branch (refs/queue/base), PR n's head and `shas` into the cache. True when every sha is there, False when the refs came but a sha is not on GitHub, None when nothing could be fetched (no cache, a git or network error: nothing is known).""" + self.fetch_error = '' # git fetch's words when it failed (fetch_checked) if not re.fullmatch(r'[\w./-]+', base or '') or not all(SHA.fullmatch(s or '') for s in shas): return False if not self.own_cache(): return None - r = self.git('fetch', '-q', '--no-tags', self.remote, f'+refs/heads/{base}:refs/queue/base', + r = self.git(*NO_DETACH, 'fetch', '-q', '--no-tags', self.remote, f'+refs/heads/{base}:refs/queue/base', f'+refs/pull/{n}/head:refs/queue/pr{n}', fetch=True) if r.returncode != 0: print(f'#{n}: {fetch_failed(r.stderr, READ_LOGIN)}') + self.fetch_error = r.stderr return None missing = [s for s in shas if not self.has(s)] if missing: # a read commit no ref reaches any more (GitHub keeps it) - r = self.git('fetch', '-q', '--no-tags', self.remote, *missing, fetch=True) + r = self.git(*NO_DETACH, 'fetch', '-q', '--no-tags', self.remote, *missing, fetch=True) if r.returncode != 0 and not GONE_RE.search(r.stderr): # a network error is no "not on GitHub" print(f'#{n}: {fetch_failed(r.stderr, READ_LOGIN)} (of {", ".join(s[:12] for s in missing)})') + self.fetch_error = r.stderr return None return all(self.has(s) for s in shas) + def fetch_checked(self, base, n, *shas, later=False): + """fetch, with the cache's objects checked (objects_problem) before anything reads them: True, False or None + as fetch returns them, and None for a cache that is still broken after a download from scratch. A cache found + broken (by this run's earlier background check, after the fetch, or after a fetch that failed for a cause + that is neither the network nor the login) is set aside and downloaded from scratch once, then checked again. + A cache whose check gives the same lines as right after its own download from scratch (FSCK_KNOWN) is not + downloaded again. `later`: the check after a good fetch runs in the background (check_later). self.unknown + says why None, in words for a PR comment (no runner paths).""" + self.unknown = 'the queue could not fetch from GitHub' + before = self.checked_before() + problem = before + if not problem: + got = self.fetch(base, n, *shas) + if got is None and self.fetch_error and not (NO_LOGIN_RE.search(self.fetch_error) + or NETWORK_RE.search(self.fetch_error)): + problem = self.objects_problem() + elif got and before is None: + if later: + self.check_later() + return got + problem = self.objects_problem() + if not problem: + return got + if self.fsck_said and self.fsck_said == self.known_broken(): + print(f'#{n}: {self.cache} {problem}: the same way as right after its download from scratch, so the ' + 'history GitHub serves fails this git fsck. Not downloaded again and not merged in: a person must ' + 'look') + self.unknown = ("the queue's git cache fails git fsck the same way as right after a download from " + "GitHub (the run's log says why), so a person must look") + return None + print(f'{self.cache} {problem}: rebuilding it') + got = self.fetch(base, n, *shas) if set_aside(self.cache) else None + if got and (problem := self.objects_problem()): + print(f'#{n}: {self.cache}, fetched again from GitHub, still {problem}: not merged in') + try: + write_text(os.path.join(self.cache, FSCK_KNOWN), self.fsck_said) + except OSError as e: + print(f'could not keep git fsck\'s lines in {self.cache}: {e}') + self.unknown = ("the queue's git cache fails git fsck even when fetched again from GitHub " + "(the run's log says why)") + return None + return got + def same_change(self, base, n, head, read): """Step 4's check (holds) on GitHub's commits: True, False (any other content or a git error), or None when nothing could be fetched (the caller keeps STATE and tries again). Only full shas count.""" @@ -627,7 +810,7 @@ def same_change(self, base, n, head, read): return False if head == read: return True - got = self.fetch(base, n, head, read) + got = self.fetch_checked(base, n, head, read) # checked right before the merge check return holds(self.git, head, read) if got else got # --- which armed PRs wait, and why ------------------------------------------------------------------------------ @@ -997,12 +1180,15 @@ def update(self, n, dry=False): f'and re-arm if the new head stays the change read at {(read or "?")[:12]}' + ('' if read else f' ({why})')) return 0 - fetched = bool(read) and self.fetch(base, n, read) - if read and fetched is None: # nothing changed yet: an armed PR stays as it was, the next run tries again - print(f'#{n}: could not fetch from GitHub (see above): nothing changed, the next run tries again') - return 1 - self.write_state({'repo': self.repo, 'pr': n, 'old': old, 'base': base, 'read': read}) - code = self.update_off(n, old, base, read, fetched, why) + try: + fetched = bool(read) and self.fetch_checked(base, n, read, later=True) + if read and fetched is None: # nothing changed yet: an armed PR stays as it was, the next run tries again + print(f'#{n}: {self.unknown} (see above): nothing changed, the next run tries again') + return 1 + self.write_state({'repo': self.repo, 'pr': n, 'old': old, 'base': base, 'read': read}) + code = self.update_off(n, old, base, read, fetched, why) + finally: + self.stop_check() if code != SILENT: self.drop_state() return code @@ -1034,7 +1220,7 @@ def update_off(self, n, old, base, read, fetched, why): return SILENT same = self.same_change(base, n, head, read) if same is None: - print(f'#{n}: could not fetch {head[:12]} to check it: STATE stays, the next run finishes it') + print(f'#{n}: {self.unknown}, so {head[:12]} was not checked: STATE stays, the next run finishes it') return SILENT if not same: return self.leave_off(n, f'the new head {head[:12]} is not exactly the change read at {read[:12]} merged ' @@ -1118,8 +1304,8 @@ def recover(self): f'off at {old}; {why or "it stored no read commit"}, so it was not re-armed: which ' 'head the review vouches for is unclear.', self.read_advice(why or '')) elif same is None: # never OFF and silent: say it once on the PR, as for a gh error - print(f'#{n}: could not fetch from GitHub to check {head[:12]}: trying again next run, no other PR meanwhile') - return self.tell_once(st, n, f'the queue could not fetch {head[:12]} from GitHub to check it') + print(f'#{n}: {self.unknown}, so {head[:12]} was not checked: trying again next run, no other PR meanwhile') + return self.tell_once(st, n, f'{self.unknown}, so it could not check {head[:12]}') elif same: r = self.arm(n, head) if r.returncode == 0: diff --git a/fork-ci/actions/pr-queue/pr_queue.test.py b/fork-ci/actions/pr-queue/pr_queue.test.py index 5d4b4d1f4f4f1..bd6701328e294 100644 --- a/fork-ci/actions/pr-queue/pr_queue.test.py +++ b/fork-ci/actions/pr-queue/pr_queue.test.py @@ -41,7 +41,7 @@ def _gh_never_ran(): atexit.register(_gh_never_ran) -import argparse, contextlib, fcntl, io, json, subprocess, unittest, unittest.mock # noqa: E402 +import argparse, contextlib, fcntl, io, json, subprocess, unittest, unittest.mock, zlib # noqa: E402 HERE = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(0, HERE) @@ -49,6 +49,8 @@ def _gh_never_ran(): import lens # noqa: E402 import pr_queue as Q # noqa: E402 +REAL_HOLDS = Q.holds # for a test that patches Q.holds and still needs the real one + REPO = 'owner/proj' CFG_RAW = { 'required_checks': ['build', 'test', 'review-check'], 'merge_method': 'squash', 'delete_branch': True, @@ -67,7 +69,8 @@ def _gh_never_ran(): # --- the World: real commits, a local "GitHub" ---------------------------------------------------------------------- # What git_env sets on every git call of the queue, whatever the job's environment holds. FORCED = {'GIT_TERMINAL_PROMPT': '0', 'LC_ALL': 'C', **Q.GIT_ENV, - 'GIT_ATTR_SOURCE': '4b825dc642cb6eb9a060e54bf8d69288fbee4904'} # the empty tree + 'GIT_ATTR_SOURCE': '4b825dc642cb6eb9a060e54bf8d69288fbee4904', # the empty tree + 'GIT_DEFAULT_HASH': 'sha1', 'GIT_DEFAULT_REF_FORMAT': 'files'} # #457 item 3: what CACHE_CONFIG names GIT_ENV = {'GIT_AUTHOR_NAME': 't', 'GIT_AUTHOR_EMAIL': 't@t', 'GIT_COMMITTER_NAME': 't', 'GIT_COMMITTER_EMAIL': 't@t', 'GIT_EDITOR': 'true'} @@ -420,7 +423,7 @@ def test_a_network_error_on_the_fetch_by_sha_is_no_gone_commit(self): # None ( real = self.q.git def git(*a, **kw): - if a[:1] == ('fetch',) and not any(x.startswith('+refs/') for x in a): + if 'fetch' in a and not any(x.startswith('+refs/') for x in a): # after NO_DETACH's -c flags return argparse.Namespace(returncode=128, stdout='', stderr='fatal: unable to access: timed out') return real(*a, **kw) self.w.set('refs/pull/7/head', self.c) @@ -1050,6 +1053,101 @@ def test_the_plan_survives_a_new_process_and_an_emptied_lock_folder(self): # S self.assertFalse(self.plan_exists()) +class ObjectsCheckInAnUpdate(Recover): # #457 fix round: the cache check in a real run + def plant_corrupt(self, name): + """A corrupt loose object under `name` in the cache: what an earlier job could leave there.""" + os.makedirs(os.path.join(self.q.cache, 'objects', name[:2]), exist_ok=True) + with open(os.path.join(self.q.cache, 'objects', name[:2], name[2:]), 'wb') as f: + f.write(zlib.compress(b'blob 5\0evil\n')) + + def test_the_check_runs_while_github_updates_the_branch(self): # state lens cost: overlaps the update and wait + f, seen, foreground = self.fake(polls=2), {}, [] + git = self.q.git + + def counted(*a, **kw): + if 'fsck' in a: + foreground.append(a) + return git(*a, **kw) + self.q.git = counted + + def gh(*a): + if QueueGh.kind(a) == 'update': + seen['check'] = self.q.later + return f(*a) + code, out = self.update(gh) + self.assertEqual((code, f.armed, f.comments), (0, self.up, []), out) + self.assertIsInstance(seen.get('check'), subprocess.Popen, 'started before the branch update') + self.assertEqual(foreground, [], 'checked once, in the background') + self.assertIsNone(self.q.later) + + def test_a_run_that_ends_before_the_merge_check_leaves_no_check_running(self): + procs, real = [], subprocess.Popen + + def popen(*a, **kw): # subprocess.run's own Popen too: only the check counts + p = real(*a, **kw) + if 'fsck' in a[0]: + procs.append(p) + return p + with unittest.mock.patch.object(Q.subprocess, 'Popen', popen): + code, out = self.update(self.fake(after=None)) # no new head in time: SILENT before the merge check + self.assertEqual(code, Q.SILENT, out) + with self.assertRaises(Killed): # killed while it waits for the new head + self.update(self.fake(kill_at='poll')) + self.assertEqual(len(procs), 2) + self.assertEqual([p.poll() is None for p in procs], [False, False], 'stopped and reaped') + self.assertIsNone(self.q.later) + + def test_a_broken_object_found_while_github_updates_is_rebuilt_before_the_merge_check(self): + f = self.fake() + self.assertTrue(run_quiet(self.q.fetch, 'main', 7, self.c)[0]) # an earlier run's cache + self.plant_corrupt(self.up) # the name update-branch's head will have + code, out = self.update(f) + self.assertEqual((code, f.armed, f.comments), (0, self.up, []), out) + self.assertIn('rebuilding it', out) + + def test_the_background_checks_answer_is_read_before_the_merge_check(self): # no fetch trips over this one + f, fscks = self.fake(), [] + self.assertTrue(run_quiet(self.q.fetch, 'main', 7, self.c)[0]) # an earlier run's cache + git = self.q.git + + def counted(*a, **kw): + if 'fsck' in a: + fscks.append(a) + return git(*a, **kw) + self.q.git = counted + self.plant_corrupt('ab' * 20) # a name no fetch needs: only git fsck sees it + code, out = self.update(f) + self.assertEqual((code, f.armed, f.comments), (0, self.up, []), out) + self.assertIn('rebuilding it', out) + self.assertEqual(len(fscks), 1, 'the background answer, then only the rebuilt cache checked in the foreground') + + def test_a_fetch_a_broken_object_fails_in_recover_is_rebuilt(self): # trust lens 1: never stuck + self.fake() + self.assertTrue(run_quiet(self.q.fetch, 'main', 7, self.c)[0]) # an earlier run's cache + self.plant_corrupt(self.up) + f = self.fake(armed=False, h1=self.up) # killed after update-branch: the new head must be fetched + code, out = self.recover(f, self.plan()) + self.assertEqual((code, f.armed, f.comments), (0, self.up, []), out) + self.assertIn('rebuilding it', out) + + def test_a_cache_still_broken_says_so_not_could_not_fetch(self): # state lens 2, trust nit 2: log and PR comment + self.q.objects_problem = lambda *a: 'has objects git fsck finds broken or missing (planted)' + f = self.fake() + code, out = self.update(f) + self.assertEqual((code, f.armed, f.comments), (Q.SILENT, None, []), out) + self.assertIn('fails git fsck', out) + self.assertNotIn('could not fetch', out) + g = self.fake(armed=False, h1=self.up) + for _ in range(2): + code, out = self.recover(g, self.saved_plan()) + self.assertEqual((code, g.armed, len(g.comments)), (1, None, 1), out) + self.assertIn('fails git fsck', out) + self.assertNotIn('could not fetch', out) + self.assertIn('fails git fsck', g.comments[0]) + self.assertNotIn('could not fetch', g.comments[0]) + self.assertNotIn(self.w.dir, g.comments[0], 'no runner path in a PR comment') + + class WhoseCommentsCount(Base): """Anyone can comment on a public repository: only comments of the configured lens associations, shown and never edited, are lens reads (lens.Review.counts). Everyone else's neither vouch for a head nor block one.""" @@ -1182,6 +1280,111 @@ def test_a_cache_with_another_config_or_open_to_others_is_rebuilt(self): self.assertIn("is not this user's alone", out) self.assertEqual(os.stat(self.q.cache).st_mode & 0o777, 0o700) + def init_writes(self, head=None, config=None): + """subprocess.run, where `git init` also writes `head` to HEAD and adds `config` to the config, as another git + would (git 3.0: HEAD names main; a reftable or SHA-256 default).""" + real = subprocess.run + + def run(cmd, **kw): + r = real(cmd, **kw) + if 'init' in cmd and r.returncode == 0: + if head is not None: + with open(os.path.join(cmd[-1], 'HEAD'), 'w') as f: + f.write(head) + if config is not None: + with open(os.path.join(cmd[-1], 'config'), 'a') as f: + f.write(config) + return r + return unittest.mock.patch.object(subprocess, 'run', run) + + def test_a_git_whose_init_names_another_branch_still_makes_the_queues_head(self): # #457 item 2: git 3.0, main + with self.init_writes(head='ref: refs/heads/main\n'): + for _ in range(3): # made once, then kept: never rebuilt for a HEAD git chose + got, out = self.fetch(self.c) + self.assertTrue(got, out) + self.assertNotIn('rebuilding', out) + with open(os.path.join(self.q.cache, 'HEAD')) as f: + self.assertEqual(f.read(), Q.CACHE_HEAD) + self.assertIsNone(self.q.cache_problem()) + + def test_a_head_or_config_that_is_a_link_is_never_followed(self): # #457 item 4: cache_problem's O_NOFOLLOW + for name, text in (('HEAD', Q.CACHE_HEAD), ('config', Q.CACHE_CONFIG)): + with self.subTest(name): + self.assertTrue(self.fetch(self.c)[0]) + target = os.path.join(self.w.dir, name + '-elsewhere') # what the queue would write, but elsewhere + with open(target, 'w') as f: + f.write(text) + path = os.path.join(self.q.cache, name) + os.remove(path) + os.symlink(target, path) + self.assertIn('cannot be checked', self.q.cache_problem()) + got, out = self.fetch(self.up) + self.assertTrue(got, out) + self.assertIn('cannot be checked', out) + self.assertIn('rebuilding it', out) + self.assertFalse(os.path.islink(path)) + got, out = self.fetch(self.up) # runs again: kept + self.assertTrue(got, out) + self.assertNotIn('rebuilding', out) + + def test_the_jobs_default_formats_never_reach_the_cache(self): # #457 item 3: git 3.0 defaults, or a job's + envs, real = [], subprocess.run + + def run(cmd, **kw): + if 'init' in cmd: + envs.append(kw.get('env')) + return real(cmd, **kw) + job = {'GIT_DEFAULT_HASH': 'sha256', 'GIT_DEFAULT_REF_FORMAT': 'reftable'} + with unittest.mock.patch.dict(os.environ, job), unittest.mock.patch.object(subprocess, 'run', run): + for _ in range(2): + got, out = self.fetch(self.c) + self.assertTrue(got, out) + self.assertNotIn('rebuilding', out) + self.assertEqual(self.q.git('rev-parse', '--show-object-format').stdout.strip(), 'sha1') + self.assertEqual(len(envs), 1, 'made once') + self.assertEqual({k: envs[0].get(k) for k in job}, Q.GIT_FORMATS) + + def test_a_git_that_makes_another_format_is_refused_clearly(self): # #457 item 3: refused, never written over + for extension in ('\n[extensions]\n\trefStorage = reftable\n', '\n[extensions]\n\tobjectFormat = sha256\n'): + with self.subTest(extension.split()[1]), self.init_writes(config=extension): + shutil.rmtree(self.q.cache, ignore_errors=True) + for _ in range(2): # runs again: the same refusal, nothing fetched, nothing merged + got, out = self.fetch(self.c) + self.assertIsNone(got, out) + self.assertIn('git init made a repository with extensions.' + extension.split()[1], out) + self.assertIn('GIT_DEFAULT_HASH=sha1', out) + self.assertFalse(os.path.lexists(self.q.cache)) + got, out = self.fetch(self.c) # a git that keeps the formats again: made as the queue's + self.assertTrue(got, out) + + def test_a_git_that_makes_another_format_version_is_refused_clearly(self): # state lens 4: M6, no [extensions] + real = subprocess.run + + def run(cmd, **kw): + r = real(cmd, **kw) + if 'init' in cmd and r.returncode == 0: + path = os.path.join(cmd[-1], 'config') + with open(path) as f: + text = f.read() + with open(path, 'w') as f: + f.write(text.replace('repositoryformatversion = 0', 'repositoryformatversion = 1')) + return r + with unittest.mock.patch.object(subprocess, 'run', run): + for _ in range(2): + got, out = self.fetch(self.c) + self.assertIsNone(got, out) + self.assertIn('git init made a repository with format version 1', out) + self.assertFalse(os.path.lexists(self.q.cache)) + + def test_a_git_whose_init_writes_more_config_still_makes_the_queues_config(self): # state lens 4: M17 (Windows) + with self.init_writes(config='\n[core]\n\tignorecase = true\n\tsymlinks = false\n'): + for _ in range(3): # made once, then kept: never rebuilt for config git chose + got, out = self.fetch(self.c) + self.assertTrue(got, out) + self.assertNotIn('rebuilding', out) + with open(os.path.join(self.q.cache, 'config')) as f: + self.assertEqual(f.read(), Q.CACHE_CONFIG) + def own(comment): """A PR comment for a fake `gh pr view`: a plain body is one the queue's own login posted.""" @@ -2256,6 +2459,258 @@ def test_a_head_planted_between_two_runs_is_found_by_the_next(self): # stopped self.assertIn('rebuilding it', out.getvalue()) +class HoldsChecksTheCachesObjects(unittest.TestCase): # #457 item 1 + """git checks an object's hash on only some reads, never of a tree or blob merge-tree reads. The cache is kept + between runs: an earlier job on the runner (the same user) could write another tree or blob under the name of one + side's, and merge-tree would merge a real conflict clean. Before each merge check the queue runs git fsck on the + cache; a cache it finds broken is set aside and fetched again, never merged in.""" + + def setUp(self): + w = self.w = World() + self.addCleanup(w.close) + m0 = w.commit({'c.json': '{"a": 1}\n'}) + self.read = w.commit({'c.json': '{"a": 2}\n'}, [m0]) + self.m1 = w.commit({'c.json': '{"a": 3}\n'}, [m0]) + # what a planted tree makes merge-tree say: the base's side alone, the change read gone without a word + self.head = w.keep(w.git(w.work, 'commit-tree', self.m1 + '^{tree}', '-p', self.read, '-p', self.m1, + '-m', 'lost')) + w.set('refs/heads/main', self.m1) + w.set('refs/pull/7/head', self.head) + self.run_again() + got, out = run_quiet(self.q.fetch, 'main', 7, self.read) + self.assertIs(got, True, out) + self.assertIs(Q.holds(self.q.git, self.head, self.read), False, 'a real conflict') + + def run_again(self): + """The next run: a new Queue on the same cache.""" + self.q = make_queue(self.w.dir, remote=self.w.remote) + self.fscks = [] + git = self.q.git + + def counted(*a, **kw): + if 'fsck' in a: + self.fscks.append(a) + return git(*a, **kw) + self.q.git = counted + + def cache_git(self, *a): + return World.git(self.w.dir, '--git-dir', self.q.cache, *a) + + def plant(self, kind='tree'): + """The read side's tree object replaced by the base side's tree, at the same name (a loose object, as a small + fetch leaves it: git reads a packed object before a loose one, so a packed one is replaced by a pack). `blob`: + the read side's c.json replaced by the merge base's (merge-tree reads a blob's content only when the three + names differ, and then takes the base side's change: clean).""" + if kind == 'tree': + victim, donor = self.cache_git('rev-parse', self.read + '^{tree}', self.m1 + '^{tree}').split() + else: + victim, donor = self.cache_git('rev-parse', self.read + ':c.json', self.read + '^:c.json').split() + body = subprocess.run(['git', '--git-dir', self.q.cache, 'cat-file', kind, donor], capture_output=True, + check=True).stdout + self.write_object(victim, kind.encode() + b' %d\0' % len(body) + body) + self.assertIs(Q.holds(self.q.git, self.head, self.read), True, + f'the planted {kind} does not merge the conflict clean on this git: the test reads nothing') + + def write_object(self, name, raw): + """`raw` (zlib-compressed) as the loose object `name`, over the one a small fetch left there.""" + path = os.path.join(self.q.cache, 'objects', name[:2], name[2:]) + self.assertTrue(os.path.exists(path), 'a small fetch leaves loose objects') + os.chmod(path, 0o644) + with open(path, 'wb') as f: + f.write(zlib.compress(raw)) + + def same_change(self): + return run_quiet(self.q.same_change, 'main', 7, self.head, self.read) + + def test_a_replaced_tree_object_makes_the_run_refuse(self): # the lens's case + for planted in range(2): # planted again in the rebuilt cache: refused again + with self.subTest(planted=planted): + self.plant() + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertIn('git fsck', out) + self.assertIn('rebuilding it', out) + fsck = subprocess.run(['git', '--git-dir', self.q.cache, 'fsck', '--no-dangling'], capture_output=True) + self.assertEqual(fsck.returncode, 0, 'the rebuilt cache holds only what was fetched') + for _ in range(2): # runs again: the rebuilt cache is kept and gives the same answer + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertNotIn('rebuilding', out) + self.assertEqual(len(self.fscks), 1, 'one check before the merge check') + + def test_a_cache_still_broken_after_the_rebuild_is_never_merged(self): # nothing is known: STATE stays + self.q.objects_problem = lambda: 'has objects git fsck finds broken (planted)' + with unittest.mock.patch.object(Q, 'holds', side_effect=AssertionError('merged in a broken cache')): + for _ in range(2): + same, out = self.same_change() + self.assertIsNone(same, out) + self.assertIn('fetched again', out) + + def test_a_kept_healthy_cache_is_checked_and_kept(self): # a cache from before the fix: the same files + for _ in range(2): + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertNotIn('rebuilding', out) + self.assertEqual(len(self.fscks), 1) + self.cache_git('repack', '-adq') # git's own gc packs it: still the queue's + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertNotIn('rebuilding', out) + + def test_a_new_empty_cache_is_made_fetched_and_checked(self): + shutil.rmtree(self.q.cache) + for _ in range(2): + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertNotIn('rebuilding', out) + self.assertEqual(len(self.fscks), 1) + + def test_a_fetch_stopped_half_way_leaves_a_cache_that_still_answers(self): # killed while writing objects + for name in (os.path.join('objects', 'pack', 'tmp_pack_x1Y2z3'), os.path.join('objects', 'ab', 'tmp_obj_x')): + path = os.path.join(self.q.cache, name) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, 'wb') as f: + f.write(b'PACK\0\0\0\2half') + for _ in range(2): + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + + + def test_a_replaced_blob_object_makes_the_run_refuse(self): # state lens 4: not only trees (and no + self.plant('blob') # --connectivity-only, which misses it) + for again in range(2): # rebuilt once, then kept with the same answer + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + (self.assertNotIn if again else self.assertIn)('rebuilding', out) + + def test_a_replaced_object_no_ref_reaches_makes_the_run_refuse(self): # M16: git fsck says exit 1 alone + """The change read was force-pushed away: the queue fetched it by sha, so no ref in the cache reaches its + tree, and git fsck finds the planted tree with exit code 1 only (no missing object: nothing reachable is + broken). Any non-zero exit is a broken cache.""" + lost = self.w.keep(self.w.git(self.w.work, 'commit-tree', self.m1 + '^{tree}', '-p', self.m1, '-m', 'lost')) + self.w.set('refs/pull/7/head', lost) + self.head = lost + self.run_again() + got, out = run_quiet(self.q.fetch, 'main', 7, self.read) + self.assertIs(got, True, out) + self.assertIs(Q.holds(self.q.git, self.head, self.read), False, 'a real conflict') + self.plant() + fsck = subprocess.run(['git', '--git-dir', self.q.cache, *Q.FSCK_IGNORE, 'fsck', '--no-dangling', + '--no-reflogs'], capture_output=True) + self.assertEqual(fsck.returncode, 1, 'the planted tree is reachable: the test reads nothing new') + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertIn('rebuilding it', out) + + def odd_base(self): + """main moves to a commit with time zone +99999: old imports have such commits, `git fetch` takes them and + plain `git fsck` calls them badTimezone.""" + raw = (f'tree {self.cache_git("rev-parse", self.m1 + "^{tree}")}\nparent {self.m1}\n' + 'author t 1700000000 +99999\ncommitter t 1700000000 +99999\n\nold import\n') + odd = self.w.keep(subprocess.run(['git', 'hash-object', '--literally', '-t', 'commit', '-w', '--stdin'], + cwd=self.w.work, input=raw, text=True, capture_output=True, + check=True).stdout.strip()) + self.w.set('refs/heads/main', odd) + return odd + + def test_old_format_history_github_serves_is_no_broken_cache(self): # state lens 1: never a refetch loop + self.odd_base() + got, out = run_quiet(self.q.fetch, 'main', 7, self.read) + self.assertIs(got, True, out) + plain = subprocess.run(['git', '--git-dir', self.q.cache, 'fsck', '--no-dangling'], text=True, + capture_output=True) + self.assertIn('badTimezone', plain.stdout + plain.stderr, + 'plain git fsck takes this commit: the test reads nothing') + for _ in range(3): # runs again: an answer every time, never a rebuild + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertNotIn('rebuilding', out) + self.assertEqual(len(self.fscks), 1) + self.plant() # what the check is for still counts with the old format ignored + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + self.assertIn('rebuilding it', out) + + def test_a_missing_object_is_still_broken_with_the_old_format_ignored(self): # FSCK_FORMAT_IDS names no loss + self.odd_base() + self.assertIs(run_quiet(self.q.fetch, 'main', 7, self.read)[0], True) + blob = self.cache_git('rev-parse', self.read + ':c.json') + os.remove(os.path.join(self.q.cache, 'objects', blob[:2], blob[2:])) + for again in range(2): + self.run_again() + same, out = self.same_change() + self.assertIs(same, False, out) + (self.assertNotIn if again else self.assertIn)('rebuilding', out) + + def test_a_fresh_download_that_fails_the_same_way_is_not_downloaded_again(self): # state lens 1: cost + self.odd_base() # as a newer git's fsck may refuse what no FSCK_IGNORE id names + fetches = [] + real_fetch = Q.Queue.fetch + + def fetch(q, *a): + fetches.append(a) + return real_fetch(q, *a) + with unittest.mock.patch.object(Q, 'FSCK_IGNORE', (), create=True), \ + unittest.mock.patch.object(Q.Queue, 'fetch', fetch), \ + unittest.mock.patch.object(Q, 'holds', side_effect=AssertionError('merged in a broken cache')): + self.run_again() + same, out = self.same_change() + self.assertIsNone(same, out) + self.assertIn('rebuilding it', out) + self.assertEqual(len(fetches), 2, 'one download from scratch') + for _ in range(2): # the next runs: the same lines from a fresh download, so no download again + fetches.clear() + self.run_again() + same, out = self.same_change() + self.assertIsNone(same, out) + self.assertNotIn('rebuilding', out) + self.assertIn('the same way', out) + self.assertIn('fails git fsck', self.q.unknown) + self.assertEqual(len(fetches), 1, out) + with unittest.mock.patch.object(Q, 'holds', REAL_HOLDS): + self.plant() # a new broken object is another answer: rebuilt again + fetches.clear() + self.run_again() + same, out = self.same_change() + self.assertIsNone(same, out) + self.assertIn('rebuilding it', out) + self.assertEqual(len(fetches), 2) + + def test_a_fetch_a_broken_object_fails_is_rebuilt_once(self): # trust lens 1: never stuck on "could not fetch" + head = self.w.keep(self.w.git(self.w.work, 'commit-tree', self.m1 + '^{tree}', '-p', self.read, '-p', self.m1, + '-m', 'lost again')) + self.w.set('refs/pull/7/head', head) + os.makedirs(os.path.join(self.q.cache, 'objects', head[:2]), exist_ok=True) + path = os.path.join(self.q.cache, 'objects', head[:2], head[2:]) + with open(path, 'wb') as f: # a corrupt object under the name the next fetch needs + f.write(zlib.compress(b'blob 5\0evil\n')) + for _ in range(2): # the first run rebuilds; the next keeps it and answers the same + self.run_again() + same, out = run_quiet(self.q.same_change, 'main', 7, head, self.read) + self.assertIs(same, False, out) + self.assertNotIn('rebuilding', out) + self.assertFalse(os.path.exists(path) and open(path, 'rb').read() == zlib.compress(b'blob 5\0evil\n')) + + def test_a_fetch_that_fails_on_the_network_runs_no_check(self): # only a cause in the cache is checked + self.q.remote = 'https://localhost:1/o/r.git' # nothing listens there: "unable to access" + self.run_again() + self.q.remote = 'https://localhost:1/o/r.git' + same, out = self.same_change() + self.assertIsNone(same, out) + self.assertEqual(self.fscks, []) + self.assertIn('could not fetch', self.q.unknown) + + class Conflict(unittest.TestCase): """An armed PR that conflicts with its base (merge state DIRTY) cannot merge, the queue cannot update it, and GitHub leaves auto-merge on without a word. It is told once per head and passed over; before, only the run log @@ -2841,6 +3296,29 @@ def run(argv, **kw): for argv, e in local: self.assertEqual({k: e[k] for k in Q.GIT_RUN if k in e}, {}, argv) + def test_the_fetches_run_gc_in_the_foreground(self): # state lens 3: no detached gc races the git fsck + w = World() + self.addCleanup(w.close) + m0 = w.commit({'x': '0\n'}) + gone = w.commit({'x': 'gone\n'}, [m0]) + w.set('refs/heads/main', m0) + w.set('refs/pull/7/head', m0) + q = make_queue(w.dir, remote=w.remote) + seen, real = [], subprocess.run + + def run(argv, **kw): + seen.append(argv) + return real(argv, **kw) + with unittest.mock.patch.object(Q.subprocess, 'run', run), contextlib.redirect_stdout(io.StringIO()) as out: + self.assertIs(q.fetch('main', 7, m0, gone), True, out.getvalue()) + fetches = [a for a in seen if 'fetch' in a] + self.assertEqual(len(fetches), 2, seen) + for argv in fetches: + before = argv[:argv.index('fetch')] + for flag in ('gc.autoDetach=false', 'maintenance.autoDetach=false'): + self.assertIn(flag, before) + self.assertEqual(before[before.index(flag) - 1], '-c') + def test_git_gets_the_login_from_gh_only_and_never_prompts(self): q, argv, env = self.git_call('run-token', 'pat-token') self.assertEqual(argv[:15], ['git', '-c', 'core.hooksPath=/dev/null', '-c', 'core.fsmonitor=false', @@ -3520,7 +3998,7 @@ def test_the_queue_drops_one_list_of_git_variables(self): # #435 item 6: every hostile = {k: 'x' for k in Q.GIT_DROP} hostile.update(GIT_CONFIG_KEY_3='core.bare', GIT_CONFIG_VALUE_3='true', GIT_TRACE='0', PATH='/bin') env = Q.git_env(hostile) - self.assertEqual(env, {'GIT_TRACE': '0', 'PATH': '/bin', **FORCED}) + self.assertEqual(env, {'PATH': '/bin', **FORCED}) # #457 item 4: a local call writes no trace either for name in ('GIT_DIR', 'GIT_WORK_TREE', 'GIT_COMMON_DIR', 'GIT_OBJECT_DIRECTORY', 'GIT_ALTERNATE_OBJECT_DIRECTORIES', 'GIT_INDEX_FILE', 'GIT_NAMESPACE', 'GIT_CONFIG', 'GIT_CONFIG_PARAMETERS', 'GIT_CONFIG_COUNT', 'GIT_REPLACE_REF_BASE', 'GIT_GRAFT_FILE', @@ -3556,6 +4034,7 @@ def test_programs_and_transports_of_the_job_reach_only_a_fetch_that_may_need_the # https with the queue's own login (or none): no ssh, no git:// proxy, no prompt, git's default protocols; # git's own helpers (remote-https, index-pack) are found through GIT_EXEC_PATH self.assertEqual(Q.git_env(job, 'https://github.com/o/r.git'), {**plain, 'GIT_EXEC_PATH': 'x'}) + self.assertEqual(Q.git_env(job, 'Https://github.com/o/r.git'), {**plain, 'GIT_EXEC_PATH': 'x'}) for remote in ('origin', '/srv/r.git', 'file:///srv/r.git', 'ssh://git@host/r.git', 'git@host:r.git', 'http://host/r.git'): # #450 item 6: plain http is no GitHub fetch either with self.subTest(remote): # the caller's remote: it may need them, so they stay @@ -3566,11 +4045,25 @@ def test_a_fetch_from_github_writes_no_trace(self): # #450 item 5: the gate pu 'GIT_TRACE2', 'GIT_TRACE2_EVENT', 'GIT_CURL_VERBOSE')} trace['GIT_TRACE_REDACT'] = '0' job = {**trace, 'PATH': '/bin'} - self.assertEqual(Q.git_env(job, 'https://github.com/o/r.git'), {'PATH': '/bin', **FORCED}) - for remote in (None, 'origin', 'ssh://git@host/r.git'): # no token of the queue's goes there + for remote in (None, 'https://github.com/o/r.git', 'HTTPS://GitHub.com/o/r.git'): # #457 item 4: local calls + with self.subTest(remote): # (the gate's merge-tree, the queue's), and https in any case + self.assertEqual(Q.git_env(job, remote), {'PATH': '/bin', **FORCED}) + for remote in ('origin', 'ssh://git@host/r.git'): # no token of the queue's goes there with self.subTest(remote): self.assertLessEqual(trace.items(), Q.git_env(job, remote).items()) + def test_only_a_url_or_a_path_is_a_remote_a_token_goes_to(self): # #457 item 4 + for remote in ('https://github.com/o/r.git', 'HTTPS://github.com/o/r.git', 'origin', '/srv/r.git', + 'file:///srv/r.git', 'ssh://git@host/r.git', 'git@host:r.git', 'C:/srv/r.git'): + with self.subTest(remote): + self.assertIsNone(Q.remote_problem(remote)) + for remote, said in (('', 'empty'), (None, 'empty'), ('http://github.com/o/r.git', 'plain http'), + ('HTTP://github.com/o/r.git', 'plain http'), ('ext::sh -c id', 'remote helper'), + ('fd::3', 'remote helper'), ('https::github.com/o/r.git', 'remote helper'), + ('--upload-pack=touch /tmp/x', 'starts with -'), ('-oProxy', 'starts with -')): + with self.subTest(remote): + self.assertIn(said, Q.remote_problem(remote)) + def test_git_speaks_one_language(self): # #450 item 4: messages are matched and decoded as UTF-8 env = Q.git_env({'LC_ALL': 'de_DE.ISO-8859-1', 'LANG': 'de_DE', 'LANGUAGE': 'de', 'LC_MESSAGES': 'de_DE'}) self.assertEqual(env['LC_ALL'], 'C') # LANGUAGE is ignored under LC_ALL=C (gettext)