Repository navigation
Commit 3363036
committed
chore(ci): fail the on-demand review when it posts nothing
MEASURED on PR #182 (run 32933651692): the review action exited `success` with
`is_error: false` having done no work at all — `num_turns: 0`,
`permission_denials_count: 4`, `total_cost_usd: 1.076`. The tracking comment
kept its placeholder ("I'll analyze this and get back to you"), the run logged
`No buffered inline comments`, and the PR received zero inline comments. The job
went green.
That is the worst failure this workflow can have. A silent no-op is
indistinguishable from a clean review, so a pull request reads as reviewed when
nothing read it, and the cost is paid either way.
`is_error` cannot gate this; it was false in that very run. The load-bearing
signal is in the action's execution file: `num_turns` counts the model's
completed turns, and a review that never took a turn cannot have posted
anything. `permission_denials_count` is failed on separately, because a denial
is how a review dies quietly — the tool it needs is not on the allowlist, it
has nowhere to say so, and it stops. A denial with turns is also failed: a
review that lost a tool is a review that could not see everything.
The step names the denied tools when the record carries them. Tool names are
structured data the runner produced, so printing them is safe here; the tool
INPUT is never printed, because it can quote the untrusted diff. That is what
makes the next occurrence self-diagnosing rather than a log excavation.
`show_full_output: true` would answer the same question and is deliberately not
used: this is a public repo and it dumps text the model produced while ingesting
an untrusted diff into a world-readable log. Counters are not attacker
controlled; model prose is.
Both execution-file shapes the action has written (a JSON array of messages, and
JSONL) are accepted, rather than pinning a shape it may change. Verified against
six fixtures including the real record from run 32933651692: the failing shape,
a healthy run, denials-with-turns, a file with no result record, a missing file,
and an unset variable.1 parent a2939c8 commit 3363036
1 file changed
Lines changed: 148 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
| 214 | + | |
214 | 215 | | |
215 | 216 | | |
216 | 217 | | |
| |||
247 | 248 | | |
248 | 249 | | |
249 | 250 | | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
0 commit comments