EBench-Astra-Tech-Report/src/components/sections/FailurePatterns.astro at main · securityCFS/EBench-Astra-Tech-Report · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
<section id="limits" class="section">
  <div class="section-inner">
    <div class="section-heading">
      <div>
        <h2>Failure Patterns: Where execution breaks down</h2>
      </div>
    </div>
    <div class="report-prose">
      <p>
        Under the EBench protocol, scoring is not binary; instead, an episode receives partial
        credit based on the intermediate stages successfully completed, with a full Score of 1.0
        assigned only upon total task completion. Across 510 evaluated episodes, 237 achieve
        complete success, while 188 of the 273 unsuccessful runs still register positive partial
        credit (with only 85 scoring zero). <strong>
          For episodes that gain partial credit but fall short of full success, this score breakdown points to two common failure modes: either the execution lacks the precision required to satisfy strict final termination criteria, or the agent stalls at an intermediate step and times out before completing the task.
        </strong>
      </p>
      <p>
        This behavioral difference is evident when comparing tasks with similarly low success
        rates: failed (zero-score) episodes make up 75% of gear installation, 65% of utensils placement
        and 50% of glasses packing, whereas they are completely absent in nut tightening, peg
        insertion, and shop (and account for only 30% in bottle). In these latter tasks, the agent
        advances through early stages despite missing full completion, a distinction
        that would be lost when looking at final success alone.
      </p>
    </div>
    <div id="episode-outcomes"></div>
    <div class="outcome-task-breakdown" id="failure-depth-analysis">
      <div class="report-prose">
        <p>
          The breakdown below splits each task's episodes into complete successes, incomplete runs
          and failures, with each row tagged by the precision and horizon the task demands.
        </p>
      </div>
      <div id="failure-depth-chart"></div>
    </div>
    <div class="report-prose">
      <p>
        These terminal scores motivate examining where precise contact fails and how local retries
        influence overall task progress. The recordings below connect these execution challenges
        directly to the reported task results.
      </p>
    </div>
    <h3>Qualitative analysis from the recordings</h3>
    <div class="limit-switch" aria-label="Execution challenge">
      <button data-limit="precision" aria-pressed="true">
        Precise contact
      </button>
      <button data-limit="horizon" aria-pressed="false">
        Long procedures
      </button>
    </div>
    <div id="limits-content"></div>
    <div class="insight-block" id="fixed-base-analysis">
      <h3>The gap in tabletop dexterous-and-precise tasks</h3>
      <div class="report-prose">
        <p>
          All seven tabletop tasks in EBench demand dexterous or precise manipulation, which we refer
          to as tabletop dexterous-and-precise (D&amp;P) tasks: frame placement, cup flipping and
          glasses packing through two-arm coordination, flipping and folding; the remaining four
          through tight contact. Even on the three D&amp;P tasks labeled low or medium precision,
          GPT-6-Astra averages 31.7% success, compared with Qwen-RobotManip’s 91.7%. Cup flipping
          reaches 10% versus 90%; glasses packing reaches 20% versus 90%.
        </p>
        <p>
          <strong>Precision labels alone do not explain the capability gap.</strong> The recordings
          point to two failures that the labels do not capture. Glasses packing is labeled medium
          precision, yet closing the case depends on folding the temples precisely: GPT-6-Astra
          completes the transfer into the case, but its corrective contacts leave the temples
          protruding, so the lid cannot close.
        </p>
        <p>
          The second is bimanual coordination. Without a demonstration, the agent does not compose a
          two-arm plan for frame placement and attempts single-arm variants that leave the frame
          unsupported; with one, it divides the work between the arms (see the frame ICL case) and
          reaches 65% in the main evaluation, above OpenWAM-&alpha;&rsquo;s 50%. Cup flipping is a
          purely bimanual problem that the demonstration does not resolve: GPT-6-Astra reaches 10%
          against Qwen-RobotManip&rsquo;s 90%. Across the five tasks we identify as involving a
          handover (our annotation, not an EBench label), GPT-6-Astra averages 34.0% against
          OpenWAM-&alpha;&rsquo;s 48.0%, placing fifth among the eight systems evaluated.
        </p>
        <p>
          Neither failure shows up in the success rate on its own, and neither is named by a single
          task label. The next step is therefore to take the annotations EBench does supply &mdash;
          mobility, horizon and precision &mdash; and ask how much of this deficit they account for.
        </p>
      </div>
      <div id="fixed-task-evidence"></div>
    </div>
  </div>
</section>