Run 1
Attempts
10
8 graded
Mean score
87.5
over graded attempts
Failures
2
errors + degenerate
Tokens out
669
125 model calls
Wall time
495s
sequential
//
qwen2.5-coder-3b
oneshot 100.0agentic 80.0
FizzBuzz
agentic✓ ok
100
score
tests3/3
deliverables100%
content checks100%
24
model calls
0
tokens out
12
SLOC
6
complexity
90s
wall
Palindrome check
agentic✓ ok
100
score
tests5/5
deliverables100%
content checks100%
24
model calls
0
tokens out
3
SLOC
3
complexity
41s
wall
Temperature conversion
agentic✓ ok
100
score
tests4/4
deliverables100%
content checks100%
24
model calls
0
tokens out
4
SLOC
1
complexity
51s
wall
Text statistics module
agentic✓ ok
0
score
tests0/1
deliverables0%
content checks0%
24
model calls
0
tokens out
0
SLOC
0
complexity
251s
wall
Vowel count
agentic✓ ok
100
score
tests5/5
deliverables100%
content checks100%
24
model calls
0
tokens out
2
SLOC
3
complexity
41s
wall
FizzBuzz
oneshot✓ ok
100
score
tests3/3
deliverables100%
content checks100%
1
model calls
138
tokens out
12
SLOC
6
complexity
6s
wall
Palindrome check
oneshot✗ error
—
score
tests—
deliverables—
content checks—
1
model calls
89
tokens out
0
SLOC
0
complexity
3s
wall
Temperature conversion
oneshot✓ ok
100
score
tests4/4
deliverables100%
content checks100%
1
model calls
79
tokens out
4
SLOC
1
complexity
2s
wall
Text statistics module
oneshot✗ error
—
score
tests—
deliverables—
content checks—
1
model calls
303
tokens out
0
SLOC
0
complexity
8s
wall
Vowel count
oneshot✓ ok
100
score
tests5/5
deliverables100%
content checks100%
1
model calls
60
tokens out
3
SLOC
3
complexity
2s
wall