Run 2

started 2026-07-17 15:31:49 UTCgit c4dd881engine 0.1.0challenge set 96e86e1711a49f7drun 2: 4 local models, 9 challenges incl. 4 harder

← leaderboard · raw JSON

Attempts

56 graded

Mean score

76.9

over graded attempts

Failures

errors + degenerate

Tokens out

144.7k

575 model calls

Wall time

14605s

sequential

glm4-9b llama3.2-3b qwen2.5-coder-3b qwen3-30b-a3b

// glm4-9b oneshot 87.0agentic 49.4

Expression evaluator (no eval) agentic✓ ok

48.6

score

tests1/7

deliverables100%

content checks100%

model calls

13.1k

tokens out

SLOC

complexity

1289s

wall

full engineer report →

FizzBuzz agentic✓ ok

score

tests3/5

deliverables100%

content checks100%

model calls

2.6k

tokens out

SLOC

complexity

230s

wall

full engineer report →

JSON Pointer (RFC 6901) agentic✓ ok

score

tests0/1

deliverables0%

content checks0%

model calls

3.1k

tokens out

SLOC

complexity

323s

wall

full engineer report →

LRU cache agentic✓ ok

score

tests0/1

deliverables100%

content checks100%

model calls

3.8k

tokens out

SLOC

complexity

372s

wall

full engineer report →

Palindrome check agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

355

tokens out

SLOC

complexity

35s

wall

full engineer report →

Temperature conversion agentic✓ ok

score

tests0/1

deliverables100%

content checks100%

model calls

4.5k

tokens out

SLOC

complexity

368s

wall

full engineer report →

Text statistics module agentic✓ ok

score

tests0/1

deliverables0%

content checks0%

model calls

7.0k

tokens out

SLOC

complexity

535s

wall

full engineer report →

Vowel count agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

405

tokens out

SLOC

complexity

30s

wall

full engineer report →

Word ladder (shortest path) agentic✓ ok

score

tests0/5

deliverables100%

content checks100%

model calls

3.4k

tokens out

SLOC

complexity

280s

wall

full engineer report →

Expression evaluator (no eval) oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

252

tokens out

SLOC

complexity

30s

wall

full engineer report →

FizzBuzz oneshot✓ ok

100

score

tests3/3

deliverables100%

content checks100%

model calls

129

tokens out

SLOC

complexity

12s

wall

full engineer report →

JSON Pointer (RFC 6901) oneshot✓ ok

score

tests7/10

deliverables100%

content checks100%

model calls

225

tokens out

SLOC

complexity

23s

wall

full engineer report →

LRU cache oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

153

tokens out

SLOC

complexity

14s

wall

full engineer report →

Palindrome check oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Temperature conversion oneshot✓ ok

100

score

tests4/4

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Text statistics module oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

242

tokens out

SLOC

complexity

17s

wall

full engineer report →

Vowel count oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Word ladder (shortest path) oneshot✓ ok

score

tests0/5

deliverables100%

content checks100%

model calls

185

tokens out

SLOC

complexity

14s

wall

full engineer report →

// llama3.2-3b oneshot 91.4agentic 77.9

Expression evaluator (no eval) agentic✓ ok

91.4

score

tests6/7

deliverables100%

content checks100%

model calls

5.9k

tokens out

SLOC

complexity

295s

wall

full engineer report →

FizzBuzz agentic✓ ok

100

score

tests3/3

deliverables100%

content checks100%

model calls

119

tokens out

SLOC

complexity

wall

full engineer report →

JSON Pointer (RFC 6901) agentic✓ ok

score

tests7/10

deliverables100%

content checks100%

model calls

5.3k

tokens out

SLOC

complexity

305s

wall

full engineer report →

LRU cache agentic✓ ok

score

tests3/5

deliverables100%

content checks100%

model calls

2.0k

tokens out

SLOC

complexity

111s

wall

full engineer report →

Palindrome check agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Temperature conversion agentic✓ ok

100

score

tests4/4

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Text statistics module agentic✓ ok

score

tests0/1

deliverables0%

content checks0%

model calls

1.2k

tokens out

SLOC

complexity

67s

wall

full engineer report →

Vowel count agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Word ladder (shortest path) agentic✓ ok

score

tests1/5

deliverables100%

content checks100%

model calls

1.6k

tokens out

SLOC

complexity

119s

wall

full engineer report →

Expression evaluator (no eval) oneshot✓ ok

score

tests0/1

deliverables100%

content checks100%

model calls

254

tokens out

SLOC

complexity

12s

wall

full engineer report →

FizzBuzz oneshot✓ ok

100

score

tests3/3

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

JSON Pointer (RFC 6901) oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

171

tokens out

SLOC

complexity

wall

full engineer report →

LRU cache oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

157

tokens out

SLOC

complexity

wall

full engineer report →

Palindrome check oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Temperature conversion oneshot✓ ok

100

score

tests4/4

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Text statistics module oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

155

tokens out

SLOC

complexity

wall

full engineer report →

Vowel count oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Word ladder (shortest path) oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

196

tokens out

SLOC

complexity

wall

full engineer report →

// qwen2.5-coder-3b oneshot 87.2agentic 49.8

Expression evaluator (no eval) agentic✓ ok

48.6

score

tests1/7

deliverables100%

content checks100%

model calls

9.4k

tokens out

SLOC

complexity

341s

wall

full engineer report →

FizzBuzz agentic✓ ok

100

score

tests3/3

deliverables100%

content checks100%

model calls

3.3k

tokens out

SLOC

complexity

129s

wall

full engineer report →

JSON Pointer (RFC 6901) agentic✓ ok

score

tests0/1

deliverables0%

content checks0%

model calls

3.5k

tokens out

SLOC

complexity

139s

wall

full engineer report →

LRU cache agentic✓ ok

score

tests0/1

deliverables0%

content checks0%

model calls

5.5k

tokens out

SLOC

complexity

239s

wall

full engineer report →

Palindrome check agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

1.5k

tokens out

SLOC

complexity

63s

wall

full engineer report →

Temperature conversion agentic✓ ok

100

score

tests4/4

deliverables100%

content checks100%

model calls

1.9k

tokens out

SLOC

complexity

77s

wall

full engineer report →

Text statistics module agentic✓ ok

score

tests0/1

deliverables0%

content checks0%

model calls

8.1k

tokens out

SLOC

complexity

384s

wall

full engineer report →

Vowel count agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

1.4k

tokens out

SLOC

complexity

63s

wall

full engineer report →

Word ladder (shortest path) agentic✓ ok

score

tests0/1

deliverables0%

content checks0%

model calls

5.1k

tokens out

SLOC

complexity

228s

wall

full engineer report →

Expression evaluator (no eval) oneshot✓ ok

48.6

score

tests1/7

deliverables100%

content checks100%

model calls

500

tokens out

SLOC

complexity

16s

wall

full engineer report →

FizzBuzz oneshot✓ ok

100

score

tests3/3

deliverables100%

content checks100%

model calls

138

tokens out

SLOC

complexity

wall

full engineer report →

JSON Pointer (RFC 6901) oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

171

tokens out

SLOC

complexity

wall

full engineer report →

LRU cache oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

148

tokens out

SLOC

complexity

wall

full engineer report →

Palindrome check oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Temperature conversion oneshot✓ ok

100

score

tests4/4

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Text statistics module oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

303

tokens out

SLOC

complexity

14s

wall

full engineer report →

Vowel count oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

tokens out

SLOC

complexity

wall

full engineer report →

Word ladder (shortest path) oneshot⚠ degenerate

—

score

tests—

deliverables—

content checks—

model calls

250

tokens out

SLOC

complexity

12s

wall

full engineer report →

// qwen3-30b-a3b oneshot 100.0agentic 100.0

Expression evaluator (no eval) agentic✗ error

—

score

tests—

deliverables—

content checks—

model calls

tokens out

SLOC

complexity

906s

wall

full engineer report →

FizzBuzz agentic✓ ok

100

score

tests3/3

deliverables100%

content checks100%

model calls

4.5k

tokens out

SLOC

complexity

215s

wall

full engineer report →

JSON Pointer (RFC 6901) agentic✗ error

—

score

tests—

deliverables—

content checks—

model calls

10.3k

tokens out

SLOC

complexity

1400s

wall

full engineer report →

LRU cache agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

3.5k

tokens out

SLOC

complexity

336s

wall

full engineer report →

Palindrome check agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

5.5k

tokens out

SLOC

complexity

380s

wall

full engineer report →

Temperature conversion agentic✓ ok

100

score

tests4/4

deliverables100%

content checks100%

model calls

2.1k

tokens out

SLOC

complexity

132s

wall

full engineer report →

Text statistics module agentic✗ error

—

score

tests—

deliverables—

content checks—

model calls

tokens out

SLOC

complexity

906s

wall

full engineer report →

Vowel count agentic✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

2.0k

tokens out

SLOC

complexity

107s

wall

full engineer report →

Word ladder (shortest path) agentic✗ error

—

score

tests—

deliverables—

content checks—

model calls

tokens out

SLOC

complexity

906s

wall

full engineer report →

Expression evaluator (no eval) oneshot✗ error

—

score

tests—

deliverables—

content checks—

model calls

tokens out

SLOC

complexity

906s

wall

full engineer report →

FizzBuzz oneshot✓ ok

100

score

tests3/3

deliverables100%

content checks100%

model calls

1.8k

tokens out

SLOC

complexity

89s

wall

full engineer report →

JSON Pointer (RFC 6901) oneshot✓ ok

100

score

tests10/10

deliverables100%

content checks100%

model calls

5.4k

tokens out

SLOC

complexity

284s

wall

full engineer report →

LRU cache oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

1.8k

tokens out

SLOC

complexity

238s

wall

full engineer report →

Palindrome check oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

2.8k

tokens out

SLOC

complexity

291s

wall

full engineer report →

Temperature conversion oneshot✓ ok

100

score

tests4/4

deliverables100%

content checks100%

model calls

838

tokens out

SLOC

complexity

52s

wall

full engineer report →

Text statistics module oneshot✗ error

—

score

tests—

deliverables—

content checks—

model calls

tokens out

SLOC

complexity

906s

wall

full engineer report →

Vowel count oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

970

tokens out

SLOC

complexity

60s

wall

full engineer report →

Word ladder (shortest path) oneshot✓ ok

100

score

tests5/5

deliverables100%

content checks100%

model calls

4.4k

tokens out

SLOC

complexity

196s

wall

full engineer report →

// full table

Model	Mode	Challenge	Score	Tests	Wall
glm4-9b	agentic	calculator	48.6	1/7	1289s	report
glm4-9b	agentic	fizzbuzz	76.0	3/5	230s	report
glm4-9b	agentic	json-pointer	0.0	0/1	323s	report
glm4-9b	agentic	lru-cache	40.0	0/1	372s	report
glm4-9b	agentic	palindrome	100.0	5/5	35s	report
glm4-9b	agentic	temperature	40.0	0/1	368s	report
glm4-9b	agentic	text-stats	0.0	0/1	535s	report
glm4-9b	agentic	vowel-count	100.0	5/5	30s	report
glm4-9b	agentic	word-ladder	40.0	0/5	280s	report
glm4-9b	oneshot	calculator	degenerate		30s	report
glm4-9b	oneshot	fizzbuzz	100.0	3/3	12s	report
glm4-9b	oneshot	json-pointer	82.0	7/10	23s	report
glm4-9b	oneshot	lru-cache	degenerate		14s	report
glm4-9b	oneshot	palindrome	100.0	5/5	6s	report
glm4-9b	oneshot	temperature	100.0	4/4	7s	report
glm4-9b	oneshot	text-stats	degenerate		17s	report
glm4-9b	oneshot	vowel-count	100.0	5/5	4s	report
glm4-9b	oneshot	word-ladder	40.0	0/5	14s	report
llama3.2-3b	agentic	calculator	91.4	6/7	295s	report
llama3.2-3b	agentic	fizzbuzz	100.0	3/3	8s	report
llama3.2-3b	agentic	json-pointer	82.0	7/10	305s	report
llama3.2-3b	agentic	lru-cache	76.0	3/5	111s	report
llama3.2-3b	agentic	palindrome	100.0	5/5	5s	report
llama3.2-3b	agentic	temperature	100.0	4/4	5s	report
llama3.2-3b	agentic	text-stats	0.0	0/1	67s	report
llama3.2-3b	agentic	vowel-count	100.0	5/5	4s	report
llama3.2-3b	agentic	word-ladder	52.0	1/5	119s	report
llama3.2-3b	oneshot	calculator	40.0	0/1	12s	report
llama3.2-3b	oneshot	fizzbuzz	100.0	3/3	4s	report
llama3.2-3b	oneshot	json-pointer	degenerate		7s	report
llama3.2-3b	oneshot	lru-cache	100.0	5/5	6s	report
llama3.2-3b	oneshot	palindrome	100.0	5/5	3s	report
llama3.2-3b	oneshot	temperature	100.0	4/4	3s	report
llama3.2-3b	oneshot	text-stats	degenerate		6s	report
llama3.2-3b	oneshot	vowel-count	100.0	5/5	2s	report
llama3.2-3b	oneshot	word-ladder	100.0	5/5	9s	report
qwen2.5-coder-3b	agentic	calculator	48.6	1/7	341s	report
qwen2.5-coder-3b	agentic	fizzbuzz	100.0	3/3	129s	report
qwen2.5-coder-3b	agentic	json-pointer	0.0	0/1	139s	report
qwen2.5-coder-3b	agentic	lru-cache	0.0	0/1	239s	report
qwen2.5-coder-3b	agentic	palindrome	100.0	5/5	63s	report
qwen2.5-coder-3b	agentic	temperature	100.0	4/4	77s	report
qwen2.5-coder-3b	agentic	text-stats	0.0	0/1	384s	report
qwen2.5-coder-3b	agentic	vowel-count	100.0	5/5	63s	report
qwen2.5-coder-3b	agentic	word-ladder	0.0	0/1	228s	report
qwen2.5-coder-3b	oneshot	calculator	48.6	1/7	16s	report
qwen2.5-coder-3b	oneshot	fizzbuzz	100.0	3/3	6s	report
qwen2.5-coder-3b	oneshot	json-pointer	degenerate		6s	report
qwen2.5-coder-3b	oneshot	lru-cache	degenerate		6s	report
qwen2.5-coder-3b	oneshot	palindrome	degenerate		3s	report
qwen2.5-coder-3b	oneshot	temperature	100.0	4/4	5s	report
qwen2.5-coder-3b	oneshot	text-stats	degenerate		14s	report
qwen2.5-coder-3b	oneshot	vowel-count	100.0	5/5	3s	report
qwen2.5-coder-3b	oneshot	word-ladder	degenerate		12s	report
qwen3-30b-a3b	agentic	calculator	error		906s	report
qwen3-30b-a3b	agentic	fizzbuzz	100.0	3/3	215s	report
qwen3-30b-a3b	agentic	json-pointer	error		1400s	report
qwen3-30b-a3b	agentic	lru-cache	100.0	5/5	336s	report
qwen3-30b-a3b	agentic	palindrome	100.0	5/5	380s	report
qwen3-30b-a3b	agentic	temperature	100.0	4/4	132s	report
qwen3-30b-a3b	agentic	text-stats	error		906s	report
qwen3-30b-a3b	agentic	vowel-count	100.0	5/5	107s	report
qwen3-30b-a3b	agentic	word-ladder	error		906s	report
qwen3-30b-a3b	oneshot	calculator	error		906s	report
qwen3-30b-a3b	oneshot	fizzbuzz	100.0	3/3	89s	report
qwen3-30b-a3b	oneshot	json-pointer	100.0	10/10	284s	report
qwen3-30b-a3b	oneshot	lru-cache	100.0	5/5	238s	report
qwen3-30b-a3b	oneshot	palindrome	100.0	5/5	291s	report
qwen3-30b-a3b	oneshot	temperature	100.0	4/4	52s	report
qwen3-30b-a3b	oneshot	text-stats	error		906s	report
qwen3-30b-a3b	oneshot	vowel-count	100.0	5/5	60s	report
qwen3-30b-a3b	oneshot	word-ladder	100.0	5/5	196s	report