we're developing https://copperbench.org - an open-source benchmark for evaluating ai agents on real, verifiable hardware design tasks.

any idea how it stacks up against zoo / zookeeper?

https://zoo.dev/research/zookeeper

mmm, someone decided to reuse `zookeeper` name, seriously?

I do not have experience with Copperhead, but I made 2 PCBs with Claude. He made a py script that generates Kicad sources, I ended up generating all files (in particular schematics and PCB) from a shared netlist, and use Kicad cli to validate and generate gerbers.

I manufactured both PCBs via PCBWay (Claude also red all the specs and configured the constraints accordingly) and both work. I'm pretty impressed, especially because I've never done a PCB myself and didn't know I could.

What I was doing manually, co-working with Claude, was placing and routing (I was doing those via Kicad, then saving, then Claude was re-importing my work in his py files).

I recommend having the LLM write Skidl, which generates a netlist, which you import in the PCB editor, which the LLM then reviews. Routing itself I think would require code-driven routing (expressing board via layout constraints, kinda like UI constraints/CSS) which does not exist yet.

Copperhead directly edits the KiCad files so that is the CLI version of that Astra demo.

not quite. copperhead operates on its own IR, which then compiles to KiCad files.

This is what I want to know as well. I'm about to kick off a project or two.