Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.


You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.


the grammar support/json-schema applies to a whole response, but I was under the impression a similar logit guide rail can be applied when a tool call is opened.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: