[GH-ISSUE #577] Explore converting existing llama.cpp GBNF 'structured output' config to use the SOTA llguidance format (which now also powers OpenAI API JSON Schema, custom tools/etc) #91

New issue

Open

opened 2026-03-03 13:52:54 +03:00 by kerem · 2 comments

kerem commented

2026-03-03 13:52:54 +03:00

Owner

Originally created by @0xdevalias on GitHub (Aug 25, 2025).
Original GitHub issue: https://github.com/jehna/humanify/issues/577

Interesting anecdote, llama.cpp has support for the rust-based version of guidance built in now, and is also what is used by the OpenAI API for their JSON Schemas now too:

https://github.com/guidance-ai/llguidance

Low-level Guidance (llguidance)

2025-05-20 LLGuidance shipped in OpenAI for JSON Schema

2025-02-01 integration merged into llama.cpp (b4613)

https://github.com/ggml-org/llama.cpp/pull/10224

https://github.com/ggml-org/llama.cpp/blob/master/docs/llguidance.md

LLGuidance supports JSON Schemas and arbitrary context-free grammars (CFGs) written in a variant of Lark syntax. It is very fast and has excellent JSON Schema coverage but requires the Rust compiler, which complicates the llama.cpp build process.

There are no new command-line arguments or modifications to common_params. When enabled, grammars starting with %llguidance are passed to LLGuidance instead of the current llama.cpp grammars. Additionally, JSON Schema requests (e.g., using the -j argument in llama-cli) are also passed to LLGuidance.

For your existing GBNF grammars, you can use gbnf_to_lark.py script to convert them to LLGuidance Lark-like format.

https://github.com/guidance-ai/llguidance/blob/main/docs/syntax.md

LLGuidance supports a variant of syntax used by Python Lark parsing toolkit. We also provide a gbnf_to_lark.py script to convert from GBNF format used in llama.cpp. These makes it easier to get started with a new grammar, and provide a familiar syntax, however neither is a drop-in replacement for Lark or GBNF.

Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/6#issuecomment-3218524933

See also:

github.com/jehna/humanify@7beba2d324/src/plugins/local-llm-rename/gbnf.ts (L1-L73)
github.com/jehna/humanify@7beba2d324/src/plugins/openai/openai-rename.ts (L62-L79)
https://platform.openai.com/docs/guides/function-calling#custom-tools
- https://platform.openai.com/docs/guides/function-calling#context-free-grammars
  - Context-free grammars
    
    A context-free grammar (CFG) is a set of rules that define how to produce valid text in a given format. For custom tools, you can provide a CFG that will constrain the model's text input for a custom tool.
    
    You can provide a custom CFG using the grammar parameter when configuring a custom tool. Currently, we support two CFG syntaxes when defining grammars: lark and regex.
  - Grammars are specified using a variation of Lark. Model sampling is constrained using LLGuidance.
  - We recommend using the Lark IDE to experiment with custom grammars.
https://guidance-ai.github.io/llguidance/llg-go-brrr
- LLGuidance: Making Structured Outputs Go Brrr
https://github.com/guidance-ai/guidance-ts

Originally created by @0xdevalias on GitHub (Aug 25, 2025). Original GitHub issue: https://github.com/jehna/humanify/issues/577 > Interesting anecdote, llama.cpp has support for the rust-based version of guidance built in now, and is also what is used by the OpenAI API for their JSON Schemas now too: > > - https://github.com/guidance-ai/llguidance > - > Low-level Guidance (llguidance) > - > 2025-05-20 LLGuidance [shipped](https://x.com/OpenAIDevs/status/1924915341052019166) in [OpenAI](https://x.com/OpenAIDevs/status/1924915343677653014) for JSON Schema > - > 2025-02-01 integration [merged](https://github.com/ggml-org/llama.cpp/pull/10224) into llama.cpp (b4613) > - https://github.com/ggml-org/llama.cpp/pull/10224 > - https://github.com/ggml-org/llama.cpp/blob/master/docs/llguidance.md > - > LLGuidance supports JSON Schemas and arbitrary context-free grammars (CFGs) written in a [variant](https://github.com/guidance-ai/llguidance/blob/main/docs/syntax.md) of Lark syntax. It is [very fast](https://github.com/guidance-ai/jsonschemabench/tree/main/maskbench) and has [excellent](https://github.com/guidance-ai/llguidance/blob/main/docs/json_schema.md) JSON Schema coverage but requires the Rust compiler, which complicates the llama.cpp build process. > - > There are no new command-line arguments or modifications to `common_params`. When enabled, grammars starting with `%llguidance` are passed to LLGuidance instead of the [current](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) llama.cpp grammars. Additionally, JSON Schema requests (e.g., using the `-j` argument in `llama-cli`) are also passed to LLGuidance. > - > For your existing GBNF grammars, you can use [gbnf_to_lark.py script](https://github.com/guidance-ai/llguidance/blob/main/python/llguidance/gbnf_to_lark.py) to convert them to LLGuidance Lark-like format. > - https://github.com/guidance-ai/llguidance/blob/main/docs/syntax.md > - > LLGuidance supports a variant of syntax used by Python [Lark parsing toolkit](https://github.com/lark-parser/lark). We also provide a [gbnf_to_lark.py script](https://github.com/guidance-ai/llguidance/blob/main/python/llguidance/gbnf_to_lark.py) to convert from [GBNF](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md) format used in [llama.cpp](https://github.com/ggerganov/llama.cpp). These makes it easier to get started with a new grammar, and provide a familiar syntax, however neither is a drop-in replacement for Lark or GBNF. > > _Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/6#issuecomment-3218524933_ See also: - https://github.com/jehna/humanify/blob/7beba2d32433e58bb77d0e1b0eda01c470fec3e2/src/plugins/local-llm-rename/gbnf.ts#L1-L73 - https://github.com/jehna/humanify/blob/7beba2d32433e58bb77d0e1b0eda01c470fec3e2/src/plugins/openai/openai-rename.ts#L62-L79 - https://platform.openai.com/docs/guides/function-calling#custom-tools - https://platform.openai.com/docs/guides/function-calling#context-free-grammars - > Context-free grammars > > A [context-free grammar](https://en.wikipedia.org/wiki/Context-free_grammar) (CFG) is a set of rules that define how to produce valid text in a given format. For custom tools, you can provide a CFG that will constrain the model's text input for a custom tool. > > You can provide a custom CFG using the `grammar` parameter when configuring a custom tool. Currently, we support two CFG syntaxes when defining grammars: `lark` and `regex`. - > Grammars are specified using a variation of [Lark](https://lark-parser.readthedocs.io/en/stable/index.html). Model sampling is constrained using [LLGuidance](https://github.com/guidance-ai/llguidance/blob/main/docs/syntax.md). - > We recommend using the [Lark IDE](https://www.lark-parser.org/ide/) to experiment with custom grammars. - https://guidance-ai.github.io/llguidance/llg-go-brrr - > LLGuidance: Making Structured Outputs Go Brrr - https://github.com/guidance-ai/guidance-ts

kerem commented

2026-03-03 13:52:55 +03:00

Author

Owner

@jehna commented on GitHub (Jan 25, 2026):

Nice. Seems that llama.cpp needs an extra build time flag for this, need to investigate if this is on by default:

llama.cpp - available via -DLLAMA_LLGUIDANCE=ON

@jehna commented on GitHub (Jan 25, 2026): Nice. Seems that llama.cpp needs an extra build time flag for this, need to investigate if this is on by default: > [llama.cpp](https://github.com/ggerganov/llama.cpp/pull/10224) - available via `-DLLAMA_LLGUIDANCE=ON`

kerem commented

2026-03-03 13:52:55 +03:00

Author

Owner

@jehna commented on GitHub (Jan 25, 2026):

And AFAIK this is only used internally by OpenAI's JSON Schema output, no way to use it via API directly

@jehna commented on GitHub (Jan 25, 2026): And AFAIK this is only used internally by OpenAI's JSON Schema output, no way to use it via API directly