[GH-ISSUE #14] Ollama Support #10

New issue

Open

opened 2026-03-03 13:52:10 +03:00 by kerem · 16 comments

kerem commented 2026-03-03 13:52:10 +03:00

Owner

Copy link

Originally created by @0xrsydn on GitHub (Jun 18, 2024).
Original GitHub issue: https://github.com/jehna/humanify/issues/14

is it possible to use llama3 via ollama rather than huggingface one?

Originally created by @0xrsydn on GitHub (Jun 18, 2024). Original GitHub issue: https://github.com/jehna/humanify/issues/14 is it possible to use llama3 via ollama rather than huggingface one?

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@jehna commented on GitHub (Jun 19, 2024):

Not possible at the moment, but should be straightforward to implement if you'd like to give it a shot

You can check LlamaCpp docs from Guidance and change (preferably parametrize) the config:
https://github.com/jehna/humanify/blob/main/local-inference/guidance_config.py

<!-- gh-comment-id:2179048718 --> @jehna commented on GitHub (Jun 19, 2024): Not possible at the moment, but should be straightforward to implement if you'd like to give it a shot You can check `LlamaCpp` docs from Guidance and change (preferably parametrize) the config: https://github.com/jehna/humanify/blob/main/local-inference/guidance_config.py

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@0xdevalias commented on GitHub (Jun 19, 2024):

I wonder if it's worth implementing a wrapper/abstraction layer like LiteLLM to make things more flexible?

• https://github.com/BerriAI/litellm

  • Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

This is what projects like aider use:

• https://aider.chat/docs/llms/other.html

  • Aider uses the litellm package to connect to hundreds of other models. You can use aider --model <model-name> to use any supported model.

    To explore the list of supported models you can run aider --models <model-name> with a partial model name. If the supplied name is not an exact match for a known model, aider will return a list of possible matching models.

Though I'm not currently sure if/how compatible that is with the guidance module you're currently using:

• https://github.com/guidance-ai/guidance

  • A guidance language for controlling large language models.

  • https://github.com/guidance-ai/guidance#loading-models

    • Loading models

See also:

• https://github.com/guidance-ai/guidance/issues/599
• https://github.com/guidance-ai/guidance/issues/687
• https://github.com/guidance-ai/guidance/issues/648
• https://github.com/guidance-ai/guidance/issues/815

  • If you look here you can find that it can use LiteLLM, which itself can work with Ollama or some kind popular of remote LLM service

    • https://github.com/guidance-ai/guidance/blob/main/guidance/models/_lite_llm.py

<!-- gh-comment-id:2179563971 --> @0xdevalias commented on GitHub (Jun 19, 2024): I wonder if it's worth implementing a wrapper/abstraction layer like LiteLLM to make things more flexible? - https://github.com/BerriAI/litellm - > Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs) This is what projects like [`aider`](https://github.com/paul-gauthier/aider) use: - https://aider.chat/docs/llms/other.html - > Aider uses the [litellm](https://docs.litellm.ai/docs/providers) package to connect to hundreds of other models. You can use aider `--model <model-name>` to use any supported model. > > To explore the list of supported models you can run `aider --models <model-name>` with a partial model name. If the supplied name is not an exact match for a known model, aider will return a list of possible matching models. Though I'm not currently sure if/how compatible that is with the `guidance` module you're currently using: - https://github.com/guidance-ai/guidance - > A guidance language for controlling large language models. - https://github.com/guidance-ai/guidance#loading-models - > Loading models See also: - https://github.com/guidance-ai/guidance/issues/599 - https://github.com/guidance-ai/guidance/issues/687 - https://github.com/guidance-ai/guidance/issues/648 - https://github.com/guidance-ai/guidance/issues/815 - > If you look here you can find that it can use LiteLLM, which itself can work with Ollama or some kind popular of remote LLM service - https://github.com/guidance-ai/guidance/blob/main/guidance/models/_lite_llm.py

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@0xdevalias commented on GitHub (Jun 20, 2024):

@jehna Curious, what aspects of guidance does humanify currently rely on? Is it using much of the deeper 'controls' provided by it?

Skimming the following prompt files:

• https://github.com/jehna/humanify/blob/main/local-inference/define.py
• https://github.com/jehna/humanify/blob/main/local-inference/rename.py

It looks like gen, stop and stop_regex are used:

• https://github.com/guidance-ai/guidance#basic-generation
• https://github.com/guidance-ai/guidance#regular-expressions

<!-- gh-comment-id:2179602099 --> @0xdevalias commented on GitHub (Jun 20, 2024): @jehna Curious, what aspects of `guidance` does `humanify` currently rely on? Is it using much of the deeper 'controls' provided by it? Skimming the following prompt files: - https://github.com/jehna/humanify/blob/main/local-inference/define.py - https://github.com/jehna/humanify/blob/main/local-inference/rename.py It looks like `gen`, `stop` and `stop_regex` are used: - https://github.com/guidance-ai/guidance#basic-generation - https://github.com/guidance-ai/guidance#regular-expressions

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@jehna commented on GitHub (Aug 12, 2024):

There's now v2 that runs on top of llama.cpp, so adding llama3 support should be even more straightforward.

@0xrsydn which version of llama3 were you planning to run? I could add it in to the new version

<!-- gh-comment-id:2284789609 --> @jehna commented on GitHub (Aug 12, 2024): There's now v2 that runs on top of llama.cpp, so adding llama3 support should be even more straightforward. @0xrsydn which version of llama3 were you planning to run? I could add it in to the new version

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@0xrsydn commented on GitHub (Aug 13, 2024):

There's now v2 that runs on top of llama.cpp, so adding llama3 support should be even more straightforward.

@0xrsydn which version of llama3 were you planning to run? I could add it in to the new version

i think the recent one (llama3.1 8b) is great. Thanks btw

<!-- gh-comment-id:2285199298 --> @0xrsydn commented on GitHub (Aug 13, 2024): > There's now v2 that runs on top of llama.cpp, so adding llama3 support should be even more straightforward. > > @0xrsydn which version of llama3 were you planning to run? I could add it in to the new version i think the recent one (llama3.1 8b) is great. Thanks btw

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@jehna commented on GitHub (Aug 14, 2024):

I researched a bit about Ollama. If I'm correct, you could run Ollama locally and Humanify could connect to its API to use any model that Ollama uses.

There seems to be an undocumented feature that allows passing GBNF grammars as an argument to the model:
https://github.com/ollama/ollama/issues/3616#issuecomment-2068195083

...but judging from other open issues about the topic I'm not really sure if it works or not. But I'll give it a try!

<!-- gh-comment-id:2288895896 --> @jehna commented on GitHub (Aug 14, 2024): I researched a bit about Ollama. If I'm correct, you could run Ollama locally and Humanify could connect to its API to use any model that Ollama uses. There seems to be an undocumented feature that allows passing GBNF grammars as an argument to the model: https://github.com/ollama/ollama/issues/3616#issuecomment-2068195083 ...but judging from other open issues about the topic I'm not really sure if it works or not. But I'll give it a try!

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@0xdevalias commented on GitHub (Aug 15, 2024):

...but judging from other open issues about the topic I'm not really sure if it works or not

This seems like it's a good overarching/summarising issue; still doesn't provide full clarity yet, but links to seemingly all the related issues, and points out that now that OpenAI supports it, it's sort of become a higher priority:

• https://github.com/ollama/ollama/issues/6237

---

Based on my read of these:

There are a few open PRs for this behaviour - the most recent one being https://github.com/ollama/ollama/pull/3618 it would be amazing to get this merged in. It's a 2 line change that exposes the llama.cpp GBNF functionality via modelfile parameters. Its not my patch but I've compiled it and used it locally and it works really well.

Originally posted by @ravenscroftj in https://github.com/ollama/ollama/issues/3616#issuecomment-2053939297

Yes you can send the grammar as an option when you submit a request with the patch I linked to above enabled. It just isn't documented!

Originally posted by @ravenscroftj in https://github.com/ollama/ollama/issues/3616#issuecomment-2068195083

It sounds like it's not currently possible to use the GBNF functionality on the current main/released version of Ollama.

According to this:

My simple personal example is this. As a newer Ollama user I actually would like to try out both approaches to see which one works better for me and my product. Right now in Ollama I simply cannot, and from appearances (which can be deceiving) it appears that what's stopping me from testing these both in Ollama is a simple code change to expose the feature in llama.cpp to me. (edit: It was brought to my attention that Ollama actually uses GBNF internally to enforce json syntax, so the only thing that's really missing is exposing this feature to the end user to customize or use different grammar.)

Originally posted by @Kinglord in https://github.com/ollama/ollama/issues/6237#issue-2453913814

It sounds like ollama currently supports JSON mode, and that is built as a GBNF grammar (presumably on top of llama.cpp's support of it), but that the ability to use a custom grammar isn't currently exposed to the end user.

<!-- gh-comment-id:2290450975 --> @0xdevalias commented on GitHub (Aug 15, 2024): > ...but judging from other open issues about the topic I'm not really sure if it works or not This seems like it's a good overarching/summarising issue; still doesn't provide full clarity yet, but links to seemingly all the related issues, and points out that now that OpenAI supports it, it's sort of become a higher priority: - https://github.com/ollama/ollama/issues/6237 --- Based on my read of these: > There are a few open PRs for this behaviour - the most recent one being https://github.com/ollama/ollama/pull/3618 it would be amazing to get this merged in. **It's a 2 line change that exposes the llama.cpp GBNF functionality via modelfile parameters.** Its not my patch but I've compiled it and used it locally and it works really well. > > _Originally posted by @ravenscroftj in https://github.com/ollama/ollama/issues/3616#issuecomment-2053939297_ > Yes you can send the grammar as an option when you submit a request **with the patch I linked to above enabled.** It just isn't documented! > > _Originally posted by @ravenscroftj in https://github.com/ollama/ollama/issues/3616#issuecomment-2068195083_ It sounds like it's not currently possible to use the GBNF functionality on the current main/released version of Ollama. According to this: > My simple personal example is this. As a newer Ollama user I actually would like to try out both approaches to see which one works better for me and my product. Right now in Ollama I simply cannot, and from appearances (which can be deceiving) it appears that what's stopping me from testing these both in Ollama is a simple code change to expose the feature in llama.cpp to me. **(edit: It was brought to my attention that Ollama actually uses GBNF internally to enforce json syntax, so the only thing that's really missing is exposing this feature to the end user to customize or use different grammar.)** > > _Originally posted by @Kinglord in https://github.com/ollama/ollama/issues/6237#issue-2453913814_ It sounds like ollama currently supports JSON mode, and that is built as a GBNF grammar (presumably on top of llama.cpp's support of it), but that the ability to use a custom grammar isn't currently exposed to the end user.

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@Kinglord commented on GitHub (Aug 15, 2024):

...but judging from other open issues about the topic I'm not really sure if it works or not

This seems like it's a good overarching/summarising issue; still doesn't provide full clarity yet, but links to seemingly all the related issues, and points out that now that OpenAI supports it, it's sort of become a higher priority:

* [Ollama Product Stance on Grammar Feature / Outstanding PRs ollama/ollama#6237](https://github.com/ollama/ollama/issues/6237)

Based on my read of these:

There are a few open PRs for this behaviour - the most recent one being ollama/ollama#3618 it would be amazing to get this merged in. It's a 2 line change that exposes the llama.cpp GBNF functionality via modelfile parameters. Its not my patch but I've compiled it and used it locally and it works really well.
Originally posted by @ravenscroftj in ollama/ollama#3616 (comment)

Yes you can send the grammar as an option when you submit a request with the patch I linked to above enabled. It just isn't documented!
Originally posted by @ravenscroftj in ollama/ollama#3616 (comment)

It sounds like it's not currently possible to use the GBNF functionality on the current main/released version of Ollama.

According to this:

My simple personal example is this. As a newer Ollama user I actually would like to try out both approaches to see which one works better for me and my product. Right now in Ollama I simply cannot, and from appearances (which can be deceiving) it appears that what's stopping me from testing these both in Ollama is a simple code change to expose the feature in llama.cpp to me. (edit: It was brought to my attention that Ollama actually uses GBNF internally to enforce json syntax, so the only thing that's really missing is exposing this feature to the end user to customize or use different grammar.)
Originally posted by @Kinglord in ollama/ollama#6237 (comment)

It sounds like ollama currently supports JSON mode, and that is built as a GBNF grammar (presumably on top of llama.cpp's support of it), but that the ability to use a custom grammar isn't currently exposed to the end user.

Sadly @0xdevalias is correct and what you want to do @jehna will not work unless you patch your version of Ollama with the PR that was linked above, the release version still has no support for GBNF outside of the built in json mode. Ollama still refuses to even reply to this issue for some really strange reason, I still have no idea why they won't talk about it all and simply let the PRs keep rolling in and sit there. At this point all we can do is keep pressuring them by raising issues, making noise both here and on the Discord until we can get someone to take 10 minutes and explain to us this decision to essentially completely block this feature from end users in Ollama.

<!-- gh-comment-id:2290736198 --> @Kinglord commented on GitHub (Aug 15, 2024): > > ...but judging from other open issues about the topic I'm not really sure if it works or not > > This seems like it's a good overarching/summarising issue; still doesn't provide full clarity yet, but links to seemingly all the related issues, and points out that now that OpenAI supports it, it's sort of become a higher priority: > > * [Ollama Product Stance on Grammar Feature / Outstanding PRs ollama/ollama#6237](https://github.com/ollama/ollama/issues/6237) > > > Based on my read of these: > > > There are a few open PRs for this behaviour - the most recent one being [ollama/ollama#3618](https://github.com/ollama/ollama/pull/3618) it would be amazing to get this merged in. **It's a 2 line change that exposes the llama.cpp GBNF functionality via modelfile parameters.** Its not my patch but I've compiled it and used it locally and it works really well. > > _Originally posted by @ravenscroftj in [ollama/ollama#3616 (comment)](https://github.com/ollama/ollama/issues/3616#issuecomment-2053939297)_ > > > Yes you can send the grammar as an option when you submit a request **with the patch I linked to above enabled.** It just isn't documented! > > _Originally posted by @ravenscroftj in [ollama/ollama#3616 (comment)](https://github.com/ollama/ollama/issues/3616#issuecomment-2068195083)_ > > It sounds like it's not currently possible to use the GBNF functionality on the current main/released version of Ollama. > > According to this: > > > My simple personal example is this. As a newer Ollama user I actually would like to try out both approaches to see which one works better for me and my product. Right now in Ollama I simply cannot, and from appearances (which can be deceiving) it appears that what's stopping me from testing these both in Ollama is a simple code change to expose the feature in llama.cpp to me. **(edit: It was brought to my attention that Ollama actually uses GBNF internally to enforce json syntax, so the only thing that's really missing is exposing this feature to the end user to customize or use different grammar.)** > > _Originally posted by @Kinglord in [ollama/ollama#6237 (comment)](https://github.com/ollama/ollama/issues/6237#issue-2453913814)_ > > It sounds like ollama currently supports JSON mode, and that is built as a GBNF grammar (presumably on top of llama.cpp's support of it), but that the ability to use a custom grammar isn't currently exposed to the end user. Sadly @0xdevalias is correct and what you want to do @jehna will not work unless you patch your version of Ollama with the PR that was linked above, the release version still has no support for GBNF outside of the built in json mode. Ollama still refuses to even reply to this issue for some really strange reason, I still have no idea why they won't talk about it all and simply let the PRs keep rolling in and sit there. At this point all we can do is keep pressuring them by raising issues, making noise both here and on the Discord until we can get someone to take 10 minutes and explain to us this decision to essentially completely block this feature from end users in Ollama.

kerem commented 2026-03-03 13:52:11 +03:00

Author

Owner

Copy link

@jehna commented on GitHub (Aug 15, 2024):

Thank you for looking into this. I just pushed ollama-support branch that should start working if they start supporting the grammar flag

<!-- gh-comment-id:2291692377 --> @jehna commented on GitHub (Aug 15, 2024): Thank you for looking into this. I just pushed `ollama-support` branch that should start working if they start supporting the grammar flag

kerem commented 2026-03-03 13:52:12 +03:00

Author

Owner

Copy link

@jehna commented on GitHub (Aug 15, 2024):

☝️ added llama3.1 8b model support

<!-- gh-comment-id:2292176639 --> @jehna commented on GitHub (Aug 15, 2024): ☝️ added llama3.1 8b model support

kerem commented 2026-03-03 13:52:12 +03:00

Author

Owner

Copy link

@dangelo352 commented on GitHub (Sep 29, 2024):

How do we use ollama with this sorry if this is a dumb question.

<!-- gh-comment-id:2381281281 --> @dangelo352 commented on GitHub (Sep 29, 2024): How do we use ollama with this sorry if this is a dumb question.

kerem commented 2026-03-03 13:52:12 +03:00

Author

Owner

Copy link

@jehna commented on GitHub (Oct 18, 2024):

@dangelo352 unfortunately there's no Ollama support yet. You can run the model locally using humanify local

<!-- gh-comment-id:2423175208 --> @jehna commented on GitHub (Oct 18, 2024): @dangelo352 unfortunately there's no Ollama support yet. You can run the model locally using `humanify local`

kerem commented 2026-03-03 13:52:12 +03:00

Author

Owner

Copy link

@jehna commented on GitHub (Oct 18, 2024):

Since v2.2.0 there's now a configurable --baseURL parameter at the OpenAI mode. Unfortunately Ollama does not yet support structured outputs, although I'm sure it's on their roadmap as the official OpenAI API supports it now.

Please keep an eye out for ollama/ollama#6473, as soon as they add the support and close that issue, you should be able to use Ollama as humanity backend by:

ollama load pull phi3.5 # or some other compatible model
humanify openai -k 'not-needed' --baseURL 'http://localhost:11434/v1' minified-file.js -m phi3.5

<!-- gh-comment-id:2423180646 --> @jehna commented on GitHub (Oct 18, 2024): Since v2.2.0 there's now a configurable `--baseURL` parameter at the OpenAI mode. Unfortunately Ollama does not yet support structured outputs, although I'm sure it's on their roadmap as the official OpenAI API supports it now. Please keep an eye out for ollama/ollama#6473, as soon as they add the support and close that issue, you should be able to use Ollama as humanity backend by: ``` ollama load pull phi3.5 # or some other compatible model humanify openai -k 'not-needed' --baseURL 'http://localhost:11434/v1' minified-file.js -m phi3.5 ```

kerem commented 2026-03-03 13:52:12 +03:00

Author

Owner

Copy link

@0xdevalias commented on GitHub (Feb 18, 2025):

Please keep an eye out for ollama/ollama#6473, as soon as they add the support and close that issue, you should be able to use Ollama as humanity backend

That issue is closed now:

This will get rolled out with https://github.com/ollama/ollama/issues/7900!

Originally posted by @ParthSareen in https://github.com/ollama/ollama/issues/6473#issuecomment-2518850099

There is a more detailed note in this related issue:

Hey everyone!

With the merging of #7900, we're introducing structured output to be able to go from a json schema to structured generation! Really appreciate all the feedback and contributions. Extremely thankful for all of you being so involved in this 🙏🏽

There are a few things we're still keeping in mind over the next few months. The first focus is going to be around performance - speed and accuracy. There has been a lot of research coming out around this, we're keeping a close eye and are going to see how we can integrate some of this into Ollama. We're also thinking about how to support structured generation in the long term and that'll play nicely with a lot of the work we're doing on our new engine.

Stoked for the coming few months, hope to improve both performance and accuracy around sampling and constrained decoding.

Thank you again for your patience, we're super excited to get this out in an upcoming release! Will spin out more issues around this as well - happy to keep you all posted as well!

Originally posted by @ParthSareen in https://github.com/ollama/ollama/issues/6237#issuecomment-2518836758

Looking at the issue it links to:

• https://github.com/ollama/ollama/pull/7900

  • Structured Outputs - Chat Endpoint

  • Structured outputs

    A longtime ask from the community - we now support the passing in of a json schema, translate to grammar and use it for sampling.

    Why not full grammar support

    We gave this a ton of thought and there's 3 main points around here:

    1. Inherent complexity of grammars - Generating a grammar for the average user is not a great experience and should be one that is abstracted away from them. Digging into the code, the API layer also needs some TLC which would mean changing some interfaces on Ollama's end while maintaining a consistent UX.
    2. Sampling performance - there are many new papers and methodologies for grammars (outlines, xgrammar, etc). We want to keep grammar generation and sampling coupled to improve the performance of sampling down the road.
    3. Parity with existing experiences - other client SDKs (e.g. OpenAI) already support structured outputs and it's imperative we keep the experience simple on our end but also support those.

    Originally posted by @ParthSareen in https://github.com/ollama/ollama/pull/7900#issue-2708318412

  • This pull request first appeared in v0.5.0-rc1

    • https://github.com/ollama/ollama/releases/tag/v0.5.0

      • Structured outputs

        Ollama now supports structured outputs, making it possible to constrain a model's output to a specific format defined by a JSON schema. The Ollama Python and JavaScript libraries have been updated to support structured outputs, together with Ollama's OpenAI-compatible API endpoints.

        REST API

        To use structured outputs in Ollama's generate or chat APIs, provide a JSON schema object in the format parameter:

        ..snip..

        JavaScript library

        Using the Ollama JavaScript library, pass in the schema as a JSON object to the format parameter as either object or use Zod (recommended) to serialize the schema using zodToJsonSchema():

        ..snip..

    • https://github.com/ollama/ollama/releases/tag/v0.5.1

      • Fixed issue where Ollama's API would generate JSON output when specifying "format": null

    • https://github.com/ollama/ollama/releases/tag/v0.5.3

      • Fixed issue where setting the format field to "" would cause an error

I'm not sure if that is sufficient for humanify to work with ollama; and if it is, whether it will work as-is or if something more needs to be done on this side to make it work. @jehna would probably be the best to know that off-hand.

---

While I haven't looked into it deeper, I noticed that there is an Ollama JS SDK, so if we wanted to build more specific support into humanify, perhaps that would be a good place to start:

• https://github.com/ollama/ollama-js

  • Ollama JavaScript Library
    The Ollama JavaScript library provides the easiest way to integrate your JavaScript project with Ollama.

<!-- gh-comment-id:2665115599 --> @0xdevalias commented on GitHub (Feb 18, 2025): > Please keep an eye out for [ollama/ollama#6473](https://github.com/ollama/ollama/issues/6473), as soon as they add the support and close that issue, you should be able to use Ollama as humanity backend That issue is closed now: > This will get rolled out with https://github.com/ollama/ollama/issues/7900! > > _Originally posted by @ParthSareen in https://github.com/ollama/ollama/issues/6473#issuecomment-2518850099_ There is a more detailed note in this related issue: > Hey everyone! > > With the merging of #7900, we're introducing structured output to be able to go from a json schema to structured generation! Really appreciate all the feedback and contributions. Extremely thankful for all of you being so involved in this 🙏🏽 > > There are a few things we're still keeping in mind over the next few months. The first focus is going to be around performance - speed and accuracy. There has been a lot of research coming out around this, we're keeping a close eye and are going to see how we can integrate some of this into Ollama. We're also thinking about how to support structured generation in the long term and that'll play nicely with a lot of the work we're doing on our new engine. > > Stoked for the coming few months, hope to improve both performance and accuracy around sampling and constrained decoding. > > Thank you again for your patience, we're super excited to get this out in an upcoming release! Will spin out more issues around this as well - happy to keep you all posted as well! > > _Originally posted by @ParthSareen in https://github.com/ollama/ollama/issues/6237#issuecomment-2518836758_ Looking at the issue it links to: - https://github.com/ollama/ollama/pull/7900 - > Structured Outputs - Chat Endpoint - > ## Structured outputs > A longtime ask from the community - we now support the passing in of a json schema, translate to grammar and use it for sampling. > > ## Why not full grammar support > We gave this a ton of thought and there's 3 main points around here: > > 1. Inherent complexity of grammars - Generating a grammar for the average user is not a great experience and should be one that is abstracted away from them. Digging into the code, the API layer also needs some TLC which would mean changing some interfaces on Ollama's end while maintaining a consistent UX. > 2. Sampling performance - there are many new papers and methodologies for grammars (outlines, xgrammar, etc). We want to keep grammar generation and sampling coupled to improve the performance of sampling down the road. > 3. Parity with existing experiences - other client SDKs (e.g. OpenAI) already support structured outputs and it's imperative we keep the experience simple on our end but also support those. > > _Originally posted by @ParthSareen in https://github.com/ollama/ollama/pull/7900#issue-2708318412_ - > This pull request first appeared in [v0.5.0-rc1](https://github.com/ollama/ollama/releases/tag/v0.5.0-rc1) - https://github.com/ollama/ollama/releases/tag/v0.5.0 - > ## Structured outputs > > Ollama now supports structured outputs, making it possible to constrain a model's output to a specific format defined by a JSON schema. The Ollama Python and JavaScript libraries have been updated to support structured outputs, together with Ollama's OpenAI-compatible API endpoints. > > ### REST API > > To use structured outputs in Ollama's generate or chat APIs, provide a JSON schema object in the format parameter: > > ..snip.. > > ### JavaScript library > > Using the [Ollama JavaScript library](https://github.com/ollama/ollama-js), pass in the schema as a JSON object to the `format` parameter as either `object` or use Zod (recommended) to serialize the schema using `zodToJsonSchema()`: > > ..snip.. - https://github.com/ollama/ollama/releases/tag/v0.5.1 - > Fixed issue where Ollama's API would generate JSON output when specifying `"format": null` - https://github.com/ollama/ollama/releases/tag/v0.5.3 - > Fixed issue where setting the `format` field to `""` would cause an error I'm not sure if that is sufficient for `humanify` to work with ollama; and if it is, whether it will work as-is or if something more needs to be done on this side to make it work. @jehna would probably be the best to know that off-hand. --- While I haven't looked into it deeper, I noticed that there is an Ollama JS SDK, so if we wanted to build more specific support into `humanify`, perhaps that would be a good place to start: - https://github.com/ollama/ollama-js - > Ollama JavaScript Library > The Ollama JavaScript library provides the easiest way to integrate your JavaScript project with [Ollama](https://github.com/jmorganca/ollama).

kerem commented 2026-03-03 13:52:12 +03:00

Author

Owner

Copy link

@0xdevalias commented on GitHub (Apr 10, 2025):

See also:

• https://github.com/jehna/humanify/issues/400
• https://github.com/jehna/humanify/issues/84
• https://github.com/jehna/humanify/issues/416
• https://github.com/jehna/humanify/issues/419
• https://github.com/jehna/humanify/pull/646

  • This adds the ollama command that can be used and uses the default model gpt-oss:20b

• https://github.com/jehna/humanify/pull/647

<!-- gh-comment-id:2791690992 --> @0xdevalias commented on GitHub (Apr 10, 2025): See also: - https://github.com/jehna/humanify/issues/400 - https://github.com/jehna/humanify/issues/84 - https://github.com/jehna/humanify/issues/416 - https://github.com/jehna/humanify/issues/419 - https://github.com/jehna/humanify/pull/646 - > This adds the `ollama` command that can be used and uses the default model `gpt-oss:20b` - https://github.com/jehna/humanify/pull/647

kerem commented 2026-03-03 13:52:12 +03:00

Author

Owner

Copy link

@0xdevalias commented on GitHub (Sep 20, 2025):

• https://ollama.com/blog/cloud-models

  • Cloud models (September 19, 2025)

  • Cloud models are now in preview, letting you run larger models with fast, datacenter-grade hardware. You can keep using your local tools while running larger models that wouldn’t fit on a personal computer. Ollama’s cloud does not retain your data to ensure privacy and security.

    The same Ollama experience is now seamless across both local and in the cloud, integrating with the existing tools you already use. Ollama’s cloud models also work via Ollama’s OpenAI-compatible API.

  • https://ollama.com/cloud

    • Run larger models, faster using Ollama's cloud: $20/mo

    • What are the usage limits for Ollama's cloud?

      Ollama's cloud includes hourly and daily limits to avoid capacity issues. Usage-based pricing will soon be available to consume models in a metered fashion.

<!-- gh-comment-id:3314315120 --> @0xdevalias commented on GitHub (Sep 20, 2025): - https://ollama.com/blog/cloud-models - > Cloud models (September 19, 2025) - > Cloud models are now in preview, letting you run larger models with fast, datacenter-grade hardware. You can keep using your local tools while running larger models that wouldn’t fit on a personal computer. Ollama’s cloud does not retain your data to ensure privacy and security. > > The same Ollama experience is now seamless across both local and in the cloud, integrating with the existing tools you already use. Ollama’s cloud models also work via Ollama’s [OpenAI-compatible API](https://docs.ollama.com/openai). - https://ollama.com/cloud - > Run larger models, faster using Ollama's cloud: $20/mo - > What are the usage limits for Ollama's cloud? > > Ollama's cloud includes hourly and daily limits to avoid capacity issues. Usage-based pricing will soon be available to consume models in a metered fashion.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

claude/rust-ast-rewrite-SjZI6

dependabot/npm_and_yarn/babel/types-7.29.0

dependabot/npm_and_yarn/globals-17.3.0

dependabot/npm_and_yarn/babel/core-7.29.0

dependabot/npm_and_yarn/commander-14.0.3

dependabot/npm_and_yarn/types/node-25.2.0

dependabot/npm_and_yarn/pkgroll-2.23.0

fix/tests

dependabot/npm_and_yarn/openai-6.17.0

dependabot/npm_and_yarn/typescript-eslint-8.54.0

dependabot/npm_and_yarn/node-llama-cpp-3.15.1

dependabot/npm_and_yarn/prettier-3.8.1

dependabot/npm_and_yarn/eslint-9.39.2

dependabot/npm_and_yarn/eslint/js-9.39.2

dependabot/npm_and_yarn/tsx-4.21.0

dependabot/npm_and_yarn/eslint-plugin-unused-imports-4.3.0

dependabot/npm_and_yarn/typescript-5.9.3

dependabot/npm_and_yarn/dotenv-17.2.3

dependabot/npm_and_yarn/ipull-4.0.3

dependabot/npm_and_yarn/google/generative-ai-0.24.1

ollama-support

add/github-azure-integration

v1

v2.2.2

v2.2.1

v2.2.0

v2.1.3

v2.1.2

v2.1.1

v2.1.0

v2.0.3

v2.0.2

v2.0.1

v2.0.0

v1.0

Labels

Clear labels   

bug

  

enhancement

  

pull-request

Mirrored from GitHub Pull Request

  

wontfix

No labels

bug

enhancement

pull-request

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/humanify#10

No description provided.