starred/humanify
1
0
Fork 0
mirror of https://github.com/jehna/humanify.git synced 2026-04-28 01:55:56 +03:00
Code Issues 88 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #400] Add an API provider abstraction layer to easily support more services/models/APIs #78

New issue
Open
opened 2026-03-03 13:52:49 +03:00 by kerem · 2 comments
kerem commented 2026-03-03 13:52:49 +03:00
Owner
Copy link

Originally created by @0xdevalias on GitHub (Apr 10, 2025).
Original GitHub issue: https://github.com/jehna/humanify/issues/400

This has been mentioned in other issues in the past:

I wonder if it's worth implementing a wrapper/abstraction layer like LiteLLM to make things more flexible?

  • https://github.com/BerriAI/litellm

    • Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

This is what projects like aider use:

  • https://aider.chat/docs/llms/other.html

    • Aider uses the litellm package to connect to hundreds of other models. You can use aider --model <model-name> to use any supported model.

      To explore the list of supported models you can run aider --models <model-name> with a partial model name. If the supplied name is not an exact match for a known model, aider will return a list of possible matching models.

Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/14#issuecomment-2179563971

And would help support issues such as the following in a more unified way:

I haven't looked too deeply into all of the options available in this space, but one I re-stumbled across again today that made me think to create this issue:

There's also this brief overview I did in a comment RE: wanting a JavaScript version of litellm:

And it may be interesting to look into options/services like this that aim to support choosing/optimising across different providers/models/etc:

  • https://www.notdiamond.ai/

    • An end-to-end multi-model framework

    • Intelligent routing
      Not Diamond can help you take any evaluation data for any set of models over any set of inputs and create an optimal routing algorithm tailored to your application.

    • Automatic prompt adaptation
      Automatically adapt prompts across LLMs so you always call the right model with the right prompt. No more manual tweaking and experimentation.

    • https://www.notdiamond.ai/pricing

See Also

Originally created by @0xdevalias on GitHub (Apr 10, 2025). Original GitHub issue: https://github.com/jehna/humanify/issues/400 This has been mentioned in other issues in the past: > I wonder if it's worth implementing a wrapper/abstraction layer like LiteLLM to make things more flexible? > > - https://github.com/BerriAI/litellm > - > Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs) > > This is what projects like [`aider`](https://github.com/paul-gauthier/aider) use: > > - https://aider.chat/docs/llms/other.html > - > Aider uses the [litellm](https://docs.litellm.ai/docs/providers) package to connect to hundreds of other models. You can use aider `--model <model-name>` to use any supported model. > > > > To explore the list of supported models you can run `aider --models <model-name>` with a partial model name. If the supplied name is not an exact match for a known model, aider will return a list of possible matching models. > > _Originally posted by @0xdevalias in https://github.com/jehna/humanify/issues/14#issuecomment-2179563971_ And would help support issues such as the following in a more unified way: - https://github.com/jehna/humanify/issues/11 - https://github.com/jehna/humanify/issues/14 - https://github.com/jehna/humanify/issues/213 - https://github.com/jehna/humanify/pull/272 - https://github.com/jehna/humanify/issues/84 - https://github.com/jehna/humanify/issues/392 I haven't looked too deeply into all of the options available in this space, but one I re-stumbled across again today that made me think to create this issue: - https://github.com/vercel/ai - > The AI Toolkit for TypeScript. From the creators of Next.js, the AI SDK is a free open-source library for building AI-powered applications and agents - https://github.com/vercel/ai#ai-sdk-core - > AI SDK Core - > The [AI SDK Core](https://sdk.vercel.ai/docs/ai-sdk-core/overview) module provides a unified API to interact with model providers like [OpenAI](https://sdk.vercel.ai/providers/ai-sdk-providers/openai), [Anthropic](https://sdk.vercel.ai/providers/ai-sdk-providers/anthropic), [Google](https://sdk.vercel.ai/providers/ai-sdk-providers/google-generative-ai), and more. > > You will then install the model provider of your choice. - https://sdk.vercel.ai/ - > The AI Toolkit for TypeScript - https://sdk.vercel.ai/docs/foundations/providers-and-models - > Providers and Models - > AI SDK Core offers a standardized approach to interacting with LLMs through a language model specification that abstracts differences between providers. This unified interface allows you to switch between providers with ease while using the same API for all providers. - https://sdk.vercel.ai/docs/foundations/providers-and-models#ai-sdk-providers - > AI SDK Providers - > The AI SDK comes with a wide range of providers that you can use to interact with different language models - > You can also use the [OpenAI Compatible provider](https://sdk.vercel.ai/providers/openai-compatible-providers) with OpenAI-compatible APIs - > Our [language model specification](https://github.com/vercel/ai/tree/main/packages/provider/src/language-model/v1) is published as an open-source package, which you can use to create [custom providers](https://sdk.vercel.ai/providers/community-providers/custom-providers). - > The open-source community has created the following providers... - https://sdk.vercel.ai/docs/foundations/providers-and-models#self-hosted-models - > Self-Hosted Models > You can access self-hosted models with the following providers... - > Additionally, any self-hosted provider that supports the OpenAI specification can be used with the [OpenAI Compatible Provider ](https://sdk.vercel.ai/providers/openai-compatible-providers). - https://sdk.vercel.ai/docs/foundations/providers-and-models#model-capabilities - > Model Capabilities > The AI providers support different language models with various capabilities. Here are the capabilities of popular models... - https://sdk.vercel.ai/docs/ai-sdk-core/generating-structured-data - > Generating Structured Data - https://sdk.vercel.ai/providers/community-providers/ollama - > [Ollama Provider](https://sdk.vercel.ai/providers/community-providers/ollama#ollama-provider) [sgomez/ollama-ai-provider](https://github.com/sgomez/ollama-ai-provider) is a community provider that uses [Ollama](https://ollama.com/) to provide language model support for the AI SDK. - https://github.com/sgomez/ollama-ai-provider - > Vercel AI Provider for running LLMs locally using Ollama - https://sdk.vercel.ai/providers/community-providers/openrouter - > OpenRouter - > [OpenRouter](https://openrouter.ai/) is a unified API gateway that provides access to hundreds of AI models from leading providers like Anthropic, Google, Meta, Mistral, and more. The OpenRouter provider for the AI SDK enables seamless integration with all these models while offering unique advantages There's also this brief overview I did in a comment RE: wanting a JavaScript version of litellm: - https://github.com/BerriAI/litellm/issues/361#issuecomment-2822862100 And it may be interesting to look into options/services like this that aim to support choosing/optimising across different providers/models/etc: - https://www.notdiamond.ai/ - > An end-to-end multi-model framework - > Intelligent routing > Not Diamond can help you take any evaluation data for any set of models over any set of inputs and create an optimal routing algorithm tailored to your application. - > Automatic prompt adaptation > Automatically adapt prompts across LLMs so you always call the right model with the right prompt. No more manual tweaking and experimentation. - https://www.notdiamond.ai/pricing ## See Also - https://github.com/jehna/humanify/issues/14 - https://github.com/jehna/humanify/issues/84 - https://github.com/jehna/humanify/issues/213 - https://github.com/jehna/humanify/issues/392 - https://github.com/jehna/humanify/issues/416 - https://github.com/jehna/humanify/issues/419 - https://github.com/jehna/humanify/issues/481 - https://github.com/jehna/humanify/issues/502
kerem commented 2026-03-03 13:52:49 +03:00
Author
Owner
Copy link

@0xdevalias commented on GitHub (Oct 8, 2025):

The OpenAI Agents SDK:

Supports different models / model providers (including non-OpenAI ones):

It looks like we can find theModelProvider / Model classes in @openai/core / @openai/agents-openai / @openai/agents:

It looks like we can find the OpenAIProvider / OpenAIChatCompletionsModel / OpenAIResponsesModel classes in @openai/agents-openai / @openai/agents:

It looks like we can find adapters for the Vercel AI SDK in @openai/agents-extensions:

<!-- gh-comment-id:3379537266 --> @0xdevalias commented on GitHub (Oct 8, 2025): The OpenAI Agents SDK: - https://github.com/openai/openai-agents-js - > OpenAI Agents SDK (JavaScript/TypeScript) - > A lightweight, powerful framework for multi-agent workflows and voice agents - > The OpenAI Agents SDK is a lightweight yet powerful framework for building multi-agent workflows in JavaScript/TypeScript. It is provider-agnostic, supporting OpenAI APIs and more. - https://openai.github.io/openai-agents-js/ Supports different models / model providers (including non-OpenAI ones): - https://openai.github.io/openai-agents-js/guides/models/ - > **Models** > > Every Agent ultimately calls an LLM. The SDK abstracts models behind two lightweight interfaces: > > - [`Model`](https://openai.github.io/openai-agents-js/openai/agents/interfaces/model) – knows how to make _one_ request against a specific API. > - [`ModelProvider`](https://openai.github.io/openai-agents-js/openai/agents/interfaces/modelprovider) – resolves human‑readable model **names** (e.g. `'gpt‑4o'`) to `Model` instances. - https://openai.github.io/openai-agents-js/guides/models/#custom-model-providers - > **Custom model providers** > > Implementing your own provider is straightforward – implement `ModelProvider` and `Model` and pass the provider to the `Runner` constructor It looks like we can find the`ModelProvider` / `Model` classes in `@openai/core` / `@openai/agents-openai` / `@openai/agents`: - https://openai.github.io/openai-agents-js/openai/agents-core/interfaces/modelprovider/ - > **ModelProvider** > > The base interface for a model provider. > > The model provider is responsible for looking up `Model` instances by name. - https://openai.github.io/openai-agents-js/openai/agents-core/interfaces/model/ - > **Model** > > The base interface for calling an LLM. It looks like we can find the `OpenAIProvider` / `OpenAIChatCompletionsModel` / `OpenAIResponsesModel` classes in `@openai/agents-openai` / `@openai/agents`: - https://openai.github.io/openai-agents-js/openai/agents-openai/classes/openaiprovider/ - > **OpenAIProvider** > > The provider of OpenAI’s models (or Chat Completions compatible ones) - https://openai.github.io/openai-agents-js/openai/agents-openai/classes/openaichatcompletionsmodel/ - > **OpenAIChatCompletionsModel** > > A model that uses (or is compatible with) OpenAI’s Chat Completions API. - https://openai.github.io/openai-agents-js/openai/agents-openai/classes/openairesponsesmodel/ - > **OpenAIResponsesModel** > > Model implementation that uses OpenAI’s Responses API to generate responses. It looks like we can find adapters for the Vercel AI SDK in `@openai/agents-extensions`: - https://openai.github.io/openai-agents-js/openai/agents-extensions/classes/aisdkmodel/ - > **AiSdkModel** > > Wraps a model from the AI SDK that adheres to the LanguageModelV2 spec to be used used as a model in the OpenAI Agents SDK to use other models. > > While you can use this with the OpenAI models, it is recommended to use the default OpenAI model provider instead. > > If tracing is enabled, the model will send generation spans to your traces processor. - https://ai-sdk.dev/docs/foundations/providers-and-models#ai-sdk-providers - > **AI SDK Providers** > > The AI SDK comes with a wide range of providers that you can use to interact with different language models: - https://ai-sdk.dev/providers/community-providers/custom-providers - > **Writing a Custom Provider** > > The AI SDK provides a [Language Model Specification](https://github.com/vercel/ai/tree/main/packages/provider/src/language-model/v2) that enables you to create custom providers compatible with the AI SDK. This specification ensures consistency across different providers. - https://github.com/vercel/ai/tree/main/packages/provider/src/language-model/v2
kerem commented 2026-03-03 13:52:49 +03:00
Author
Owner
Copy link

@0xdevalias commented on GitHub (Oct 8, 2025):

The Vercel AI SDK v5 supports the concept of a Global Provider:

  • https://vercel.com/blog/ai-sdk-5#global-provider

    • Global Provider

      The AI SDK 5 includes a global provider feature that allows you to specify a model using just a plain model ID string:

      import { streamText } from 'ai';

const result = await streamText({
  model: 'openai/gpt-4o', // Uses the global provider (defaults to AI Gateway)
  prompt: 'Invent a new holiday and describe its traditions.',
});

      By default, the global provider is set to the Vercel AI Gateway.

    • Customizing the Global Provider

      You can set your own preferred global provider:

      import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';

// Initialise once during startup:
globalThis.AI_SDK_DEFAULT_PROVIDER = openai;

// Somewhere else in your codebase:
const result = streamText({
  model: 'gpt-4o', // Uses OpenAI provider without prefix
  prompt: 'Invent a new holiday and describe its traditions.',
});

      This simplifies provider usage and makes it easier to switch between providers without changing your model references throughout your codebase.

Which defaults to using the Vercel AI Gateway, which you can read more about here:

  • https://vercel.com/ai-gateway

    • The AI Gateway For Developers

      Robustly access hundreds of AI models through a centralized interface and ship with ease.

    • Can I try the Gateway for free?

      Yes! When you sign up for a Vercel account, you get $5 of credits every 30 days to try out any model from our model list. We don’t restrict access to premium models.

      Note: After you make your first payment, you are considered a paid customer and will no longer receive the free credits.

    • How is the Gateway priced?

      We offer tokens at list price from the upstream providers with no markup.

      If you bring your own key, we will not add any markup to your token price (0%).

      You are responsible for any payment processing fees.

    • Do you have any rate limits?

      While the upstream providers may have limits, Vercel doesn’t place any rate limits on your queries.

      We are constantly working with the upstream providers to get you the maximum limits, throughput and reliability.

    • https://vercel.com/docs/ai-gateway
    • https://vercel.com/ai-gateway/models

      • Browse AI Gateway Models

        A catalog of models to help you build AI features for your Vercel project.

    • https://vercel.com/docs/ai-gateway/byok

      • Bring Your Own Key (BYOK)

        Using your own credentials with an external AI provider allows AI Gateway to authenticate requests on your behalf with no added markup. This approach is useful for utilizing credits provided by the AI provider or executing AI queries that access private cloud data. If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability.

    • https://vercel.com/docs/ai-gateway/app-attribution

      • App Attribution

        App attribution allows Vercel to identify the application making a request through AI Gateway. When provided, your app can be featured on AI Gateway pages, driving awareness.

<!-- gh-comment-id:3379563928 --> @0xdevalias commented on GitHub (Oct 8, 2025): The Vercel AI SDK v5 supports the concept of a Global Provider: - https://vercel.com/blog/ai-sdk-5#global-provider - > **Global Provider** > > The AI SDK 5 includes a global provider feature that allows you to specify a model using just a plain model ID string: > > ```typescript > import { streamText } from 'ai'; > > const result = await streamText({ > model: 'openai/gpt-4o', // Uses the global provider (defaults to AI Gateway) > prompt: 'Invent a new holiday and describe its traditions.', > }); > ``` > > By default, the global provider is set to the [Vercel AI Gateway](https://vercel.com/docs/ai-gateway). - > **Customizing the Global Provider** > > You can set your own preferred global provider: > > ```typescript > import { openai } from '@ai-sdk/openai'; > import { streamText } from 'ai'; > > // Initialise once during startup: > globalThis.AI_SDK_DEFAULT_PROVIDER = openai; > > // Somewhere else in your codebase: > const result = streamText({ > model: 'gpt-4o', // Uses OpenAI provider without prefix > prompt: 'Invent a new holiday and describe its traditions.', > }); > ``` > > This simplifies provider usage and makes it easier to switch between providers without changing your model references throughout your codebase. Which defaults to using the Vercel AI Gateway, which you can read more about here: - https://vercel.com/ai-gateway - > **The AI Gateway For Developers** > > Robustly access hundreds of AI models through a centralized interface and ship with ease. - > **Can I try the Gateway for free?** > > Yes! When you sign up for a Vercel account, you get $5 of credits every 30 days to try out any model from our model list. We don’t restrict access to premium models. > > Note: After you make your first payment, you are considered a paid customer and will no longer receive the free credits. - > **How is the Gateway priced?** > > We offer tokens at list price from the upstream providers with no markup. > > If you bring your own key, we will not add any markup to your token price (0%). > > You are responsible for any payment processing fees. - > Do you have any rate limits? > > While the upstream providers may have limits, Vercel doesn’t place any rate limits on your queries. > > We are constantly working with the upstream providers to get you the maximum limits, throughput and reliability. - https://vercel.com/docs/ai-gateway - > AI Gateway - > The [AI Gateway](https://vercel.com/ai-gateway) provides a unified API to access [hundreds of models](https://vercel.com/ai-gateway/models) through a single endpoint. It gives you the ability to set budgets, monitor usage, load-balance requests, and manage fallbacks. > > The design allows it to work seamlessly with [AI SDK 5](https://vercel.com/docs/ai-gateway/getting-started), [OpenAI SDK](https://vercel.com/docs/ai-gateway/openai-compat), or your [preferred framework](https://vercel.com/docs/ai-gateway/framework-integrations). - https://vercel.com/ai-gateway/models - > **Browse AI Gateway Models** > > A catalog of models to help you build AI features for your Vercel project. - https://vercel.com/docs/ai-gateway/byok - > **Bring Your Own Key (BYOK)** > > Using your own credentials with an external AI provider allows AI Gateway to authenticate requests on your behalf with [no added markup](https://vercel.com/docs/ai-gateway/pricing#using-a-custom-api-key). This approach is useful for utilizing credits provided by the AI provider or executing AI queries that access private cloud data. If a query using your credentials fails, AI Gateway will retry the query with its system credentials to improve service availability. - https://vercel.com/docs/ai-gateway/app-attribution - > **App Attribution** > > App attribution allows Vercel to identify the application making a request through AI Gateway. When provided, your app can be featured on AI Gateway pages, driving awareness.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
claude/rust-ast-rewrite-SjZI6
dependabot/npm_and_yarn/babel/types-7.29.0
dependabot/npm_and_yarn/globals-17.3.0
dependabot/npm_and_yarn/babel/core-7.29.0
dependabot/npm_and_yarn/commander-14.0.3
dependabot/npm_and_yarn/types/node-25.2.0
dependabot/npm_and_yarn/pkgroll-2.23.0
fix/tests
dependabot/npm_and_yarn/openai-6.17.0
dependabot/npm_and_yarn/typescript-eslint-8.54.0
dependabot/npm_and_yarn/node-llama-cpp-3.15.1
dependabot/npm_and_yarn/prettier-3.8.1
dependabot/npm_and_yarn/eslint-9.39.2
dependabot/npm_and_yarn/eslint/js-9.39.2
dependabot/npm_and_yarn/tsx-4.21.0
dependabot/npm_and_yarn/eslint-plugin-unused-imports-4.3.0
dependabot/npm_and_yarn/typescript-5.9.3
dependabot/npm_and_yarn/dotenv-17.2.3
dependabot/npm_and_yarn/ipull-4.0.3
dependabot/npm_and_yarn/google/generative-ai-0.24.1
ollama-support
add/github-azure-integration
v1
v2.2.2
v2.2.1
v2.2.0
v2.1.3
v2.1.2
v2.1.1
v2.1.0
v2.0.3
v2.0.2
v2.0.1
v2.0.0
v1.0
Labels
Clear labels   
bug

   
enhancement

   
pull-request

Mirrored from GitHub Pull Request

  
wontfix
No labels
bug
enhancement
pull-request
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/humanify#78
No description provided.