starred/humanify
1
0
Fork 0
mirror of https://github.com/jehna/humanify.git synced 2026-04-27 01:26:00 +03:00
Code Issues 88 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #50] Issue running local model on a newly installed Windows 11 machine. #25

New issue
Open
opened 2026-03-03 13:52:18 +03:00 by kerem · 7 comments
kerem commented 2026-03-03 13:52:18 +03:00
Owner
Copy link

Originally created by @TheGreyRaven on GitHub (Aug 22, 2024).
Original GitHub issue: https://github.com/jehna/humanify/issues/50

Hey!
I have a freshly installed Windows 11 machine with Node 22 installed, when I try to run humanify local <file>.js I get the following crash:

ggml_vulkan: Found 1 Vulkan devices:
Vulkan0: NVIDIA RTX A500 Laptop GPU (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32
[node-llama-cpp] Using this model ("C:\Users\Raven\.humanifyjs\models\Phi-3.1-mini-4k-instruct-Q4_K_M.gguf") to tokenize text with special tokens and then detokenize it resulted in a different text. There might be an issue with the model or the tokenizer implementation. Using this model may not work as intended
ggml_vulkan: Device memory allocation of size 314576896 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate NVIDIA RTX A500 Laptop GPU buffer of size 314576896
[node-llama-cpp] llama_new_context_with_model: failed to allocate compute buffers
file:///C:/Users/Ravem/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:461
                throw new Error("Failed to create context");
                      ^

Error: Failed to create context
    at LlamaContext._create (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:461:23)
    at async Object.<anonymous> (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaModel/LlamaModel.js:274:24)
    at async withLock (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/lifecycle-utils/dist/withLock.js:36:16)
    at async LlamaModel.createContext (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaModel/LlamaModel.js:271:16)
    at async llama (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:157:19)
    at async Command.<anonymous> (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:56702:18)

Node.js v22.6.0

I have also downloaded the local model, any ideas what could be wrong?

Originally created by @TheGreyRaven on GitHub (Aug 22, 2024). Original GitHub issue: https://github.com/jehna/humanify/issues/50 Hey! I have a freshly installed Windows 11 machine with Node 22 installed, when I try to run ``humanify local <file>.js`` I get the following crash: ``` ggml_vulkan: Found 1 Vulkan devices: Vulkan0: NVIDIA RTX A500 Laptop GPU (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 [node-llama-cpp] Using this model ("C:\Users\Raven\.humanifyjs\models\Phi-3.1-mini-4k-instruct-Q4_K_M.gguf") to tokenize text with special tokens and then detokenize it resulted in a different text. There might be an issue with the model or the tokenizer implementation. Using this model may not work as intended ggml_vulkan: Device memory allocation of size 314576896 failed. ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory ggml_gallocr_reserve_n: failed to allocate NVIDIA RTX A500 Laptop GPU buffer of size 314576896 [node-llama-cpp] llama_new_context_with_model: failed to allocate compute buffers file:///C:/Users/Ravem/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:461 throw new Error("Failed to create context"); ^ Error: Failed to create context at LlamaContext._create (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:461:23) at async Object.<anonymous> (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaModel/LlamaModel.js:274:24) at async withLock (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/lifecycle-utils/dist/withLock.js:36:16) at async LlamaModel.createContext (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaModel/LlamaModel.js:271:16) at async llama (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:157:19) at async Command.<anonymous> (file:///C:/Users/Raven/AppData/Roaming/npm/node_modules/humanifyjs/dist/index.mjs:56702:18) Node.js v22.6.0 ``` I have also downloaded the local model, any ideas what could be wrong?
kerem commented 2026-03-03 13:52:19 +03:00
Author
Owner
Copy link

@jehna commented on GitHub (Aug 22, 2024):

How much GPU memory does your NVIDIA RTX A500 have? Humanify should run with 7gb GPU memory pretty well, but no guarantees for a system with less memory than that.

You can however use the --disableGpu flag to run the model on your CPU. This may be slower though.

<!-- gh-comment-id:2303933189 --> @jehna commented on GitHub (Aug 22, 2024): How much GPU memory does your NVIDIA RTX A500 have? Humanify should run with 7gb GPU memory pretty well, but no guarantees for a system with less memory than that. You can however use the `--disableGpu` flag to run the model on your CPU. This may be slower though.
kerem commented 2026-03-03 13:52:19 +03:00
Author
Owner
Copy link

@TheGreyRaven commented on GitHub (Aug 22, 2024):

It has 4GB of Video memory so that could be the issue, though even if I run humanify local --disableGpu <file> I still get the exact same crash:

ggml_vulkan: Found 1 Vulkan devices:
Vulkan0: NVIDIA RTX A500 Laptop GPU (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32
ggml_vulkan: Device memory allocation of size 314576896 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate NVIDIA RTX A500 Laptop GPU buffer of size 314576896
[node-llama-cpp] llama_new_context_with_model: failed to allocate compute buffers
.......
.......
.......
<!-- gh-comment-id:2304028451 --> @TheGreyRaven commented on GitHub (Aug 22, 2024): It has 4GB of Video memory so that could be the issue, though even if I run ``humanify local --disableGpu <file>`` I still get the exact same crash: ``` ggml_vulkan: Found 1 Vulkan devices: Vulkan0: NVIDIA RTX A500 Laptop GPU (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 ggml_vulkan: Device memory allocation of size 314576896 failed. ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory ggml_gallocr_reserve_n: failed to allocate NVIDIA RTX A500 Laptop GPU buffer of size 314576896 [node-llama-cpp] llama_new_context_with_model: failed to allocate compute buffers ....... ....... ....... ```
kerem commented 2026-03-03 13:52:19 +03:00
Author
Owner
Copy link

@jehna commented on GitHub (Aug 24, 2024):

Seems that you've found a bug! I'll create a patch for that, thank you for sending good debug info

<!-- gh-comment-id:2308178982 --> @jehna commented on GitHub (Aug 24, 2024): Seems that you've found a bug! I'll create a patch for that, thank you for sending good debug info
kerem commented 2026-03-03 13:52:19 +03:00
Author
Owner
Copy link

@0xdevalias commented on GitHub (Aug 26, 2024):

With the bugfix PR merged now, can this issue be closed?

<!-- gh-comment-id:2309179487 --> @0xdevalias commented on GitHub (Aug 26, 2024): With the bugfix PR merged now, can this issue be closed? - #55 - #56
kerem commented 2026-03-03 13:52:19 +03:00
Author
Owner
Copy link

@TheGreyRaven commented on GitHub (Aug 26, 2024):

Glad to help out, I did a small workaround for now by running humanify through WSL with installed Nvidia drivers and there everything works great!

<!-- gh-comment-id:2309959351 --> @TheGreyRaven commented on GitHub (Aug 26, 2024): Glad to help out, I did a small workaround for now by running humanify through WSL with installed Nvidia drivers and there everything works great!
kerem commented 2026-03-03 13:52:20 +03:00
Author
Owner
Copy link

@TheGreyRaven commented on GitHub (Jan 21, 2025):

Hello again, so I have once again tried to run the project on my Windows 11 machine but it seems like the bug is still present.

ggml_vulkan: Compiling shaders...................................................Done!
ggml_vulkan: Device memory allocation of size 1610612736 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
[node-llama-cpp] llama_kv_cache_init: failed to allocate buffer for kv cache
[node-llama-cpp] llama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache
file:///C:/Users/oscar/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:612
                    throw new Error("Failed to create context");

Once again it runs fine in WSL

<!-- gh-comment-id:2603879942 --> @TheGreyRaven commented on GitHub (Jan 21, 2025): Hello again, so I have once again tried to run the project on my Windows 11 machine but it seems like the bug is still present. ``` ggml_vulkan: Compiling shaders...................................................Done! ggml_vulkan: Device memory allocation of size 1610612736 failed. ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory [node-llama-cpp] llama_kv_cache_init: failed to allocate buffer for kv cache [node-llama-cpp] llama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache file:///C:/Users/oscar/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:612 throw new Error("Failed to create context"); ``` Once again it runs fine in WSL
kerem commented 2026-03-03 13:52:20 +03:00
Author
Owner
Copy link

@0xdevalias commented on GitHub (Mar 24, 2025):

it seems like the bug is still present.

@TheGreyRaven Can you confirm that when you say the bug is still present, what version of humanify are you running, and are you passing the --disableGpu flag as suggested earlier, etc?

I notice the last error was:

ggml_vulkan: Device memory allocation of size 314576896 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate NVIDIA RTX A500 Laptop GPU buffer of size 314576896
[node-llama-cpp] llama_new_context_with_model: failed to allocate compute buffers
file:///C:/Users/Ravem/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:461
                throw new Error("Failed to create context");

Whereas this one is:

[node-llama-cpp] llama_kv_cache_init: failed to allocate buffer for kv cache
[node-llama-cpp] llama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache
file:///C:/Users/oscar/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:612
                    throw new Error("Failed to create context");

Not sure if the nuance between those two makes a difference, but figured I would note it just in case.

<!-- gh-comment-id:2746856988 --> @0xdevalias commented on GitHub (Mar 24, 2025): > it seems like the bug is still present. @TheGreyRaven Can you confirm that when you say the bug is still present, what version of `humanify` are you running, and are you passing the `--disableGpu` flag as suggested earlier, etc? I notice the last error was: ``` ggml_vulkan: Device memory allocation of size 314576896 failed. ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory ggml_gallocr_reserve_n: failed to allocate NVIDIA RTX A500 Laptop GPU buffer of size 314576896 [node-llama-cpp] llama_new_context_with_model: failed to allocate compute buffers file:///C:/Users/Ravem/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:461 throw new Error("Failed to create context"); ``` Whereas this one is: ``` [node-llama-cpp] llama_kv_cache_init: failed to allocate buffer for kv cache [node-llama-cpp] llama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache file:///C:/Users/oscar/AppData/Roaming/npm/node_modules/humanifyjs/node_modules/node-llama-cpp/dist/evaluator/LlamaContext/LlamaContext.js:612 throw new Error("Failed to create context"); ``` Not sure if the nuance between those two makes a difference, but figured I would note it just in case.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
claude/rust-ast-rewrite-SjZI6
dependabot/npm_and_yarn/babel/types-7.29.0
dependabot/npm_and_yarn/globals-17.3.0
dependabot/npm_and_yarn/babel/core-7.29.0
dependabot/npm_and_yarn/commander-14.0.3
dependabot/npm_and_yarn/types/node-25.2.0
dependabot/npm_and_yarn/pkgroll-2.23.0
fix/tests
dependabot/npm_and_yarn/openai-6.17.0
dependabot/npm_and_yarn/typescript-eslint-8.54.0
dependabot/npm_and_yarn/node-llama-cpp-3.15.1
dependabot/npm_and_yarn/prettier-3.8.1
dependabot/npm_and_yarn/eslint-9.39.2
dependabot/npm_and_yarn/eslint/js-9.39.2
dependabot/npm_and_yarn/tsx-4.21.0
dependabot/npm_and_yarn/eslint-plugin-unused-imports-4.3.0
dependabot/npm_and_yarn/typescript-5.9.3
dependabot/npm_and_yarn/dotenv-17.2.3
dependabot/npm_and_yarn/ipull-4.0.3
dependabot/npm_and_yarn/google/generative-ai-0.24.1
ollama-support
add/github-azure-integration
v1
v2.2.2
v2.2.1
v2.2.0
v2.1.3
v2.1.2
v2.1.1
v2.1.0
v2.0.3
v2.0.2
v2.0.1
v2.0.0
v1.0
Labels
Clear labels   
bug

   
enhancement

   
pull-request

Mirrored from GitHub Pull Request

  
wontfix
No labels
bug
enhancement
pull-request
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/humanify#25
No description provided.