[PR #625] [MERGED] Reduce indexing chunk size to avoid overflowing sonic's buffer when unicode multi-point characters push char-count past 20,000 #1252

New issue

Closed

opened 2026-03-01 14:49:02 +03:00 by kerem · 0 comments

kerem commented

2026-03-01 14:49:02 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/ArchiveBox/ArchiveBox/pull/625
Author: @jdcaballerov
Created: 1/20/2021
Status: ✅ Merged
Merged: 1/20/2021
Merged by: @pirate

Base: dev ← Head: jdcaballerov-patch-1-1

📝 Commits (1)

14df0cb Update sonic.py

📊 Changes

1 file changed (+1 additions, -1 deletions)

View changed files

📝 archivebox/search/backends/sonic.py (+1 -1)

📄 Description

Sonic buffer accepts 20.000 bytes not unicode characters, since the chunking here is on unicode characters, sending 20.000 characters will overflow sonic's buffer.
UTF-8 can take up to 6 bytes, so sending less than (20.000 / 6) rounded minus should be ok.

Summary

Changes these areas

Bugfixes
Feature behavior
Command line interface
Configuration options
Internal architecture
Snapshot data layout on disk

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/ArchiveBox/ArchiveBox/pull/625 **Author:** [@jdcaballerov](https://github.com/jdcaballerov) **Created:** 1/20/2021 **Status:** ✅ Merged **Merged:** 1/20/2021 **Merged by:** [@pirate](https://github.com/pirate) **Base:** `dev` ← **Head:** `jdcaballerov-patch-1-1` --- ### 📝 Commits (1) - [`14df0cb`](https://github.com/ArchiveBox/ArchiveBox/commit/14df0cbb7c8e948c46a69a398af4ec2e502dc7dd) Update sonic.py ### 📊 Changes **1 file changed** (+1 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `archivebox/search/backends/sonic.py` (+1 -1) </details> ### 📄 Description Sonic buffer accepts 20.000 bytes not unicode characters, since the chunking here is on unicode characters, sending 20.000 characters will overflow sonic's buffer. UTF-8 can take up to 6 bytes, so sending less than (20.000 / 6) rounded minus should be ok.  # Summary  # Related issues  # Changes these areas - [x] Bugfixes - [ ] Feature behavior - [ ] Command line interface - [ ] Configuration options - [ ] Internal architecture - [ ] Snapshot data layout on disk --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>