[PR #770] [MERGED] core: add stdin tokenizer/router #1551

New issue

Closed

opened 2026-03-14 09:42:45 +03:00 by kerem · 0 comments

kerem commented

2026-03-14 09:42:45 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/anomalyco/opentui/pull/770
Author: @simonklee
Created: 3/2/2026
Status: ✅ Merged
Merged: 3/9/2026
Merged by: @kommander

Base: main ← Head: stdin-unified

📝 Commits (10+)

bfe7b0d core: add stdin tokenizer/router
6c4efaf fix overflow and memory model
375e317 labeled switch parser
8426a56 add focus + mouse event test
4f7281a terminal-palette cleanup
5707b64 fix zig stdin timeout parity and overflow recovery
78ce9d7 more memory mng issues
612efed memory ownership clarification
0088a95 remove unused fields
804a6bb fix zig stdin paste parity and high-byte timeout fallback

📊 Changes

24 files changed (+4048 additions, -1648 deletions)

View changed files

📝 packages/core/src/lib/KeyHandler.integration.test.ts (+20 -10)
📝 packages/core/src/lib/KeyHandler.stopPropagation.test.ts (+23 -13)
📝 packages/core/src/lib/KeyHandler.test.ts (+38 -31)
📝 packages/core/src/lib/KeyHandler.ts (+3 -21)
➕ packages/core/src/lib/clock.ts (+21 -0)
📝 packages/core/src/lib/index.ts (+2 -1)
📝 packages/core/src/lib/parse.keypress-kitty.test.ts (+23 -0)
📝 packages/core/src/lib/parse.keypress.test.ts (+51 -1)
📝 packages/core/src/lib/parse.keypress.ts (+22 -6)
📝 packages/core/src/lib/parse.mouse.test.ts (+7 -0)
📝 packages/core/src/lib/parse.mouse.ts (+7 -8)
➖ packages/core/src/lib/stdin-buffer.test.ts (+0 -926)
➖ packages/core/src/lib/stdin-buffer.ts (+0 -423)
➕ packages/core/src/lib/stdin-parser.test.ts (+1660 -0)
➕ packages/core/src/lib/stdin-parser.ts (+1233 -0)
📝 packages/core/src/lib/terminal-palette.test.ts (+33 -0)
📝 packages/core/src/lib/terminal-palette.ts (+128 -103)
📝 packages/core/src/renderables/TabSelect.test.ts (+11 -2)
📝 packages/core/src/renderables/__tests__/Textarea.stress.test.ts (+24 -0)
📝 packages/core/src/renderer.ts (+128 -64)

...and 4 more files

📄 Description

Fixes #748 + related #705, #682, #727, #686, OpenCode 15760

We currently have a chunk-level either/or stdin split in CliRenderer:

if chunk looks like mouse, route mouse and return early
otherwise send the whole chunk to generic stdin buffering
That design is unsafe for mixed chunks and can silently drop input (e.g. mouse + key in same read).
Examples:
\x1b[<64;10;5Mx -> trailing x can be dropped
x\x1b[<64;10;5M -> mouse can be swallowed by non-mouse path

I want to have one canonical stdin pipeline, but i tried to add this a merge-safe way, which is why this PR is a bit larger and more of a refactor than just a straight parser swap.

main uses a split stdin pipeline:

raw bytes are decoded to UTF-8 too early, before the parser stack can understand the protocol
mouse handling is pre-filtered before buffering
KeyHandler reparses strings the parser stack already understands

This makes input harder to reason about and can lose details for mixed protocol chunks, X10 mouse payload bytes, delayed escape continuations, and legacy high-byte/meta cases. For instance, issue 661, i now realize is most likely a case of utf8 decoding happening on X10 mouse bytes

The boundary is:

StdinParser owns byte framing and protocol recognition. It calls parseKeypress() and MouseParser internally and creates fully typed StdinEvent objects (key, mouse, paste, response).
KeyHandler owns event dispatch and turns ParsedKey into KeyEvent. It is not a parser.
Renderer owns application routing only, switching on event.type and forwarding to the right handler.

This removes:

StdinBuffer string tokenizer and its even emitter
setEncoding("utf8") on stdin (decoded strings before framing, wrong order for a binary protocol)
MouseParser instance in renderer that consumed mouse bytes before they ever reached the buffer
parseKeypress() call inside KeyHandler.processInput()
Paste-size coupling to the parser's main byte buffer

Also:

A Clock interface (I used a similar abstraction in my last Go project) to manually advance time in tests without relying on wall clock time. I only used this in the relevant tests, but if we like this we can expand it to other tests that use timers. Side-effect is that it makes timing tests quick and deterministic, which is a nice bonus.
In main we're not testing the stdin ReadStream directly AFAIK, which is why the setEncoding issues have gone unnoticed, i think. I haven't addressed this yet. - Added many tests to the parser :)
Its a large PR, but the lines added is deceptive. The new parser is a larger because we're operating on bytes, but other things are mostly glue + tests + comments.

In one sentence: main decodes first, parses later. In the PR we parse bytes first, decode only when semantics are known.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/anomalyco/opentui/pull/770 **Author:** [@simonklee](https://github.com/simonklee) **Created:** 3/2/2026 **Status:** ✅ Merged **Merged:** 3/9/2026 **Merged by:** [@kommander](https://github.com/kommander) **Base:** `main` ← **Head:** `stdin-unified` --- ### 📝 Commits (10+) - [`bfe7b0d`](https://github.com/anomalyco/opentui/commit/bfe7b0d33a8ded32eebc046acbc298d42949dd21) core: add stdin tokenizer/router - [`6c4efaf`](https://github.com/anomalyco/opentui/commit/6c4efaf446890bf7a855e2d725825ce2b791badb) fix overflow and memory model - [`375e317`](https://github.com/anomalyco/opentui/commit/375e317ac66fb8d606feedab5dd5e80cb07909e7) labeled switch parser - [`8426a56`](https://github.com/anomalyco/opentui/commit/8426a56cc7d90ecad06c9f588954abe2f1791fb1) add focus + mouse event test - [`4f7281a`](https://github.com/anomalyco/opentui/commit/4f7281adfbfd6254d8b7695a2dba96dc0fd0156d) terminal-palette cleanup - [`5707b64`](https://github.com/anomalyco/opentui/commit/5707b647f7eb3012e2c16ce8337bcba3e42e6f7c) fix zig stdin timeout parity and overflow recovery - [`78ce9d7`](https://github.com/anomalyco/opentui/commit/78ce9d74d460344183e11be7dbe5efaa707c1b18) more memory mng issues - [`612efed`](https://github.com/anomalyco/opentui/commit/612efedd407c925122a08354e6c49bc4c34c0365) memory ownership clarification - [`0088a95`](https://github.com/anomalyco/opentui/commit/0088a95e6bc93dcfd3cac3b62626f7b461177a6b) remove unused fields - [`804a6bb`](https://github.com/anomalyco/opentui/commit/804a6bba132ab59434f77e4a5bef95dc689ddf49) fix zig stdin paste parity and high-byte timeout fallback ### 📊 Changes **24 files changed** (+4048 additions, -1648 deletions) <details> <summary>View changed files</summary> 📝 `packages/core/src/lib/KeyHandler.integration.test.ts` (+20 -10) 📝 `packages/core/src/lib/KeyHandler.stopPropagation.test.ts` (+23 -13) 📝 `packages/core/src/lib/KeyHandler.test.ts` (+38 -31) 📝 `packages/core/src/lib/KeyHandler.ts` (+3 -21) ➕ `packages/core/src/lib/clock.ts` (+21 -0) 📝 `packages/core/src/lib/index.ts` (+2 -1) 📝 `packages/core/src/lib/parse.keypress-kitty.test.ts` (+23 -0) 📝 `packages/core/src/lib/parse.keypress.test.ts` (+51 -1) 📝 `packages/core/src/lib/parse.keypress.ts` (+22 -6) 📝 `packages/core/src/lib/parse.mouse.test.ts` (+7 -0) 📝 `packages/core/src/lib/parse.mouse.ts` (+7 -8) ➖ `packages/core/src/lib/stdin-buffer.test.ts` (+0 -926) ➖ `packages/core/src/lib/stdin-buffer.ts` (+0 -423) ➕ `packages/core/src/lib/stdin-parser.test.ts` (+1660 -0) ➕ `packages/core/src/lib/stdin-parser.ts` (+1233 -0) 📝 `packages/core/src/lib/terminal-palette.test.ts` (+33 -0) 📝 `packages/core/src/lib/terminal-palette.ts` (+128 -103) 📝 `packages/core/src/renderables/TabSelect.test.ts` (+11 -2) 📝 `packages/core/src/renderables/__tests__/Textarea.stress.test.ts` (+24 -0) 📝 `packages/core/src/renderer.ts` (+128 -64) _...and 4 more files_ </details> ### 📄 Description Fixes #748 + related #705, #682, #727, #686, [OpenCode 15760](https://github.com/anomalyco/opencode/issues/15760) We currently have a chunk-level either/or stdin split in `CliRenderer`: - if chunk looks like mouse, route mouse and return early - otherwise send the whole chunk to generic stdin buffering That design is unsafe for mixed chunks and can silently drop input (e.g. mouse + key in same read). Examples: - `\x1b[<64;10;5Mx` -> trailing `x` can be dropped - `x\x1b[<64;10;5M` -> mouse can be swallowed by non-mouse path I want to have one canonical stdin pipeline, but i tried to add this a merge-safe way, which is why this PR is a _bit larger_ and more of a refactor than just a straight parser swap. `main` uses a split stdin pipeline: - raw bytes are decoded to UTF-8 too early, before the parser stack can understand the protocol - mouse handling is pre-filtered before buffering - `KeyHandler` reparses strings the parser stack already understands This makes input harder to reason about and can lose details for mixed protocol chunks, X10 mouse payload bytes, delayed escape continuations, and legacy high-byte/meta cases. For instance, [issue 661](https://github.com/anomalyco/opentui/issues/661), i now realize is most likely a case of utf8 decoding happening on X10 mouse bytes The boundary is: - StdinParser owns byte framing and protocol recognition. It calls `parseKeypress()` and `MouseParser` internally and creates fully typed `StdinEvent` objects (key, mouse, paste, response). - KeyHandler owns event dispatch and turns `ParsedKey` into `KeyEvent`. It is not a parser. - Renderer owns application routing only, switching on `event.type` and forwarding to the right handler. This removes: - `StdinBuffer` string tokenizer and its even emitter - `setEncoding("utf8")` on stdin (decoded strings before framing, wrong order for a binary protocol) - `MouseParser` instance in renderer that consumed mouse bytes before they ever reached the buffer - `parseKeypress()` call inside `KeyHandler.processInput()` - Paste-size coupling to the parser's main byte buffer Also: - A Clock interface (I used a similar abstraction in my last Go project) to manually advance time in tests without relying on wall clock time. I only used this in the relevant tests, but if we like this we can expand it to other tests that use timers. Side-effect is that it makes timing tests quick and deterministic, which is a nice bonus. - In main we're not testing the `stdin` ReadStream directly AFAIK, which is why the setEncoding issues have gone unnoticed, i think. I haven't addressed this yet. - Added many tests to the parser :) - Its a large PR, but the lines added is deceptive. The new parser is a larger because we're operating on bytes, but other things are mostly glue + tests + comments. In one sentence: main decodes first, parses later. In the PR we parse bytes first, decode only when semantics are known. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>