[PR #12] [MERGED] feat(proxy): Markdown for Agents — Accept: text/markdown content negotiation #16

New issue

Closed

opened 2026-03-02 05:12:32 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 05:12:32 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/gotempsh/temps/pull/12
Author: @dviejokfs
Created: 2/18/2026
Status: ✅ Merged
Merged: 2/18/2026
Merged by: @dviejokfs

Base: main ← Head: feat/markdown-for-agents

📝 Commits (4)

fceb0d0 feat(proxy): add Accept: text/markdown support for AI agents
45ef12a fix(proxy): fix clippy unnecessary_literal_unwrap in markdown test and update changelog
dd26b9f fix(proxy): extract content before HTML-to-Markdown conversion
a111d54 test(proxy): add pipeline integration tests for markdown edge cases

📊 Changes

4 files changed (+1826 additions, -4 deletions)

View changed files

📝 CHANGELOG.md (+1 -0)
📝 Cargo.lock (+365 -2)
📝 crates/temps-proxy/Cargo.toml (+2 -0)
📝 crates/temps-proxy/src/proxy.rs (+1458 -2)

📄 Description

Summary

Implements Markdown for Agents content negotiation in the Pingora reverse proxy, compatible with Cloudflare's emerging standard for AI-native content delivery.

When an AI agent or HTTP client sends Accept: text/markdown, Temps now converts the upstream HTML response to Markdown on the fly before delivering it — reducing token waste and improving response quality for AI systems consuming deployed apps.

How it works

curl https://your-app.temps.sh/ -H "Accept: text/markdown"
# → Content-Type: text/markdown; charset=utf-8
# → Vary: Accept
# → X-Markdown-Tokens: <estimated count>
# → [Markdown body]

Filter chain

Stage	What happens
`early_request_filter`	Detects `Accept: text/markdown`; sets `ctx.wants_markdown = true`; disables upstream compression
`upstream_response_filter`	Confirms upstream is `text/html` (not SSE/WS); adds `Vary: Accept`; cancels conversion otherwise
`response_filter`	Rewrites `Content-Type → text/markdown; charset=utf-8`; removes `Content-Length` and `Content-Encoding`; adds `X-Markdown-Tokens: 0` placeholder
`response_body_filter`	Buffers chunks; converts HTML → Markdown via `htmd` on `end_of_stream`; falls back to passthrough on error or size limit exceeded

Safety guards

SSE/WebSocket: conversion is cancelled if ctx.is_sse or ctx.is_websocket is set — streaming responses always pass through unchanged
Non-HTML: only text/html upstream responses are converted; JSON, images, etc. pass through as-is
2 MB limit: responses larger than 2 MB fall back to passthrough, mirroring Cloudflare's own constraint
Conversion failure: if htmd fails, the original HTML bytes are returned so the client always gets something

Tests

18 unit tests added in proxy::markdown_tests:

Accept header parsing (exact, quality values, case-insensitive, negative)
Content-type gating (HTML converts, JSON/SSE/WS do not)
Multi-chunk body accumulation
Size guard (>2 MB disables conversion)
HTML→Markdown conversion correctness
SSE passthrough safety
Token estimation

Dependencies

htmd 0.5 — pure Rust HTML→Markdown converter, no C dependencies

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/gotempsh/temps/pull/12 **Author:** [@dviejokfs](https://github.com/dviejokfs) **Created:** 2/18/2026 **Status:** ✅ Merged **Merged:** 2/18/2026 **Merged by:** [@dviejokfs](https://github.com/dviejokfs) **Base:** `main` ← **Head:** `feat/markdown-for-agents` --- ### 📝 Commits (4) - [`fceb0d0`](https://github.com/gotempsh/temps/commit/fceb0d0a9163018a9383431bc2e256a59cfd73c5) feat(proxy): add Accept: text/markdown support for AI agents - [`45ef12a`](https://github.com/gotempsh/temps/commit/45ef12a6629c823b8a8aa2889ffe594d6fc4b11e) fix(proxy): fix clippy unnecessary_literal_unwrap in markdown test and update changelog - [`dd26b9f`](https://github.com/gotempsh/temps/commit/dd26b9fd382b6673d6c70bc69c6fe02b36d31d2d) fix(proxy): extract <main> content before HTML-to-Markdown conversion - [`a111d54`](https://github.com/gotempsh/temps/commit/a111d54625ffd558e4ed43c31050c24747a80415) test(proxy): add pipeline integration tests for markdown edge cases ### 📊 Changes **4 files changed** (+1826 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+1 -0) 📝 `Cargo.lock` (+365 -2) 📝 `crates/temps-proxy/Cargo.toml` (+2 -0) 📝 `crates/temps-proxy/src/proxy.rs` (+1458 -2) </details> ### 📄 Description ## Summary Implements [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) content negotiation in the Pingora reverse proxy, compatible with Cloudflare's emerging standard for AI-native content delivery. When an AI agent or HTTP client sends `Accept: text/markdown`, Temps now converts the upstream HTML response to Markdown on the fly before delivering it — reducing token waste and improving response quality for AI systems consuming deployed apps. ## How it works ``` curl https://your-app.temps.sh/ -H "Accept: text/markdown" # → Content-Type: text/markdown; charset=utf-8 # → Vary: Accept # → X-Markdown-Tokens: <estimated count> # → [Markdown body] ``` ### Filter chain | Stage | What happens | |---|---| | `early_request_filter` | Detects `Accept: text/markdown`; sets `ctx.wants_markdown = true`; disables upstream compression | | `upstream_response_filter` | Confirms upstream is `text/html` (not SSE/WS); adds `Vary: Accept`; cancels conversion otherwise | | `response_filter` | Rewrites `Content-Type → text/markdown; charset=utf-8`; removes `Content-Length` and `Content-Encoding`; adds `X-Markdown-Tokens: 0` placeholder | | `response_body_filter` | Buffers chunks; converts HTML → Markdown via `htmd` on `end_of_stream`; falls back to passthrough on error or size limit exceeded | ### Safety guards - **SSE/WebSocket**: conversion is cancelled if `ctx.is_sse` or `ctx.is_websocket` is set — streaming responses always pass through unchanged - **Non-HTML**: only `text/html` upstream responses are converted; JSON, images, etc. pass through as-is - **2 MB limit**: responses larger than 2 MB fall back to passthrough, mirroring Cloudflare's own constraint - **Conversion failure**: if `htmd` fails, the original HTML bytes are returned so the client always gets something ## Tests 18 unit tests added in `proxy::markdown_tests`: - Accept header parsing (exact, quality values, case-insensitive, negative) - Content-type gating (HTML converts, JSON/SSE/WS do not) - Multi-chunk body accumulation - Size guard (>2 MB disables conversion) - HTML→Markdown conversion correctness - SSE passthrough safety - Token estimation ## Dependencies - [`htmd`](https://crates.io/crates/htmd) `0.5` — pure Rust HTML→Markdown converter, no C dependencies --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>