[PR #1558] Resolve "Failed to determine the content-type for the url" error #1857

New issue

Open

opened 2026-03-02 11:59:29 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 11:59:29 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/karakeep-app/karakeep/pull/1558
Author: @graealex
Created: 6/7/2025
Status: 🔄 Open

Base: main ← Head: main

📝 Commits (2)

014bd90 Update metascraper to 5.47.1 to avoid problems with metascraper-logo, remove metascraper-readability since metadata is never used
f1e7ef1 Add proper request headers to HTTP(S) requests to node-fetch

📊 Changes

3 files changed (+484 additions, -320 deletions)

View changed files

📝 apps/workers/package.json (+13 -13)
📝 apps/workers/workers/crawlerWorker.ts (+22 -8)
📝 pnpm-lock.yaml (+449 -299)

📄 Description

This PR adds reasonable HTTP(S) headers to requests done by node-fetch and metascraper/got, it solves issue #1518

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/karakeep-app/karakeep/pull/1558 **Author:** [@graealex](https://github.com/graealex) **Created:** 6/7/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (2) - [`014bd90`](https://github.com/karakeep-app/karakeep/commit/014bd90fb3605d9ff8f2131ce1ffec2e648afe48) Update metascraper to 5.47.1 to avoid problems with metascraper-logo, remove metascraper-readability since metadata is never used - [`f1e7ef1`](https://github.com/karakeep-app/karakeep/commit/f1e7ef130046a1d033dde90127e77c38f7064b9c) Add proper request headers to HTTP(S) requests to node-fetch ### 📊 Changes **3 files changed** (+484 additions, -320 deletions) <details> <summary>View changed files</summary> 📝 `apps/workers/package.json` (+13 -13) 📝 `apps/workers/workers/crawlerWorker.ts` (+22 -8) 📝 `pnpm-lock.yaml` (+449 -299) </details> ### 📄 Description This PR adds reasonable HTTP(S) headers to requests done by node-fetch and metascraper/got, it solves issue #1518 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem added the

pull-request

label

2026-03-02 11:59:29 +03:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/karakeep#1857

No description provided.

Rows
Columns