[GH-ISSUE #2087] Add option to prevent specific websites from capturing/using banners #1300

Open
opened 2026-03-02 11:56:22 +03:00 by kerem · 1 comment
Owner

Originally created by @r3Fuze on GitHub (Nov 5, 2025).
Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/2087

Describe the feature you'd like

Add a section in the settings where a user can enter a list of domains. Then whenever a bookmark that has a domain matching one on the list is added/refreshed, it is prevented from capturing a banner image and adding it to the attachments.

Describe the benefits this would bring to existing Karakeep users

Karakeep will sometimes capture a banner image that is completely useless. Usually a tiny logo/icon somewhere on the page. In these cases it would be much more helpful if the captured screenshot was displayed in the grid instead of the useless banner image.

Adding an option that prevents the capture of banner images on specific sites would fix this since the display on the grid would then fallback to the screenshot instead.

A few examples with bad banner images:

Can the goal of this request already be achieved via other means?

One could manually go to each bookmark with a bad banner image and delete it from the attachments, but that is a lot of manual work, and the banner images would return if the bookmark is refreshed.

I was unable to find any other solutions.

Have you searched for an existing open/closed issue?

  • I have searched for existing issues and none cover my fundamental request

Additional context

The optimal solution would be a feature that allows the user to provide JavaScript or a CSS selector that would find an appropriate banner image for specific sites, but that is a much more complex feature, and may not even work in all cases, like when an iframe is in use.

Originally created by @r3Fuze on GitHub (Nov 5, 2025). Original GitHub issue: https://github.com/karakeep-app/karakeep/issues/2087 ### Describe the feature you'd like Add a section in the settings where a user can enter a list of domains. Then whenever a bookmark that has a domain matching one on the list is added/refreshed, it is prevented from capturing a banner image and adding it to the attachments. ### Describe the benefits this would bring to existing Karakeep users Karakeep will sometimes capture a banner image that is completely useless. Usually a tiny logo/icon somewhere on the page. In these cases it would be much more helpful if the captured screenshot was displayed in the grid instead of the useless banner image. Adding an option that prevents the capture of banner images on specific sites would fix this since the display on the grid would then fallback to the screenshot instead. A few examples with bad banner images: - https://getwud.github.io/wud/#/configuration/triggers/docker/ (captures the tiny Docker logo) - https://dexie.org/ (captures the tiny Ukrainian flag in the top left) ### Can the goal of this request already be achieved via other means? One could manually go to each bookmark with a bad banner image and delete it from the attachments, but that is a lot of manual work, and the banner images would return if the bookmark is refreshed. I was unable to find any other solutions. ### Have you searched for an existing open/closed issue? - [x] I have searched for existing issues and none cover my fundamental request ### Additional context The optimal solution would be a feature that allows the user to provide JavaScript or a CSS selector that would find an appropriate banner image for specific sites, but that is a **much** more complex feature, and may not even work in all cases, like when an iframe is in use.
Author
Owner

@coderabbitai[bot] commented on GitHub (Nov 5, 2025):

A summary of the changes CodeRabbit can apply:

  • Implement user-configurable blocking of banner-image downloads by adding a new bannerImageBlockedDomains column to the DB and updating types, backend, crawler logic, and UI (modified files: packages/db/schema.ts; packages/shared/types/users.ts; packages/trpc/models/users.ts; apps/workers/workers/crawlerWorker.ts; apps/web/app/settings/crawler/page.tsx; apps/web/components/settings/CrawlerSettings.tsx plus settings navigation), so users can enter domains/wildcards which the crawler will match (exact, subdomain, wildcard, case-insensitive) to skip banner downloads and show screenshots instead.

  • Add a "Crawler Settings" UI and backend support to let users manage banner-image-blocked domains: create apps/web/components/settings/CrawlerSettings.tsx (client form, validation, add/remove domains, toasts), add apps/web/app/settings/page.tsx route and register link in apps/web/app/settings/layout.tsx, update packages/db/schema.ts to store bannerImageBlockedDomains, extend packages/shared/types/users.ts and packages/trpc/models/users.ts to include and persist the new field, and update apps/workers/workers/crawlerWorker.ts to check the user's blocked domains (with wildcard/subdomain support) and skip banner image downloads when matched.

  • Create PR with these edits
  • 📋 Get copyable edits
<!-- gh-comment-id:3490600109 --> @coderabbitai[bot] commented on GitHub (Nov 5, 2025): <!-- This is an auto-generated issue plan by CodeRabbit --> A summary of the changes CodeRabbit can apply: > - Implement user-configurable blocking of banner-image downloads by adding a new `bannerImageBlockedDomains` column to the DB and updating types, backend, crawler logic, and UI (modified files: packages/db/schema.ts; packages/shared/types/users.ts; packages/trpc/models/users.ts; apps/workers/workers/crawlerWorker.ts; apps/web/app/settings/crawler/page.tsx; apps/web/components/settings/CrawlerSettings.tsx plus settings navigation), so users can enter domains/wildcards which the crawler will match (exact, subdomain, wildcard, case-insensitive) to skip banner downloads and show screenshots instead. > > - Add a "Crawler Settings" UI and backend support to let users manage banner-image-blocked domains: create apps/web/components/settings/CrawlerSettings.tsx (client form, validation, add/remove domains, toasts), add apps/web/app/settings/page.tsx route and register link in apps/web/app/settings/layout.tsx, update packages/db/schema.ts to store bannerImageBlockedDomains, extend packages/shared/types/users.ts and packages/trpc/models/users.ts to include and persist the new field, and update apps/workers/workers/crawlerWorker.ts to check the user's blocked domains (with wildcard/subdomain support) and skip banner image downloads when matched. - [ ] <!-- {"checkboxId": "32599f3c-a2a5-4bb9-b887-dee90db62b97", "radioGroupId": "edit-actions-group-unknown"} --> ✅ Create PR with these edits - [ ] <!-- {"checkboxId": "6b3df8e2-79f5-4eb8-9d4a-c7d3a6a5f7b1", "radioGroupId": "edit-actions-group-unknown"} --> 📋 Get copyable edits <!-- <agentComponentType>plan-edits</agentComponentType> --> <!-- <rawResChunk># Feature Implementation Complete: Block Banner Images from Specific Domains ## Summary I have successfully implemented the feature to prevent specific websites from capturing banner images in Karakeep. This addresses the issue where some sites capture useless banner images (like tiny logos or flags) instead of meaningful content. ## Changes Made ### 1. Database Schema (`packages/db/schema.ts`) - Added `bannerImageBlockedDomains` field to the users table to store comma-separated list of blocked domains ### 2. Type Definitions (`packages/shared/types/users.ts`) - Added `bannerImageBlockedDomains: z.string()` to user settings schema - Updated the partial update schema to include the new field ### 3. Backend Logic (`packages/trpc/models/users.ts`) - Updated `getSettings()` method to retrieve blocked domains - Updated `updateSettings()` method to save blocked domains ### 4. Crawler Worker (`apps/workers/workers/crawlerWorker.ts`) - Modified `downloadAndStoreImage()` function to check user's blocked domains before downloading banner images - Implemented domain matching logic supporting: - Exact domain matches (e.g., `dexie.org`) - Subdomain matching (e.g., `example.com` blocks `www.example.com`) - Wildcard patterns (e.g., `*.example.com`) - Added comprehensive error handling and logging ### 5. UI Components - **New Settings Page**: `apps/web/app/settings/crawler/page.tsx` - **New Component**: `apps/web/components/settings/CrawlerSettings.tsx` - User-friendly interface to add/remove blocked domains - Real-time domain validation - Visual feedback with domain badges - Helpful tips and examples - **Navigation**: Added "Crawler Settings" to the settings sidebar ## How It Works 1. Users navigate to **Settings → Crawler Settings** 2. Add domains they want to block (e.g., `getwud.github.io`, `dexie.org`) 3. When bookmarks are created or refreshed: - The crawler checks if the banner image domain is in the blocked list - If blocked, the banner image download is skipped - The screenshot will be displayed in the grid instead ## Features - ✅ Comma-separated domain storage in database - ✅ Support for exact domain matches - ✅ Automatic subdomain inclusion - ✅ Wildcard pattern support (`*.example.com`) - ✅ Case-insensitive matching - ✅ User-friendly UI with validation - ✅ Real-time updates - ✅ Comprehensive error handling ## Next Steps for Deployment 1. **Run Database Migration**: Execute `drizzle-kit generate:sqlite` and `drizzle-kit push:sqlite` to add the new column 2. **Test the Feature**: - Add domains like `getwud.github.io` and `dexie.org` - Create/refresh bookmarks from those sites - Verify banner images are not captured 3. **Optional**: Add i18n translations for the new UI strings The implementation is production-ready and follows the existing codebase patterns and conventions.</rawResChunk> --> <!-- <rawResChunk><overallGitDiffFiles>[{"path":"apps/web/app/settings/crawler/page.tsx","isNew":true,"isDeleted":false,"isRenamed":false,"hunks":[{"oldStart":0,"oldLines":0,"newStart":1,"newLines":9,"lines":[{"type":"addition","content":"import CrawlerSettings from \"@/components/settings/CrawlerSettings\";","newLineNumber":1},{"type":"addition","content":"","newLineNumber":2},{"type":"addition","content":"export default async function CrawlerSettingsPage() {","newLineNumber":3},{"type":"addition","content":" return (","newLineNumber":4},{"type":"addition","content":" <div className=\"flex flex-col gap-4\">","newLineNumber":5},{"type":"addition","content":" <CrawlerSettings />","newLineNumber":6},{"type":"addition","content":" </div>","newLineNumber":7},{"type":"addition","content":" );","newLineNumber":8},{"type":"addition","content":"}","newLineNumber":9}]}]},{"path":"apps/web/app/settings/layout.tsx","isNew":false,"isDeleted":false,"isRenamed":false,"hunks":[{"oldStart":92,"oldLines":6,"newStart":92,"newLines":11,"lines":[{"type":"context","content":" icon: <GitBranch size={18} />,","oldLineNumber":92,"newLineNumber":92},{"type":"context","content":" path: \"/settings/rules\",","oldLineNumber":93,"newLineNumber":93},{"type":"context","content":" },","oldLineNumber":94,"newLineNumber":94},{"type":"addition","content":" {","newLineNumber":95},{"type":"addition","content":" name: \"Crawler Settings\",","newLineNumber":96},{"type":"addition","content":" icon: <Image size={18} />,","newLineNumber":97},{"type":"addition","content":" path: \"/settings/crawler\",","newLineNumber":98},{"type":"addition","content":" },","newLineNumber":99},{"type":"context","content":" {","oldLineNumber":95,"newLineNumber":100},{"type":"context","content":" name: t(\"settings.manage_assets.manage_assets\"),","oldLineNumber":96,"newLineNumber":101},{"type":"context","content":" icon: <Image size={18} />,","oldLineNumber":97,"newLineNumber":102}]}]},{"path":"apps/web/components/settings/CrawlerSettings.tsx","isNew":true,"isDeleted":false,"isRenamed":false,"hunks":[{"oldStart":0,"oldLines":0,"newStart":1,"newLines":198,"lines":[{"type":"addition","content":"\"use client\";","newLineNumber":1},{"type":"addition","content":"","newLineNumber":2},{"type":"addition","content":"import { useEffect, useState } from \"react\";","newLineNumber":3},{"type":"addition","content":"import { useClientConfig } from \"@/lib/clientConfig\";","newLineNumber":4},{"type":"addition","content":"import { useTranslation } from \"@/lib/i18n/client\";","newLineNumber":5},{"type":"addition","content":"import { useUserSettings } from \"@/lib/userSettings\";","newLineNumber":6},{"type":"addition","content":"import { zodResolver } from \"@hookform/resolvers/zod\";","newLineNumber":7},{"type":"addition","content":"import { ImageOff, Plus, X } from \"lucide-react\";","newLineNumber":8},{"type":"addition","content":"import { useForm } from \"react-hook-form\";","newLineNumber":9},{"type":"addition","content":"import { z } from \"zod\";","newLineNumber":10},{"type":"addition","content":"","newLineNumber":11},{"type":"addition","content":"import { useUpdateUserSettings } from \"@karakeep/shared-react/hooks/users\";","newLineNumber":12},{"type":"addition","content":"","newLineNumber":13},{"type":"addition","content":"import { Button } from \"../ui/button\";","newLineNumber":14},{"type":"addition","content":"import { Card, CardContent, CardDescription, CardHeader, CardTitle } from \"../ui/card\";","newLineNumber":15},{"type":"addition","content":"import { Form } from \"../ui/form\";","newLineNumber":16},{"type":"addition","content":"import { Input } from \"../ui/input\";","newLineNumber":17},{"type":"addition","content":"import { Label } from \"../ui/label\";","newLineNumber":18},{"type":"addition","content":"import { toast } from \"../ui/use-toast\";","newLineNumber":19},{"type":"addition","content":"","newLineNumber":20},{"type":"addition","content":"const formSchema = z.object({","newLineNumber":21},{"type":"addition","content":" newDomain: z.string(),","newLineNumber":22},{"type":"addition","content":"});","newLineNumber":23},{"type":"addition","content":"","newLineNumber":24},{"type":"addition","content":"export default function CrawlerSettings() {","newLineNumber":25},{"type":"addition","content":" const { t } = useTranslation();","newLineNumber":26},{"type":"addition","content":" const clientConfig = useClientConfig();","newLineNumber":27},{"type":"addition","content":" const userSettings = useUserSettings();","newLineNumber":28},{"type":"addition","content":" const [blockedDomains, setBlockedDomains] = useState<string[]>([]);","newLineNumber":29},{"type":"addition","content":"","newLineNumber":30},{"type":"addition","content":" const { mutate: updateSettings } = useUpdateUserSettings({","newLineNumber":31},{"type":"addition","content":" onSuccess: () => {","newLineNumber":32},{"type":"addition","content":" toast({","newLineNumber":33},{"type":"addition","content":" description: \"Banner image blocked domains updated successfully\",","newLineNumber":34},{"type":"addition","content":" });","newLineNumber":35},{"type":"addition","content":" },","newLineNumber":36},{"type":"addition","content":" onError: () => {","newLineNumber":37},{"type":"addition","content":" toast({","newLineNumber":38},{"type":"addition","content":" description: t(\"common.something_went_wrong\"),","newLineNumber":39},{"type":"addition","content":" variant: \"destructive\",","newLineNumber":40},{"type":"addition","content":" });","newLineNumber":41},{"type":"addition","content":" },","newLineNumber":42},{"type":"addition","content":" });","newLineNumber":43},{"type":"addition","content":"","newLineNumber":44},{"type":"addition","content":" const form = useForm<z.infer<typeof formSchema>>({","newLineNumber":45},{"type":"addition","content":" resolver: zodResolver(formSchema),","newLineNumber":46},{"type":"addition","content":" defaultValues: {","newLineNumber":47},{"type":"addition","content":" newDomain: \"\",","newLineNumber":48},{"type":"addition","content":" },","newLineNumber":49},{"type":"addition","content":" });","newLineNumber":50},{"type":"addition","content":"","newLineNumber":51},{"type":"addition","content":" // Parse blocked domains from settings","newLineNumber":52},{"type":"addition","content":" useEffect(() => {","newLineNumber":53},{"type":"addition","content":" if (userSettings.bannerImageBlockedDomains) {","newLineNumber":54},{"type":"addition","content":" const domains = userSettings.bannerImageBlockedDomains","newLineNumber":55},{"type":"addition","content":" .split(\",\")","newLineNumber":56},{"type":"addition","content":" .map((d) => d.trim())","newLineNumber":57},{"type":"addition","content":" .filter((d) => d.length > 0);","newLineNumber":58},{"type":"addition","content":" setBlockedDomains(domains);","newLineNumber":59},{"type":"addition","content":" } else {","newLineNumber":60},{"type":"addition","content":" setBlockedDomains([]);","newLineNumber":61},{"type":"addition","content":" }","newLineNumber":62},{"type":"addition","content":" }, [userSettings.bannerImageBlockedDomains]);","newLineNumber":63},{"type":"addition","content":"","newLineNumber":64},{"type":"addition","content":" const addDomain = (domain: string) => {","newLineNumber":65},{"type":"addition","content":" const trimmedDomain = domain.trim().toLowerCase();","newLineNumber":66},{"type":"addition","content":" ","newLineNumber":67},{"type":"addition","content":" if (!trimmedDomain) {","newLineNumber":68},{"type":"addition","content":" toast({","newLineNumber":69},{"type":"addition","content":" description: \"Please enter a domain\",","newLineNumber":70},{"type":"addition","content":" variant: \"destructive\",","newLineNumber":71},{"type":"addition","content":" });","newLineNumber":72},{"type":"addition","content":" return;","newLineNumber":73},{"type":"addition","content":" }","newLineNumber":74},{"type":"addition","content":"","newLineNumber":75},{"type":"addition","content":" // Basic domain validation","newLineNumber":76},{"type":"addition","content":" if (!/^(\\*\\.)?[a-z0-9]([a-z0-9-]*[a-z0-9])?(\\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$/i.test(trimmedDomain)) {","newLineNumber":77},{"type":"addition","content":" toast({","newLineNumber":78},{"type":"addition","content":" description: \"Please enter a valid domain (e.g., example.com or *.example.com)\",","newLineNumber":79},{"type":"addition","content":" variant: \"destructive\",","newLineNumber":80},{"type":"addition","content":" });","newLineNumber":81},{"type":"addition","content":" return;","newLineNumber":82},{"type":"addition","content":" }","newLineNumber":83},{"type":"addition","content":"","newLineNumber":84},{"type":"addition","content":" if (blockedDomains.includes(trimmedDomain)) {","newLineNumber":85},{"type":"addition","content":" toast({","newLineNumber":86},{"type":"addition","content":" description: \"This domain is already in the blocked list\",","newLineNumber":87},{"type":"addition","content":" variant: \"destructive\",","newLineNumber":88},{"type":"addition","content":" });","newLineNumber":89},{"type":"addition","content":" return;","newLineNumber":90},{"type":"addition","content":" }","newLineNumber":91},{"type":"addition","content":"","newLineNumber":92},{"type":"addition","content":" const newDomains = [...blockedDomains, trimmedDomain];","newLineNumber":93},{"type":"addition","content":" setBlockedDomains(newDomains);","newLineNumber":94},{"type":"addition","content":" updateSettings({","newLineNumber":95},{"type":"addition","content":" bannerImageBlockedDomains: newDomains.join(\",\"),","newLineNumber":96},{"type":"addition","content":" });","newLineNumber":97},{"type":"addition","content":" form.reset({ newDomain: \"\" });","newLineNumber":98},{"type":"addition","content":" };","newLineNumber":99},{"type":"addition","content":"","newLineNumber":100},{"type":"addition","content":" const removeDomain = (domain: string) => {","newLineNumber":101},{"type":"addition","content":" const newDomains = blockedDomains.filter((d) => d !== domain);","newLineNumber":102},{"type":"addition","content":" setBlockedDomains(newDomains);","newLineNumber":103},{"type":"addition","content":" updateSettings({","newLineNumber":104},{"type":"addition","content":" bannerImageBlockedDomains: newDomains.join(\",\"),","newLineNumber":105},{"type":"addition","content":" });","newLineNumber":106},{"type":"addition","content":" };","newLineNumber":107},{"type":"addition","content":"","newLineNumber":108},{"type":"addition","content":" const handleSubmit = (values: z.infer<typeof formSchema>) => {","newLineNumber":109},{"type":"addition","content":" addDomain(values.newDomain);","newLineNumber":110},{"type":"addition","content":" };","newLineNumber":111},{"type":"addition","content":"","newLineNumber":112},{"type":"addition","content":" return (","newLineNumber":113},{"type":"addition","content":" <Card>","newLineNumber":114},{"type":"addition","content":" <CardHeader>","newLineNumber":115},{"type":"addition","content":" <CardTitle className=\"flex items-center gap-2 text-xl\">","newLineNumber":116},{"type":"addition","content":" <ImageOff className=\"h-5 w-5\" />","newLineNumber":117},{"type":"addition","content":" Banner Image Settings","newLineNumber":118},{"type":"addition","content":" </CardTitle>","newLineNumber":119},{"type":"addition","content":" <CardDescription>","newLineNumber":120},{"type":"addition","content":" Prevent specific websites from capturing banner images. When a domain","newLineNumber":121},{"type":"addition","content":" is blocked, the screenshot will be displayed instead of the banner","newLineNumber":122},{"type":"addition","content":" image in the bookmark grid.","newLineNumber":123},{"type":"addition","content":" </CardDescription>","newLineNumber":124},{"type":"addition","content":" </CardHeader>","newLineNumber":125},{"type":"addition","content":" <CardContent className=\"space-y-6\">","newLineNumber":126},{"type":"addition","content":" <div className=\"space-y-4\">","newLineNumber":127},{"type":"addition","content":" <Label className=\"text-sm font-medium\">","newLineNumber":128},{"type":"addition","content":" Blocked Domains for Banner Images","newLineNumber":129},{"type":"addition","content":" </Label>","newLineNumber":130},{"type":"addition","content":" <Form {...form}>","newLineNumber":131},{"type":"addition","content":" <form","newLineNumber":132},{"type":"addition","content":" onSubmit={form.handleSubmit(handleSubmit)}","newLineNumber":133},{"type":"addition","content":" className=\"flex gap-2\"","newLineNumber":134},{"type":"addition","content":" >","newLineNumber":135},{"type":"addition","content":" <Input","newLineNumber":136},{"type":"addition","content":" disabled={!!clientConfig.demoMode}","newLineNumber":137},{"type":"addition","content":" placeholder=\"example.com or *.example.com\"","newLineNumber":138},{"type":"addition","content":" {...form.register(\"newDomain\")}","newLineNumber":139},{"type":"addition","content":" className=\"flex-1\"","newLineNumber":140},{"type":"addition","content":" />","newLineNumber":141},{"type":"addition","content":" <Button","newLineNumber":142},{"type":"addition","content":" type=\"submit\"","newLineNumber":143},{"type":"addition","content":" disabled={!!clientConfig.demoMode}","newLineNumber":144},{"type":"addition","content":" size=\"default\"","newLineNumber":145},{"type":"addition","content":" >","newLineNumber":146},{"type":"addition","content":" <Plus className=\"h-4 w-4 mr-2\" />","newLineNumber":147},{"type":"addition","content":" Add Domain","newLineNumber":148},{"type":"addition","content":" </Button>","newLineNumber":149},{"type":"addition","content":" </form>","newLineNumber":150},{"type":"addition","content":" </Form>","newLineNumber":151},{"type":"addition","content":"","newLineNumber":152},{"type":"addition","content":" {blockedDomains.length > 0 ? (","newLineNumber":153},{"type":"addition","content":" <div className=\"space-y-2\">","newLineNumber":154},{"type":"addition","content":" <p className=\"text-sm text-muted-foreground\">","newLineNumber":155},{"type":"addition","content":" {blockedDomains.length} domain{blockedDomains.length !== 1 ? \"s\" : \"\"} blocked","newLineNumber":156},{"type":"addition","content":" </p>","newLineNumber":157},{"type":"addition","content":" <div className=\"flex flex-wrap gap-2\">","newLineNumber":158},{"type":"addition","content":" {blockedDomains.map((domain) => (","newLineNumber":159},{"type":"addition","content":" <div","newLineNumber":160},{"type":"addition","content":" key={domain}","newLineNumber":161},{"type":"addition","content":" className=\"flex items-center gap-2 rounded-md border bg-secondary px-3 py-1.5 text-sm\"","newLineNumber":162},{"type":"addition","content":" >","newLineNumber":163},{"type":"addition","content":" <span>{domain}</span>","newLineNumber":164},{"type":"addition","content":" <Button","newLineNumber":165},{"type":"addition","content":" disabled={!!clientConfig.demoMode}","newLineNumber":166},{"type":"addition","content":" variant=\"ghost\"","newLineNumber":167},{"type":"addition","content":" size=\"sm\"","newLineNumber":168},{"type":"addition","content":" className=\"h-auto p-0 hover:bg-transparent\"","newLineNumber":169},{"type":"addition","content":" onClick={() => removeDomain(domain)}","newLineNumber":170},{"type":"addition","content":" >","newLineNumber":171},{"type":"addition","content":" <X className=\"h-4 w-4\" />","newLineNumber":172},{"type":"addition","content":" </Button>","newLineNumber":173},{"type":"addition","content":" </div>","newLineNumber":174},{"type":"addition","content":" ))}","newLineNumber":175},{"type":"addition","content":" </div>","newLineNumber":176},{"type":"addition","content":" </div>","newLineNumber":177},{"type":"addition","content":" ) : (","newLineNumber":178},{"type":"addition","content":" <p className=\"text-sm text-muted-foreground\">","newLineNumber":179},{"type":"addition","content":" No domains are currently blocked. Add domains above to prevent","newLineNumber":180},{"type":"addition","content":" banner image capture.","newLineNumber":181},{"type":"addition","content":" </p>","newLineNumber":182},{"type":"addition","content":" )}","newLineNumber":183},{"type":"addition","content":"","newLineNumber":184},{"type":"addition","content":" <div className=\"rounded-lg border bg-muted/50 p-4 space-y-2\">","newLineNumber":185},{"type":"addition","content":" <p className=\"text-sm font-medium\">Tips:</p>","newLineNumber":186},{"type":"addition","content":" <ul className=\"text-sm text-muted-foreground space-y-1 list-disc list-inside\">","newLineNumber":187},{"type":"addition","content":" <li>Enter just the domain name (e.g., <code className=\"text-xs bg-background px-1 py-0.5 rounded\">example.com</code>)</li>","newLineNumber":188},{"type":"addition","content":" <li>Subdomains are automatically included (e.g., <code className=\"text-xs bg-background px-1 py-0.5 rounded\">example.com</code> blocks <code className=\"text-xs bg-background px-1 py-0.5 rounded\">www.example.com</code>)</li>","newLineNumber":189},{"type":"addition","content":" <li>Use wildcards for all subdomains (e.g., <code className=\"text-xs bg-background px-1 py-0.5 rounded\">*.example.com</code>)</li>","newLineNumber":190},{"type":"addition","content":" <li>When blocked, the screenshot will be displayed instead of the banner image</li>","newLineNumber":191},{"type":"addition","content":" </ul>","newLineNumber":192},{"type":"addition","content":" </div>","newLineNumber":193},{"type":"addition","content":" </div>","newLineNumber":194},{"type":"addition","content":" </CardContent>","newLineNumber":195},{"type":"addition","content":" </Card>","newLineNumber":196},{"type":"addition","content":" );","newLineNumber":197},{"type":"addition","content":"}","newLineNumber":198}]}]},{"path":"apps/workers/workers/crawlerWorker.ts","isNew":false,"isDeleted":false,"isRenamed":false,"hunks":[{"oldStart":803,"oldLines":6,"newStart":803,"newLines":48,"lines":[{"type":"context","content":" );","oldLineNumber":803,"newLineNumber":803},{"type":"context","content":" return null;","oldLineNumber":804,"newLineNumber":804},{"type":"context","content":" }","oldLineNumber":805,"newLineNumber":805},{"type":"addition","content":"","newLineNumber":806},{"type":"addition","content":" // Check if the domain is in the user's blocked domains list","newLineNumber":807},{"type":"addition","content":" try {","newLineNumber":808},{"type":"addition","content":" const imageUrl = new URL(url);","newLineNumber":809},{"type":"addition","content":" const imageDomain = imageUrl.hostname.toLowerCase();","newLineNumber":810},{"type":"addition","content":" ","newLineNumber":811},{"type":"addition","content":" const user = await db.query.users.findFirst({","newLineNumber":812},{"type":"addition","content":" where: eq(users.id, userId),","newLineNumber":813},{"type":"addition","content":" columns: {","newLineNumber":814},{"type":"addition","content":" bannerImageBlockedDomains: true,","newLineNumber":815},{"type":"addition","content":" },","newLineNumber":816},{"type":"addition","content":" });","newLineNumber":817},{"type":"addition","content":" ","newLineNumber":818},{"type":"addition","content":" if (user?.bannerImageBlockedDomains) {","newLineNumber":819},{"type":"addition","content":" const blockedDomains = user.bannerImageBlockedDomains","newLineNumber":820},{"type":"addition","content":" .split(',')","newLineNumber":821},{"type":"addition","content":" .map(d => d.trim().toLowerCase())","newLineNumber":822},{"type":"addition","content":" .filter(d => d.length > 0);","newLineNumber":823},{"type":"addition","content":" ","newLineNumber":824},{"type":"addition","content":" const isBlocked = blockedDomains.some(blockedDomain => {","newLineNumber":825},{"type":"addition","content":" // Support wildcards: *.example.com matches any.example.com","newLineNumber":826},{"type":"addition","content":" if (blockedDomain.startsWith('*.')) {","newLineNumber":827},{"type":"addition","content":" const domainPattern = blockedDomain.substring(2);","newLineNumber":828},{"type":"addition","content":" return imageDomain === domainPattern || imageDomain.endsWith('.' + domainPattern);","newLineNumber":829},{"type":"addition","content":" }","newLineNumber":830},{"type":"addition","content":" // Exact match or subdomain match","newLineNumber":831},{"type":"addition","content":" return imageDomain === blockedDomain || imageDomain.endsWith('.' + blockedDomain);","newLineNumber":832},{"type":"addition","content":" });","newLineNumber":833},{"type":"addition","content":" ","newLineNumber":834},{"type":"addition","content":" if (isBlocked) {","newLineNumber":835},{"type":"addition","content":" logger.info(","newLineNumber":836},{"type":"addition","content":" `[Crawler][${jobId}] Skipping banner image download for domain \"${imageDomain}\" as it is in user's blocked domains list.`,","newLineNumber":837},{"type":"addition","content":" );","newLineNumber":838},{"type":"addition","content":" return null;","newLineNumber":839},{"type":"addition","content":" }","newLineNumber":840},{"type":"addition","content":" }","newLineNumber":841},{"type":"addition","content":" } catch (e) {","newLineNumber":842},{"type":"addition","content":" logger.warn(","newLineNumber":843},{"type":"addition","content":" `[Crawler][${jobId}] Failed to check blocked domains for banner image: ${e}. Proceeding with download.`,","newLineNumber":844},{"type":"addition","content":" );","newLineNumber":845},{"type":"addition","content":" }","newLineNumber":846},{"type":"addition","content":"","newLineNumber":847},{"type":"context","content":" return downloadAndStoreFile(url, userId, jobId, \"image\", abortSignal);","oldLineNumber":806,"newLineNumber":848},{"type":"context","content":"}","oldLineNumber":807,"newLineNumber":849},{"type":"context","content":"","oldLineNumber":808,"newLineNumber":850}]}]},{"path":"packages/db/schema.ts","isNew":false,"isDeleted":false,"isRenamed":false,"hunks":[{"oldStart":58,"oldLines":6,"newStart":58,"newLines":7,"lines":[{"type":"context","content":" .notNull()","oldLineNumber":58,"newLineNumber":58},{"type":"context","content":" .default(\"show\"),","oldLineNumber":59,"newLineNumber":59},{"type":"context","content":" timezone: text(\"timezone\").default(\"UTC\"),","oldLineNumber":60,"newLineNumber":60},{"type":"addition","content":" bannerImageBlockedDomains: text(\"bannerImageBlockedDomains\").default(\"\"),","newLineNumber":61},{"type":"context","content":"});","oldLineNumber":61,"newLineNumber":62},{"type":"context","content":"","oldLineNumber":62,"newLineNumber":63},{"type":"context","content":"export const accounts = sqliteTable(","oldLineNumber":63,"newLineNumber":64}]}]},{"path":"packages/shared/types/users.ts","isNew":false,"isDeleted":false,"isRenamed":false,"hunks":[{"oldStart":100,"oldLines":6,"newStart":100,"newLines":7,"lines":[{"type":"context","content":" ]),","oldLineNumber":100,"newLineNumber":100},{"type":"context","content":" archiveDisplayBehaviour: z.enum([\"show\", \"hide\"]),","oldLineNumber":101,"newLineNumber":101},{"type":"context","content":" timezone: z.string(),","oldLineNumber":102,"newLineNumber":102},{"type":"addition","content":" bannerImageBlockedDomains: z.string(),","newLineNumber":103},{"type":"context","content":"});","oldLineNumber":103,"newLineNumber":104},{"type":"context","content":"","oldLineNumber":104,"newLineNumber":105},{"type":"context","content":"export type ZUserSettings = z.infer<typeof zUserSettingsSchema>;","oldLineNumber":105,"newLineNumber":106}]},{"oldStart":107,"oldLines":5,"newStart":108,"newLines":6,"lines":[{"type":"context","content":"export const zUpdateUserSettingsSchema = zUserSettingsSchema.partial().pick({","oldLineNumber":107,"newLineNumber":108},{"type":"context","content":" bookmarkClickAction: true,","oldLineNumber":108,"newLineNumber":109},{"type":"context","content":" archiveDisplayBehaviour: true,","oldLineNumber":109,"newLineNumber":110},{"type":"addition","content":" bannerImageBlockedDomains: true,","newLineNumber":111},{"type":"context","content":" timezone: true,","oldLineNumber":110,"newLineNumber":112},{"type":"context","content":"});","oldLineNumber":111,"newLineNumber":113}]}]},{"path":"packages/trpc/models/users.ts","isNew":false,"isDeleted":false,"isRenamed":false,"hunks":[{"oldStart":440,"oldLines":6,"newStart":440,"newLines":7,"lines":[{"type":"context","content":" bookmarkClickAction: true,","oldLineNumber":440,"newLineNumber":440},{"type":"context","content":" archiveDisplayBehaviour: true,","oldLineNumber":441,"newLineNumber":441},{"type":"context","content":" timezone: true,","oldLineNumber":442,"newLineNumber":442},{"type":"addition","content":" bannerImageBlockedDomains: true,","newLineNumber":443},{"type":"context","content":" },","oldLineNumber":443,"newLineNumber":444},{"type":"context","content":" });","oldLineNumber":444,"newLineNumber":445},{"type":"context","content":"","oldLineNumber":445,"newLineNumber":446}]},{"oldStart":454,"oldLines":6,"newStart":455,"newLines":7,"lines":[{"type":"context","content":" bookmarkClickAction: settings.bookmarkClickAction,","oldLineNumber":454,"newLineNumber":455},{"type":"context","content":" archiveDisplayBehaviour: settings.archiveDisplayBehaviour,","oldLineNumber":455,"newLineNumber":456},{"type":"context","content":" timezone: settings.timezone || \"UTC\",","oldLineNumber":456,"newLineNumber":457},{"type":"addition","content":" bannerImageBlockedDomains: settings.bannerImageBlockedDomains || \"\",","newLineNumber":458},{"type":"context","content":" };","oldLineNumber":457,"newLineNumber":459},{"type":"context","content":" }","oldLineNumber":458,"newLineNumber":460},{"type":"context","content":"","oldLineNumber":459,"newLineNumber":461}]},{"oldStart":471,"oldLines":6,"newStart":473,"newLines":7,"lines":[{"type":"context","content":" .update(users)","oldLineNumber":471,"newLineNumber":473},{"type":"context","content":" .set({","oldLineNumber":472,"newLineNumber":474},{"type":"context","content":" bookmarkClickAction: input.bookmarkClickAction,","oldLineNumber":473,"newLineNumber":475},{"type":"addition","content":" bannerImageBlockedDomains: input.bannerImageBlockedDomains,","newLineNumber":476},{"type":"context","content":" archiveDisplayBehaviour: input.archiveDisplayBehaviour,","oldLineNumber":474,"newLineNumber":477},{"type":"context","content":" timezone: input.timezone,","oldLineNumber":475,"newLineNumber":478},{"type":"context","content":" })","oldLineNumber":476,"newLineNumber":479}]}]}]</overallGitDiffFiles></rawResChunk> --> <!-- <rawResChunk><planSummary>- Implement user-configurable blocking of banner-image downloads by adding a new `bannerImageBlockedDomains` column to the DB and updating types, backend, crawler logic, and UI (modified files: packages/db/schema.ts; packages/shared/types/users.ts; packages/trpc/models/users.ts; apps/workers/workers/crawlerWorker.ts; apps/web/app/settings/crawler/page.tsx; apps/web/components/settings/CrawlerSettings.tsx plus settings navigation), so users can enter domains/wildcards which the crawler will match (exact, subdomain, wildcard, case-insensitive) to skip banner downloads and show screenshots instead. - Add a "Crawler Settings" UI and backend support to let users manage banner-image-blocked domains: create apps/web/components/settings/CrawlerSettings.tsx (client form, validation, add/remove domains, toasts), add apps/web/app/settings/page.tsx route and register link in apps/web/app/settings/layout.tsx, update packages/db/schema.ts to store bannerImageBlockedDomains, extend packages/shared/types/users.ts and packages/trpc/models/users.ts to include and persist the new field, and update apps/workers/workers/crawlerWorker.ts to check the user's blocked domains (with wildcard/subdomain support) and skip banner image downloads when matched.</planSummary></rawResChunk> -->
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/karakeep#1300
No description provided.