[GH-ISSUE #1193] Bug: Search sometimes shows the same snapshot twice

kerem commented

2026-03-01 14:45:59 +03:00

Owner

Originally created by @melyux on GitHub (Jul 26, 2023).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1193

Describe the bug

When doing a search (sonic), sometimes the same snapshot will show up in two rows right after one another.

Steps to reproduce

Not quite sure.

Screenshots or log output

N/A

ArchiveBox version

0.6.3
ArchiveBox v0.6.3 40ddd33 Cpython Linux Linux-6.1.0-10-amd64-x86_64-with-glibc2.31 x86_64
DEBUG=False IN_DOCKER=True IS_TTY=True TZ=UTC FS_ATOMIC=True FS_REMOTE=False FS_PERMS=644 1000:1000 SEARCH_BACKEND=ripgrep

[i] Dependency versions:
 √  PYTHON_BINARY         v3.11.4         valid     /usr/local/bin/python3.11                                                   
 √  SQLITE_BINARY         v2.6.0          valid     /usr/local/lib/python3.11/sqlite3/dbapi2.py                                 
 √  DJANGO_BINARY         v3.1.14         valid     /usr/local/lib/python3.11/site-packages/django/__init__.py                  
 √  ARCHIVEBOX_BINARY     v0.6.3          valid     /usr/local/bin/archivebox                                                   

 √  CURL_BINARY           v7.74.0         valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.21           valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v18.16.1        valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v1.0.44         valid     /usr/lib/node_modules/single-file-cli/single-file                           
 √  READABILITY_BINARY    v0.0.2          valid     /node/node_modules/readability-extractor/readability-extractor              
 √  MERCURY_BINARY        v1.0.0          valid     /node/node_modules/@postlight/mercury-parser/cli.js                         
 -  GIT_BINARY            -               disabled  /usr/bin/git                                                                
 √  YOUTUBEDL_BINARY      v2023.07.06     valid     /usr/local/bin/yt-dlp                                                       
 √  CHROME_BINARY         v114.0.5735.198  valid     /usr/bin/chromium                                                           
 √  RIPGREP_BINARY        v12.1.1         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /app/archivebox                                                             
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates                                                   
 -  CUSTOM_TEMPLATES_DIR  -               disabled                                                                              

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled                                                                              
 -  COOKIES_FILE          -               disabled                                                                              


[i] Data locations:

Originally created by @melyux on GitHub (Jul 26, 2023). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1193  #### Describe the bug  When doing a search (sonic), sometimes the same snapshot will show up in two rows right after one another. #### Steps to reproduce  Not quite sure. #### Screenshots or log output  N/A #### ArchiveBox version  ```logs 0.6.3 ArchiveBox v0.6.3 40ddd33 Cpython Linux Linux-6.1.0-10-amd64-x86_64-with-glibc2.31 x86_64 DEBUG=False IN_DOCKER=True IS_TTY=True TZ=UTC FS_ATOMIC=True FS_REMOTE=False FS_PERMS=644 1000:1000 SEARCH_BACKEND=ripgrep [i] Dependency versions: √ PYTHON_BINARY v3.11.4 valid /usr/local/bin/python3.11 √ SQLITE_BINARY v2.6.0 valid /usr/local/lib/python3.11/sqlite3/dbapi2.py √ DJANGO_BINARY v3.1.14 valid /usr/local/lib/python3.11/site-packages/django/__init__.py √ ARCHIVEBOX_BINARY v0.6.3 valid /usr/local/bin/archivebox √ CURL_BINARY v7.74.0 valid /usr/bin/curl √ WGET_BINARY v1.21 valid /usr/bin/wget √ NODE_BINARY v18.16.1 valid /usr/bin/node √ SINGLEFILE_BINARY v1.0.44 valid /usr/lib/node_modules/single-file-cli/single-file √ READABILITY_BINARY v0.0.2 valid /node/node_modules/readability-extractor/readability-extractor √ MERCURY_BINARY v1.0.0 valid /node/node_modules/@postlight/mercury-parser/cli.js - GIT_BINARY - disabled /usr/bin/git √ YOUTUBEDL_BINARY v2023.07.06 valid /usr/local/bin/yt-dlp √ CHROME_BINARY v114.0.5735.198 valid /usr/bin/chromium √ RIPGREP_BINARY v12.1.1 valid /usr/bin/rg [i] Source-code locations: √ PACKAGE_DIR 23 files valid /app/archivebox √ TEMPLATES_DIR 3 files valid /app/archivebox/templates - CUSTOM_TEMPLATES_DIR - disabled [i] Secrets locations: - CHROME_USER_DATA_DIR - disabled - COOKIES_FILE - disabled [i] Data locations: ```

kerem

2026-03-01 14:45:59 +03:00

closed this issue
added the
why: functionality

expected: next release

status: backlog

size: easy

good first ticket

help wanted
labels

kerem commented

2026-03-01 14:46:00 +03:00

Author

Owner

@pirate commented on GitHub (Jul 28, 2023):

Thanks for reporting.

I'm not entirely surprised by this given how it works. We augment the default Django search of the db fields with the Sonic results, so it's possible the dedupe step is failing which leads to results showing twice if both their content and metadata match.

PR's welcome, otherwise I'll probably get to this in the next bug fixing passes after 0.7.0 is released.

@pirate commented on GitHub (Jul 28, 2023): Thanks for reporting. I'm not entirely surprised by this given how it works. We augment the default Django search of the db fields with the Sonic results, so it's possible the dedupe step is failing which leads to results showing twice if both their content and metadata match. PR's welcome, otherwise I'll probably get to this in the next bug fixing passes after 0.7.0 is released.

kerem commented

2026-03-01 14:46:00 +03:00

Author

Owner

@neel-suthar commented on GitHub (Jan 18, 2024):

@pirate You mean handling dupes in the following function right? We just need to make sure it returns a distinct query set.

def get_queryset(self, **kwargs):
        qs = super().get_queryset(**kwargs)
        query = self.request.GET.get('q')
        if query and query.strip():
            qs = qs.filter(Q(title__icontains=query) | Q(url__icontains=query) | Q(timestamp__icontains=query) | Q(tags__name__icontains=query))
            try:
                qs = qs | query_search_index(query)
            except Exception as err:
                print(f'[!] Error while using search backend: {err.__class__.__name__} {err}')
        return qs

Will be happy to work on this. Please let me know.

@neel-suthar commented on GitHub (Jan 18, 2024): @pirate You mean handling dupes in the following function right? We just need to make sure it returns a distinct query set. ``` def get_queryset(self, **kwargs): qs = super().get_queryset(**kwargs) query = self.request.GET.get('q') if query and query.strip(): qs = qs.filter(Q(title__icontains=query) | Q(url__icontains=query) | Q(timestamp__icontains=query) | Q(tags__name__icontains=query)) try: qs = qs | query_search_index(query) except Exception as err: print(f'[!] Error while using search backend: {err.__class__.__name__} {err}') return qs ``` Will be happy to work on this. Please let me know.

kerem commented

2026-03-01 14:46:00 +03:00

Author

Owner

@pirate commented on GitHub (Jan 19, 2024):

Yeah, that's the spot! If you wanna open a PR to change it to return qs.distinct() I'll approve + merge it into dev (0.7.3-rc). :)

@pirate commented on GitHub (Jan 19, 2024): Yeah, that's the spot! If you wanna open a PR to change it to `return qs.distinct()` I'll approve + merge it into `dev` (0.7.3-rc). :)

kerem commented

2026-03-01 14:46:00 +03:00

Author

Owner

@neel-suthar commented on GitHub (Jan 19, 2024):

@pirate I found another place where we have the same kind of logic but I am not sure if it requires a distinct call or not. Can you please confirm? Here is the code...

from django.contrib import messages

from archivebox.search import query_search_index

class SearchResultsAdminMixin:
    def get_search_results(self, request, queryset, search_term: str):
        """Enhances the search queryset with results from the search backend"""
        
        qs, use_distinct = super().get_search_results(request, queryset, search_term)

        search_term = search_term.strip()
        if not search_term:
            return qs, use_distinct
        try:
            qsearch = query_search_index(search_term)
            qs = qs | qsearch
        except Exception as err:
            print(f'[!] Error while using search backend: {err.__class__.__name__} {err}')
            messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}')
        
        return qs, use_distinct

@neel-suthar commented on GitHub (Jan 19, 2024): @pirate I found another place where we have the same kind of logic but I am not sure if it requires a distinct call or not. Can you please confirm? Here is the code... ``` from django.contrib import messages from archivebox.search import query_search_index class SearchResultsAdminMixin: def get_search_results(self, request, queryset, search_term: str): """Enhances the search queryset with results from the search backend""" qs, use_distinct = super().get_search_results(request, queryset, search_term) search_term = search_term.strip() if not search_term: return qs, use_distinct try: qsearch = query_search_index(search_term) qs = qs | qsearch except Exception as err: print(f'[!] Error while using search backend: {err.__class__.__name__} {err}') messages.add_message(request, messages.WARNING, f'Error from the search backend, only showing results from default admin search fields - Error: {err}') return qs, use_distinct ```

kerem commented

2026-03-01 14:46:00 +03:00

Author

Owner

@pirate commented on GitHub (Jan 19, 2024):

good catch @neel-suthar, want to also add return qs.distinct(), use_distinct there? sorry I saw this after merging your first PR already so you have to open another one

@pirate commented on GitHub (Jan 19, 2024): good catch @neel-suthar, want to also add `return qs.distinct(), use_distinct` there? sorry I saw this after merging your first PR already so you have to open another one

kerem commented

2026-03-01 14:46:00 +03:00

Author

Owner

@neel-suthar commented on GitHub (Jan 19, 2024):

@pirate will take care of it. This time I will try to reproduce the issue as well. Maybe will include a clip showing that the issue is fixed. Thanks.

@neel-suthar commented on GitHub (Jan 19, 2024): @pirate will take care of it. This time I will try to reproduce the issue as well. Maybe will include a clip showing that the issue is fixed. Thanks.

kerem commented

2026-03-01 14:46:00 +03:00

Author

Owner

@pirate commented on GitHub (Jan 20, 2024):

Closing as fixed, Thanks @neel-suthar! will be out in the next release 0.7.3.

@pirate commented on GitHub (Jan 20, 2024): Closing as fixed, Thanks @neel-suthar! will be out in the next release `0.7.3`.

kerem referenced this issue

2026-03-01 14:49:07 +03:00

[PR #740] [CLOSED] Arch Linux AUR bug fixes backport #1272

kerem referenced this issue

2026-03-01 18:00:44 +03:00

[PR #740] Arch Linux AUR bug fixes backport #2779

kerem referenced this issue

2026-03-15 01:36:21 +03:00

[PR #740] [CLOSED] Arch Linux AUR bug fixes backport #4287