starred/cypht

Fork 0

mirror of https://github.com/cypht-org/cypht.git synced 2026-04-26 21:46:02 +03:00

[GH-ISSUE #1195] Mails missing from combined view (dovecot 2.3.21) #601

New issue

Closed

opened 2026-02-25 21:35:29 +03:00 by kerem · 13 comments

kerem commented

2026-02-25 21:35:29 +03:00

Owner

Originally created by @indridieinarsson on GitHub (Aug 23, 2024).
Original GitHub issue: https://github.com/cypht-org/cypht/issues/1195

Originally assigned to: @Shadow243 on GitHub.

🐛 Bugreport

I have cypht (2.2.0) running as docker container. I have 2 mail accounts, each on a different mail server. One of the mailservers is a self-hosted one (docker mailserver, running dovecot 2.3.21 as imap backend).

Both mail accounts work as expected when clicking the respective inbox in the "Email" drop down.
However, all mails from the self hosted one (dovecot 2.3.21) are missing from all combined views (Everything, Unread , All etc.)

I have poked around a bit, and found the following search query being sent to the imap server
SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED NOT HEADER X-Auto-Bcc cypht

I didn't quite go so far as to look into the actual reply from the imap server, but at least cypht parses it without any errors, and returns an empty array of messages (and I know there are some messages in this period of time).

If I just commented out some of the code that generates the search query, namely the following lines from hm-imap.php :

if ($only_auto_bcc) {
$fld .= ' HEADER X-Auto-Bcc cypht';
}
if ($exclude_auto_bcc && !mb_strstr($this->server, 'yahoo') && $this->server_supports_custom_headers()) {
$fld .= ' NOT HEADER X-Auto-Bcc cypht';
}

making the search command look like this :
SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED

After this change, the reply from the imap server is parsed, and the Everything view is populated with emails from both of my accounts.

Originally created by @indridieinarsson on GitHub (Aug 23, 2024). Original GitHub issue: https://github.com/cypht-org/cypht/issues/1195 Originally assigned to: @Shadow243 on GitHub. ## 🐛 Bugreport I have cypht (2.2.0) running as docker container. I have 2 mail accounts, each on a different mail server. One of the mailservers is a self-hosted one (docker mailserver, running dovecot 2.3.21 as imap backend). Both mail accounts work as expected when clicking the respective inbox in the "Email" drop down. However, all mails from the self hosted one (dovecot 2.3.21) are missing from all combined views (Everything, Unread , All etc.) I have poked around a bit, and found the following search query being sent to the imap server `SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED NOT HEADER X-Auto-Bcc cypht` I didn't quite go so far as to look into the actual reply from the imap server, but at least cypht parses it without any errors, and returns an empty array of messages (and I know there are some messages in this period of time). If I just commented out some of the code that generates the search query, namely the following lines from hm-imap.php : > if ($only_auto_bcc) { > $fld .= ' HEADER X-Auto-Bcc cypht'; > } > if ($exclude_auto_bcc && !mb_strstr($this->server, 'yahoo') && $this->server_supports_custom_headers()) { > $fld .= ' NOT HEADER X-Auto-Bcc cypht'; > } making the search command look like this : `SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED` After this change, the reply from the imap server is parsed, and the Everything view is populated with emails from both of my accounts.

kerem closed this issue

2026-02-25 21:35:29 +03:00

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@indridieinarsson commented on GitHub (Aug 24, 2024):

Ok - I was a bit too quick to celebrate. When actually looking at the messages, I am getting all messages UNTIL 16-Aug-2024, but not the ones after that.
I must admit, that I don't know a lot (as in nothing) about imap standards and inner workings.

@indridieinarsson commented on GitHub (Aug 24, 2024): Ok - I was a bit too quick to celebrate. When actually looking at the messages, I am getting all messages UNTIL 16-Aug-2024, but not the ones after that. I must admit, that I don't know a lot (as in nothing) about imap standards and inner workings.

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@indridieinarsson commented on GitHub (Aug 27, 2024):

Getting messages from the wrong time period was just due to me messing with the search command (removed the UID part for some testing, forgot to add it back).

But it seems that my mailserver chokes on the NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ). It seems to get a positive answer from the server, the function returns true, but the subsequent search command returns no results.

@indridieinarsson commented on GitHub (Aug 27, 2024): Getting messages from the wrong time period was just due to me messing with the search command (removed the `UID` part for some testing, forgot to add it back). But it seems that my mailserver chokes on the `NOT HEADER X-Auto-Bcc cypht` part. There is a function that checks for support (`server_supports_custom_headers()` ). It seems to get a positive answer from the server, the function returns true, but the subsequent search command returns no results.

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@marclaporte commented on GitHub (Aug 27, 2024):

Related MR: https://github.com/cypht-org/cypht/pull/1152 from @Shadow243

@marclaporte commented on GitHub (Aug 27, 2024): Related MR: https://github.com/cypht-org/cypht/pull/1152 from @Shadow243

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@Shadow243 commented on GitHub (Aug 27, 2024):

NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ).

Just to confirm a doubt, can you please try to add the condition here: https://github.com/cypht-org/cypht/pull/1152/files#diff-2a16fbb7ff5f2c47ea84a2060f25affdc0fbeb9df48cff9f9438fa7ec705f624R1099

if ($only_auto_bcc && $this->server_supports_custom_headers()) {}

@Shadow243 commented on GitHub (Aug 27, 2024): > `NOT HEADER X-Auto-Bcc cypht` part. There is a function that checks for support (`server_supports_custom_headers()` ). Just to confirm a doubt, can you please try to add the condition here: https://github.com/cypht-org/cypht/pull/1152/files#diff-2a16fbb7ff5f2c47ea84a2060f25affdc0fbeb9df48cff9f9438fa7ec705f624R1099 like ``` if ($only_auto_bcc && $this->server_supports_custom_headers()) {} ```

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@indridieinarsson commented on GitHub (Aug 27, 2024):

Thanks @Shadow243.
I am already running with the code in the commit you mention, getting no messages from my dovecot mailserver if I run it as is. Only changes I have made is to comment out lines 1102:1104, then the search query returns the messages I expect it to.

I'm a bit uncertain what you mean exactly, could you perhaps specify more precisely what changes you want me to make.

Greetings,
Indriði

@indridieinarsson commented on GitHub (Aug 27, 2024): Thanks @Shadow243. I am already running with the code in the commit you mention, getting no messages from my dovecot mailserver if I run it as is. Only changes I have made is to comment out lines 1102:1104, then the search query returns the messages I expect it to. I'm a bit uncertain what you mean exactly, could you perhaps specify more precisely what changes you want me to make. Greetings, Indriði

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@indridieinarsson commented on GitHub (Sep 6, 2024):

I've looked into this some more, connecting to the server with pythons imaplib. The server is responding with OK and empty results list when searching with
SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT HEADER X-Auto-Bcc cypht
and also when searching with
SEARCH (ALL) ALL SINCE "16-Aug-2024" HEADER X-Auto-Bcc cypht

If searching by custom headers wasn't supported, the server ought to answer with an error message stating that, I think.

This looks very much like the issue is with this (version of the) server, rather than cypht.

@indridieinarsson commented on GitHub (Sep 6, 2024): I've looked into this some more, connecting to the server with pythons imaplib. The server is responding with OK and empty results list when searching with ` SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT HEADER X-Auto-Bcc cypht` and also when searching with ` SEARCH (ALL) ALL SINCE "16-Aug-2024" HEADER X-Auto-Bcc cypht` If searching by custom headers wasn't supported, the server ought to answer with an error message stating that, I think. This looks very much like the issue is with this (version of the) server, rather than cypht.

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@marclaporte commented on GitHub (Sep 10, 2024):

This looks very much like the issue is with this (version of the) server, rather than cypht.

@indridieinarsson Dovecot has a gigantic install base so this can affect a lot of people. I suggest to report the issue to Dovecot to see where they stand and what we can learn to make Cypht better.

@marclaporte commented on GitHub (Sep 10, 2024): > This looks very much like the issue is with this (version of the) server, rather than cypht. @indridieinarsson Dovecot has a gigantic install base so this can affect a lot of people. I suggest to report the issue to Dovecot to see where they stand and what we can learn to make Cypht better.

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@indridieinarsson commented on GitHub (Sep 17, 2024):

I've been able to track this down to the dovecot-fts-xapian plugin for dovecot.
The issue is not present if I disable full text search and use the default dovecot index instead (which is not nearly as flexible, and mostly unusable for real-life search in gigantic mailboxes), but re-appears as soon as I enable fts again.

In the source for dovecot-fts-xapian, I found that it is indexing only a selected (hard-coded) subset of email headers, and apparently returning an empty search result list for all searches that involve non-indexed headers.

Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either.

@indridieinarsson commented on GitHub (Sep 17, 2024): I've been able to track this down to the dovecot-fts-xapian plugin for dovecot. The issue is not present if I disable full text search and use the default dovecot index instead (which is not nearly as flexible, and mostly unusable for real-life search in gigantic mailboxes), but re-appears as soon as I enable fts again. In the source for dovecot-fts-xapian, I found that it is indexing only a selected (hard-coded) subset of email headers, and apparently returning an empty search result list for all searches that involve non-indexed headers. Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either.

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@Shadow243 commented on GitHub (Sep 17, 2024):

Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either.

That's great. I spent a day looking for it in cypht, I couldn't even reproduce thought.

Can you provide more details so we can document it somewhere?

@Shadow243 commented on GitHub (Sep 17, 2024): > Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either. That's great. I spent a day looking for it in cypht, I couldn't even reproduce thought. Can you provide more details so we can document it somewhere?

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@marclaporte commented on GitHub (Sep 30, 2024):

Re-opening because others seem to be getting this: https://github.com/cypht-org/cypht/issues/1259

So maybe we can detect the specific issue, and report an issue upstream? Then, maybe we can put a warning in Cypht?

@marclaporte commented on GitHub (Sep 30, 2024): Re-opening because others seem to be getting this: https://github.com/cypht-org/cypht/issues/1259 So maybe we can detect the specific issue, and report an issue upstream? Then, maybe we can put a warning in Cypht?

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@indridieinarsson commented on GitHub (Sep 30, 2024):

Is it confirmed that #1259 is the same issue?

My issue (i.e. #1195) was due to a very specific peculiarity in fts-xapian: It indexes only a fixed set of headers, and returns an empty result set for all search queries involving other headers (i.e. "HEADER X-Auto-Bcc cypht" and "NOT HEADER X-Auto-Bcc cypht" are both empty)
If so, there is a cypht-specific fix here : github.com/indridieinarsson/fts-xapian@a4f0edc8fb and here github.com/indridieinarsson/fts-xapian@a0d52e7470
Running cypht against a dovecot server with this version of fts-xapian is confirmed working.

@indridieinarsson commented on GitHub (Sep 30, 2024): Is it confirmed that #1259 is the same issue? My issue (i.e. #1195) was due to a very specific peculiarity in fts-xapian: It indexes only a fixed set of headers, and returns an empty result set for all search queries involving other headers (i.e. "HEADER X-Auto-Bcc cypht" and "NOT HEADER X-Auto-Bcc cypht" are both empty) If so, there is a cypht-specific fix here : https://github.com/indridieinarsson/fts-xapian/commit/a4f0edc8fb5b1e646bdfa8f96409f52a87f263ae and here https://github.com/indridieinarsson/fts-xapian/commit/a0d52e747009780ef3582419e13b82ef9e52e796 Running cypht against a dovecot server with this version of fts-xapian is confirmed working.

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@marclaporte commented on GitHub (Sep 30, 2024):

Is it confirmed that #1259 is the same issue?

I don't know but symptoms are similar. One of Cypht's main selling points (compared to almost everything else) is email aggregation. So if there are cases where this fails, we should try to detect and inform the user how to solve.

Can you propose a PR to https://github.com/grosjo/fts-xapian?
https://github.com/grosjo/fts-xapian/compare/master...indridieinarsson:fts-xapian:master

Thanks!

@marclaporte commented on GitHub (Sep 30, 2024): > Is it confirmed that #1259 is the same issue? I don't know but symptoms are similar. One of Cypht's main selling points (compared to almost everything else) is email aggregation. So if there are cases where this fails, we should try to detect and inform the user how to solve. Can you propose a PR to https://github.com/grosjo/fts-xapian? https://github.com/grosjo/fts-xapian/compare/master...indridieinarsson:fts-xapian:master Thanks!

kerem commented

2026-02-25 21:35:29 +03:00

Author

Owner

@indridieinarsson commented on GitHub (Mar 11, 2025):

Using fts-xapian version 1.9.0, this issue is no longer reproducible.

@indridieinarsson commented on GitHub (Mar 11, 2025): Using fts-xapian version 1.9.0, this issue is no longer reproducible.