mirror of
https://github.com/cypht-org/cypht.git
synced 2026-04-25 04:56:03 +03:00
[GH-ISSUE #9] Enhancement: support for filtering emails #6
Labels
No labels
2fa
I18N
PGP
Security
Security
account
advanced_search
advanced_search
announcement
api_login
authentication
awaiting feedback
blocker
bug
bug
bug
calendar
config
contacts
core
core
devops
docker
docs
duplicate
dynamic_login
enhancement
epic
feature
feeds
framework
github
github
gmail_contacts
good first issue
help wanted
history
history
imap
imap_folders
inline_message
installation
keyboard_shortcuts
keyboard_shortcuts
ldap_contacts
mobile
need-ssh-access
new module set
nux
pop3
profiles
pull-request
question
refactor
release
research
saved_searches
smtp
strategic
tags
tests
themes
website
wordpress
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/cypht#6
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @dumblob on GitHub (Jul 30, 2015).
Original GitHub issue: https://github.com/cypht-org/cypht/issues/9
Originally assigned to: @jasonmunro on GitHub.
Are there any plans to support message filtering? It would be sufficient to provide just a list of extended regular expressions applied line-by-line on all headers and a list of actions for each regexp in case it matched.
@jasonmunro commented on GitHub (Jul 30, 2015):
Definitely. I built client side filtering in my last mail client and plan to support that eventually here as well.
@dumblob commented on GitHub (Jul 30, 2015):
Awesome. If something should appear, I'd advocate for allowing passing feeds (RSS...) through these filters as well. Yes, matching only headers wouldn't suffice, but extending matching to whole message bodies should be quite easy.
Btw, is there any todo list (not necessarily a firm/fixed one) I could follow (and prevent e.g. duplication of questions)?
@jasonmunro commented on GitHub (Jul 30, 2015):
As much as possible, I want any feature added to Cypht to apply to all applicable data sources (feeds, E-mail, whatever), so I agree with your proposal.
Insofar as a todo list, not really :) I love any and all feedback though, so feel free to file as many issue items as you like and we can use this as an ad-hoc todo list until we have a more formal roadmap. There are 2 major things I want to complete before starting on something like filters:
Once these are done I think we can call ourselves a basic, but reasonably feature complete, E-mail client. Then we can start on the fun stuff like filters and threading :)
@dumblob commented on GitHub (Aug 7, 2015):
Thanks for the plan overview. I'll happily do some "unofficial" code reviews on the latest commits.
Also, there should be some subtle button to import and export all user filters to a formatted file (either some ad-hoc JSON/... structure or standardized format like Sieve in case the filters will be implemented according to the corresponding standard). The export function is more important (and easier to implement) than the import feature.
In my very own opinion, we don't have to support Sieve filters, because they're quite tricky to implement (especially because to make them enough expressive for real life usage, one must implement many additional RFC standards extending the original Sieve standard) and slow, but rather some dead-easy regex matching performed separately on the email headers section and on the rest of the email. For such regex matching, we should guarantee, that the matching won't get stuck in regex exponential algorithms and thus we should use DFA-based engine, e.g. RE2 engine (maybe https://github.com/arraypad/php-re2).
@jasonmunro commented on GitHub (Aug 11, 2015):
Really appreciate the support! If you see anything in a commit that looks suspect, I would love your feedback! I agree with all your suggestions WRT a filtering mechanism. I think we could make use of the built in PCRE engine in PHP by white-listing patterns we allow, as well as adding anchors to avoid inadvertent backtracking. I would love to support a standard like Sieve eventually, but a simple regex is a good start in the meantime (and likely good enough for most requirements).
@dumblob commented on GitHub (Aug 25, 2015):
After some consideration and investigation involving RFC 5228 (Sieve), it's implementation in dovecot including extensions, and RFC 5804 (ManageSieve) I came to conclusion, that regexps would be much better (Sieve with all the existing RFC extensions/plugins is still very limited), but their implementation is more or less on the same level of complexity as incorporating some PHP library for the ManageSieve protocol.
This means, that it might make sense to first implement support for ManageSieve and then consider if it's worth implementing direct support for any other filtering (be it Sieve or ad-hoc regexps or both), because e.g. Gmail uses it's own non-standard filters (ManageSieve is not supported).
I hope the PHP libraries mentioned on http://sieve.info/libraries would suffice (in terms of security and velocity).
@chlarsen commented on GitHub (Sep 27, 2016):
Agreed to that, Jason: The nice thing with server-site filtering is that it can be centralised for bigger environments, while client-side is always a matter of individual preferences. This is not only about controlling mails, but also, for instance, to have certain mails go into certain folders routinely, by default, for operational reasons, for all staff, etc. Cheers!
@marclaporte commented on GitHub (Jul 3, 2019):
For the Cypht integration in Tiki, we plan (medium term) to add more automation. Ex.: filtering mail for a certain criteria, parsing to extract content, and do different actions (ex.: add info to a database, forward a modified email, etc.)
All this should happen server side, but we want a nice client side GUI to build the filters and actions.
@marclaporte commented on GitHub (Mar 9, 2020):
Let's get started on this.
An important first step is to pick the best lib. Once we pick and integrate the lib, we can progressively add features and improve the UI. So here is some research:
Used by Roundcube and Kolab, https://github.com/pear/Net_Sieve is a contender but https://pear.php.net/package/Net_Sieve reports that "This package has been superseded, but is still maintained for bugs and security fixes. Package has moved to channel pear.horde.org, package Horde_ManageSieve.", which can be found here: https://github.com/horde/ManageSieve
Some other PHP options:
https://github.com/sergiosgc/Sieve_Parser
https://github.com/ProtonMail/libsieve-php/
Nextcloud mail is working on this: https://github.com/nextcloud/mail/pull/1822
And strangely, they include 2 libs:
https://github.com/nextcloud/mail/pull/1822/files#diff-b5d0ee8c97c7abd7e3fa29b9a27d1780
"Ingo, the "Email Filter Rules Manager", started as a frontend for the Sieve filter language, and is now a generic and complete filter rule frontend that currently is able to create Sieve, procmail, maildrop, and IMAP filter rules."
https://www.horde.org/apps/ingo
And now, some non-PHP projects of interest:
A former Thunderbird add-on which has become an Electron app:
https://github.com/thsmi/sieve/
SOGo (Objective-C) has its own system or permits an external Sieve script:
https://sogo.nu/bugs/view.php?id=4387
Apache James jSieve (Java)
https://github.com/apache/james-jsieve
Do you know of any information that can help our choice?
Thanks!
@marclaporte commented on GitHub (Apr 30, 2020):
@dumblob @jasonmunro: Do you know of any more options or we can start comparing the one I found and proceed by elimination?
Thanks!
@dumblob commented on GitHub (May 1, 2020):
@marclaporte no, don't know any other off the head. But as I pointed out above, please focus rather on regexps and discard (or at least postpone) the idea of Sieve as they're extremely limited, slow, etc. Sieve just doesn't fit into the Cyph minimalism at all. Not speaking about the fact that you can achieve everything what Sieve can with regexes.
The cool part is, that support for regex filtering will have the very same or simpler UI than Sieve. So again, please reconsider the priorities. Additionally I claim, that everybody who uses Sieve can write regexes (and those who don't use Sieve, but some very limited subset through some specially crafted UI can be satisfied with the same UI which under the hood uses regexes instead of Sieve).
I find Sieve a complete dead end slowly dying over the last decade or more.
The only thing which could have a marginal chance of convincing me to additionally support Sieve (along regexes) is the ManageSieve protocol. But I'm convinced it's not necessary for Cypht as Cypht is a web mail and not all mail providers offer ManageSieve, so one couldn't even use it everywhere, so it's way easier and safer to implement local matching based on regexes IMHO which'll work always and everywhere.
@marclaporte commented on GitHub (May 1, 2020):
Ok, but how would we do server-side filters with a mail server we don't control? ( shared hosting)
in Tiki, we can use http://doc.tiki.org/Browser-Automation but what about standalone Cypht?
@dumblob commented on GitHub (May 1, 2020):
First of, we don't. The point is there shouldn't be any need for this. Cypht knows which messages are "really new" (i.e. unseen/not_yet_opened) and which are just "set unread" by the user; so the filtering happens at each login or if already logged in each time a new message appears in the inbox.
What did you have in mind with browser automation? Everything will run on the Cypht server, so it has nothing to do with any browser.
@marclaporte commented on GitHub (May 2, 2020):
Users may access mailbox from another client, so filters should be running in the background.
Browser automation: If some of the actions are client-side.
But the best is to put all this in PHP, on a cron job.
@dumblob commented on GitHub (May 2, 2020):
I intentionally wrote The point is there shouldn't be any need for this. because if someone decides to use Cypht, there is no practical reason to use other email clients. If you'll find such "combined" use cases, I'd be happy to discuss them 😉.
That'd require the user to be logged in all the time which will compromise security. But maybe we could make it a user setting in the end.
Very unlikely to be needed as it's definitely not wanted (it'd complicate the client-server communication in both ways and more importantly increase the bandwidth in both ways - and we know that especially upload is usually a scarcity at least on mobile devices).
@marclaporte commented on GitHub (May 3, 2020):
Some users will want offline access, which could be done via PWA: https://github.com/jasonmunro/cypht/issues/395
I want filters to work throughout accounts. Ex.: I want a filter that detects invoices and moves out of my inbox and into the inbox of the bookkeeper. This should happen even if I am offline for a while.
@dumblob commented on GitHub (May 3, 2020):
I'm afraid I don't follow. Why would one need any message filters (don't forget we're talking about newly incoming message filters here) in an offline mode (when by definition you can't get any new messages 😉).
This means you'd like the computer to act on your behalf which translates to you'd like to delegate part of your responsibility to a third party (Cypht server and its "affiliates" who usually do explicitly limit their liability to virtually zero).
So I think you'd need to tick the option "stay logged in after log out" to lower security in exchange for more convenient functionality 😉.
@marclaporte commented on GitHub (May 3, 2020):
"I'm afraid I don't follow." You asked why someone would use something else than Cypht.
@dumblob commented on GitHub (May 3, 2020):
Ah, you mean Cypht doesn't offer full offline functionality now, so that could be the reason to use another client. Fair enough.
In that case though I'm afraid PWA has nothing to do with offline access, so we should make a separate issue for offline mode. And I mean a fully "automatic" offline mode - i.e. the client will work as usually even without internet (i.e. no manual intervention/switching will be needed), but will just show some "exclamation mark icon" when composing & sending message which'll disappear once no work is left (i.e. all queued messages to send will be successfully sent out).
@marclaporte commented on GitHub (May 4, 2020):
"PWA has nothing to do with offline access"
What is a better way to create an offline web app? (Beside Electron)
@jasonmunro commented on GitHub (May 4, 2020):
Lets move "offline access" to the PWA issue and continue the discussion on filtering here. I'm of two minds on this topic:
If I were to build this in Cypht right now, I would start with simple regex based client side filtering in the Cypht PHP code via a module set. Not logged in? No filtering. Using another client? No filtering.
Cypht is a lightweight E-mail client, not an always on server side inbox filter.
I would build this with an eye on abstracting the implementation of defined filters such that it might be possible to extend it to write procmail or sieve rules, so for sites that support it you can have nice server side filtering outside of Cypht.
@dumblob commented on GitHub (May 4, 2020):
Oh, now I see the misunderstanding. I just meant that for offline access there is no need to implement PWA (because offline access is not equivalent to loading an app offline). And vice versa - if you're PWA-ready you have the app loading offline, but the app itself doesn't need to work offline at all (i.e. doesn't need to do any meaningful work; meaningful from the user's point of view).
Awesome. That's the third option - to implement both. I didn't emphasize this option (just initially wrote about ManageSieve as the underlying core) as I anticipated it to be more complicated and difficult to maintain. But I'm glad it's on the todo list 😉. Thanks!
@marclaporte commented on GitHub (May 15, 2020):
JMAP & Sieve:
https://tools.ietf.org/html/draft-murchison-jmap-sieve-01
https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/jmap_sieve.c
@marclaporte commented on GitHub (Jun 17, 2020):
https://www.ietf.org/id/draft-gondwana-sieve-mailboxid-02.txt
@dumblob commented on GitHub (Sep 2, 2020):
Maybe we could finally forget about built-in support for Sieve in Cypht if Cypht will integrate with parts of (or at least refer to) Sieve Editor.
Another interesting thing is that Sieve Editor supports the Regular Expressions Extension of Sieve which could mean that the whole functionality requested in this GitHub issue is covered by Sieve Editor. I'll try to find some time to test it. Feel free to do the same 😉.
@marclaporte commented on GitHub (Sep 23, 2020):
@dumblob: Sieve Editor looks powerful and mature. Can you describe how you envisage the integration?
Thanks!
@dumblob commented on GitHub (Sep 23, 2020):
I'd sum my vision as "as loosely as possible while maintaining partially smooth workflow with filters". Maybe just
iframeon a few places? In other words, I don't want to integrate Sieve editor into Cypht, but rather integrate Sieve editor with Cypht if that makes sense.@marclaporte commented on GitHub (Sep 23, 2020):
Yes, sure it makes sense. It looks quite advanced and it could provide some quick wins.
My big concern: How would Sieve editor have Cypht's account list and permit to move emails from one account to another, and to forward mail?
Thanks!
@marclaporte commented on GitHub (Oct 21, 2020):
@dumblob Someone is more ambitious than you :-) https://github.com/thsmi/sieve/issues/258
@marclaporte commented on GitHub (Feb 14, 2021):
Some interesting ideas here:
https://forum.virtualmin.com/t/improvements-to-mail-filtering/108798
@marclaporte commented on GitHub (Jan 14, 2022):
We are starting work on this.
Ref: https://gitter.im/cypht-org/community?at=61e1929c2a210c38c1cafb2e
@dumblob commented on GitHub (Jan 16, 2022):
Seeing you adding related links to filtering, I'll add one too 😉.
This is what I did under pressure when the discussion didn't seem to go anywhere: Nextcloud Mail - UI proposal for filtering (it's all based around "filters as views" idea).
@marclaporte commented on GitHub (Mar 8, 2022):
For the record, @henrique-borba has started on this: https://github.com/henrique-borba/php-sieve-manager
@marclaporte commented on GitHub (Apr 6, 2022):
Screenshot:
https://gitter.im/cypht-org/community?at=624d9bdbd1b64840dba0efa8
@marclaporte commented on GitHub (May 26, 2022):
Draft: https://github.com/jasonmunro/cypht/pull/569/
It uses the lib, which has progressed: https://github.com/henrique-borba/php-sieve-manager/commits/master
@marclaporte commented on GitHub (Jul 9, 2022):
Very good progress: https://github.com/jasonmunro/cypht/pull/569
@marclaporte commented on GitHub (Jul 31, 2022):
Closing as done: https://github.com/jasonmunro/cypht/pull/569 (Just need @jasonmunro to approve merge request)
@marclaporte commented on GitHub (Feb 24, 2023):
A follow up: https://github.com/jasonmunro/cypht/issues/679
FYI @dumblob