mirror of
https://github.com/cypht-org/cypht.git
synced 2026-04-25 13:05:53 +03:00
[GH-ISSUE #180] Cypht Webmail (aggregator): Add JMAP support (as an alternative to IMAP + SMTP) and permit unified inbox (and even move or copy emails from one account to another) #145
Labels
No labels
2fa
I18N
PGP
Security
Security
account
advanced_search
advanced_search
announcement
api_login
authentication
awaiting feedback
blocker
bug
bug
bug
calendar
config
contacts
core
core
devops
docker
docs
duplicate
dynamic_login
enhancement
epic
feature
feeds
framework
github
github
gmail_contacts
good first issue
help wanted
history
history
imap
imap_folders
inline_message
installation
keyboard_shortcuts
keyboard_shortcuts
ldap_contacts
mobile
need-ssh-access
new module set
nux
pop3
profiles
pull-request
question
refactor
release
research
saved_searches
smtp
strategic
tags
tests
themes
website
wordpress
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/cypht#145
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @marclaporte on GitHub (Mar 27, 2017).
Original GitHub issue: https://github.com/cypht-org/cypht/issues/180
Originally assigned to: @jasonmunro on GitHub.
"JMAP is intended to be a new standard for email clients to connect to mail stores. It therefore intends to primarily replace IMAP + SMTP submission. It is also designed to be more generic such that it can be extended with contacts, calendars in the future (replacing CardDAV/CalDAV). It does not replace MTA-to-MTA SMTP transmission."
http://jmap.io/
@jasonmunro commented on GitHub (Mar 27, 2017):
Thanks for the feedback. I have heard of jmap before, and briefly reviewed it a while back, but I'm not super familiar with the specification. Could be a cool addition to Cypht as either a stand alone module set or as part of the imap modules.
@marclaporte commented on GitHub (Mar 27, 2017):
Super!
Someone else asked me "Who uses or supports it?" My answer copied below:
Cyrus IMAP and Apache James are the notable ones here: http://jmap.io/software.html
OpenPaaS is an open source Entreprise Social Platform which is led by LINAGORA (which has more than 150 employees)
https://linagora.com/
http://www.open-paas.org
OpenPaaS is using Apache James. And you can see JMAP used in the code:
https://github.com/linagora/openpaas-esn/search?utf8=%E2%9C%93&q=jmap&type=
In this video: https://www.youtube.com/watch?v=2QZIi3EO6rQ , Benoit Tellier explains why they decided to invest in James and JMAP (among other things). LINAGORA has 5 full time employees which are devoted to the project.
Why LINAGORA picked JMAP:
https://medium.com/linagora-engineering/you-cant-do-imap-in-the-browser-can-you-208838b0ba17
FastMail started this initiative:
"JMAP is FastMail's protocol with the warts removed. We leverage existing standards like HTTP, JSON, and native push channels on platforms which have them - making it easy for developers to work with."
https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/
https://blog.fastmail.com/2016/12/12/why-we-contribute/
https://blog.fastmail.com/2016/12/09/jmap-test-suite/
https://blog.fastmail.com/2015/12/23/the-jmap-momentum-builds/
https://blog.fastmail.com/2015/08/11/an-open-source-jmap-proxy-javascript-library-and-webmail-demo/
https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/
Roundcube Next will support JMAP as well.
https://github.com/search?q=org%3Aroundcube-next+jmap&type=Code
Thanks!
@jasonmunro commented on GitHub (Mar 27, 2017):
Thanks for that info. I took another peek at the JMAP spec after you submitted this. Looks like a pretty straight forward rest-style api. Cypht was designed to aggregate data from different sources, so a lot of the work of uncoupling from a particular api or protocol is baked in already. Having worked with the IMAP protocol for 15+ years, I understand more than most why something like JMAP is needed :) I think it differs enough from IMAP that it would make sense to create a stand alone module set (also the IMAP module set is already a tremendous amount of code).
I can't promise I will get to this real soon, but I am intrigued. I suspect initial support for this would entail a similar amount of work that the Github module set took - I also suspect we could abstract various bits of the existing IMAP code where the two protocols might overlap.
@marclaporte commented on GitHub (Jan 8, 2019):
Dear Jason,
Now is a good time to start the implementation:
https://fastmail.blog/2018/12/27/jmap-is-on-the-home-straight/
Thanks!!!
@dumblob commented on GitHub (Jan 8, 2019):
@marclaporte how I see it, the most important information is, that during the upcoming weeks/months, JMAP will finally become an IETF standard. Which is a huge step from my point of view, but seems to be also a huge step forward form the JMAP developers point of view (the IETF processes increased the JMAP quality significantly).
@jasonmunro commented on GitHub (Jan 8, 2019):
@dumblob @marclaporte I agree and that was an interesting read, thanks for the link. It's a new year and I'm feeling ambitious so I'm going to bump this up and see what I can get done. The other new feature in the works is cardDav, which has read only support now, and I want to expand that to read/write then move on to calDav support. No reason we can't push both that and JMAP forward at the same time :)
One prerequisite for this is I want to create a "skeleton" module set for new data sources. Sort of a template of all (or at least most) of the entry points required to create a new data source inside Cypht. Then I can use that as the foundation for supporting JMAP, as well as making it easier to build support for any other protocol we want to implement.
@ulfgebhardt commented on GitHub (Jan 8, 2019):
A slim PHP Library is missing at current time.
There is one for Zend:
https://github.com/WikiSuite/zend-jmap
There is many for JS (which is not compatible with current architecture of cypht?!)
https://github.com/search?l=JavaScript&q=jmap&type=Repositories
Protocol Meta for Client:
https://github.com/jmapio/jmap/blob/master/client-guide/jmap-client-guide.mdown
I think it is possible to extract the Implementation out of the Zend Framework from here:
https://github.com/WikiSuite/zend-jmap/blob/master/src/Jmap.php
(and parent folder respectively)
But the License is quite limiting:
https://github.com/WikiSuite/zend-jmap/blob/master/LICENSE.md
@dumblob commented on GitHub (Jan 9, 2019):
For the record, there is also a JMAP<->IMAP proxy which could be useful for testing of some corner cases.
@jasonmunro commented on GitHub (Jan 9, 2019):
@ulfgebhardt and @dumblob thanks for the resources! I suspect we will be writing our own JMAP library in PHP for this. Once we have the "skeleton" module set in place for data sources, most of the boilerplate required to set this up in Cypht is in place. Since JMAP is very RESTful, we can leverage our existing API wrapper code that makes doing RESTy things easy. The final bit is the actual work of deciphering the spec and building out the parsing and logical flows which I suspect won't be nearly as hard as IMAP. We also have a nice set of functions in the core module set for common operations so we can just re-use things we use now (like decoding mime encoded header values for example).
@jasonmunro commented on GitHub (Jan 11, 2019):
As I dig into this I think I might put JMAP support into the IMAP and SMTP module sets. It will likely be a LOT easier with the only downside being you need those 2 sets enabled to use JMAP.
After much head scratching and trial and error, I have a git checkout of Cyrus IMAP running locally with JMAP support - which should be the most up to date WRT JMAP specs as we can get :)
@jasonmunro commented on GitHub (Jan 15, 2019):
Update: spent some time on this over the weekend. Some random thoughts:
After getting folder navigation going I tried to add old school mailbox browsing, but can't seem to get quite what I need yet. Surely I'm doing something wrong but I filed an issue anyway because whatever is not working I just don't see it yet: https://github.com/cyrusimap/cyrus-imapd/issues/2627
@jasonmunro commented on GitHub (Jan 15, 2019):
As expected the issue I was having with Cyrus JMAP was not a bug, but a configuration issue on my end. With there help we are back in business :)
@jasonmunro commented on GitHub (Jan 16, 2019):
Update: Now able to browse mailboxes and have a basic message view working. Here is the roadmap to MVP:
Once we get that far, I want to circle back around to the library code and see what can be done to improve the performance. Right now we are using JMAP in a very IMAP like way, but we can streamline that in a number of places to take advantage of JAMP features like multiple "commands" in a single API call.
@jasonmunro commented on GitHub (Jan 17, 2019):
Update: Ran into another problem with Cyrus JMAP that maybe is not me doing something wrong :) filed an issue (https://github.com/cyrusimap/cyrus-imapd/issues/2631).
In the meantime, Cypht JMAP support now works with combined views (unread, flagged, everything). I also reworked the folder logic in the library and was able to remove some module level work-arounds that were icky. Then I started on search integration but was thwarted by the above issue.
JMAP is complicated (E-mail is complicated!), but it truly is IMAP simplified and modernized. There is lots more to do but we are making great progress a lot faster than I expected. And I'm sort of obsessed with making it feature complete now that I started working on it :)
@jasonmunro commented on GitHub (Jan 17, 2019):
Update: Wrong again. My latest issue was me doing something wrong (specifically not configuring the Cyrus search backend properly). Thanks to the helpful feedback from Cyrus devs it is resolved and search works now with JMAP, as does some message operations (flag/read/unread/etc). For core IMAP stuff needing to be converted, I have move, copy, delete, and folder management up next, following by downloading and then I think we are pretty close to IMAP feature complete!
@jasonmunro commented on GitHub (Jan 23, 2019):
Compatibility layer for JMAP is feature complete! Needs more testing and love but it's working pretty well so far. Here is the code:
https://github.com/jasonmunro/cypht/blob/master/modules/imap/hm-jmap.php
Comparing this to the IMAP lib:
IMAP uses 3036 lines of code excluding comments
JMAP uses 861 lines of code excluding comments
WOW! Up next is wiring the UI to add/delete JMAP servers from the Settings -> Servers page.
@jasonmunro commented on GitHub (Jan 25, 2019):
UI update to add/delete JAMP servers is pushed to the master branch :) Next I'm going to scope out using JMAP for SMTP. This should be way easier as we only communicate with SMTP in 2 places of the code if I recall.
@dumblob commented on GitHub (Jan 25, 2019):
Is anyone of you in contact with JMAP developers or just people around JMAP?
It seems Cypht will be one of the very few first clients supporting JMAP - that could be used to raise awareness of Cypht and its security and mobile advantages. We could start by letting Cypht being linked from JMAP websites. Then maybe write a blog post on JMAP sites, etc.
@jasonmunro commented on GitHub (Jan 25, 2019):
@dumblob I have been in contact with the JMAP folks, and we are now listed here: https://jmap.io/software.html. Also I wrote a blog post on JMAP that made it to the front page of hacker news last night and it's driving a TON of traffic :)
https://unencumberedbyfacts.com/2019/01/24/jmap-its-like-imap-but-not-really/
@marclaporte commented on GitHub (Feb 4, 2019):
Bron Gondwana presented JMAP at FOSDEM 2019.
Slides with Speakers' Notes:
https://fosdem.org/2019/schedule/event/email_standards/attachments/paper/3116/export/events/attachments/email_standards/paper/3116/jmap_slides_fosdem_2019_notes.pdf
General page: https://fosdem.org/2019/schedule/event/email_standards/
Well done Bron!
And well done Jason for such a quick implementation!
@OJFord commented on GitHub (Feb 10, 2020):
For the benefit of others linked to this issue from the JMAP site, my understanding is that this issue remains open only because the JMAP feature isn't yet in a (non-RC) release?
I.e. it is in master, and the (at time of writing) latest 'release', v1.2.0-rc1.
I look forward to trying it, thanks!
@marclaporte commented on GitHub (Feb 11, 2020):
@OJFord : yes, that is correct. And as this is a new feature, on a new protocol, expect it to be bumpy, and please file new issues for bugs and missing features.
I encourage you to get Cypht master and start testing it out. I feel master is stable enough to be released.
In Tiki, we get Cypht master from Packagist and it works quite well:
https://github.com/jasonmunro/cypht/issues/311
If you have trouble installing Cypht master, you can install a daily build of Tiki
20.x is stable with Cypht (I use this daily with 3 mail accounts)
21.x is not released yet and has CalDAV / CardDAV as well (also expect this to be bumpy, and UI is still uncoordinated)
https://dev.tiki.org/Daily-Build
@marclaporte commented on GitHub (Apr 23, 2020):
Cyrus IMAP 3.2.0-rc1 released (with JMAP support)
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-April/041191.html
https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-rc1.html#major-changes-since-the-3-0-x-series
@marclaporte commented on GitHub (May 31, 2020):
Cyrus IMAP 3.2.1 released (with JMAP support)
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041269.html
https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.1.html
@marclaporte commented on GitHub (Jun 11, 2020):
JMAP online meetings/hackathons occur from time to time. We could join one at one point to review our implementation:
https://mailarchive.ietf.org/arch/browse/jmap/
@marclaporte commented on GitHub (Oct 18, 2020):
After over 1100 commits, we can see that the JMAP mail spec has stabilized:
The JMAP community is now focusing on other aspects of JMAP, like contacts, calendars, etc.
So it's a good time to get back to JMAP in Cypht. AFAICT, the most complete server implementation is with Cyrus. There are over 1800 commits for the term "JMAP":
https://github.com/cyrusimap/cyrus-imapd/search?o=desc&q=jmap&s=committer-date&type=commits
Maybe we can use https://github.com/cyrusimap/cyrus-docker/ which @ajaysusarla has recently worked on?
Thanks!
@marclaporte commented on GitHub (Oct 29, 2020):
I was recently made aware of https://github.com/barnslig/jmap-proxy
@mildred commented on GitHub (Apr 21, 2021):
Hi, Thank you for the great work. I'm in the process of integrating cypht in a email stack. I just tested JMAP by manually configuring it and it works great.
However, there is no docker configuration options for JMAP (as there is for IMAP) and nothing configurable in cypht/hm3.ini either.
How can JMAP be globally configured?
Thanks.
@kroky commented on GitHub (May 10, 2021):
@jasonmunro
RE:
AND
I see that you postponed the idea of a "skeleton" module set but now we need it to support https://dev.tiki.org/Email-as-a-first-class-citizen - we want a Tiki-based data storage for emails and still use the excellent Cypht UI to handle the email-related actions like displaying, replying, etc. I started adding as part of the imap module but that quite heavily depends on Hm_IMAP and Hm_JMAP. I think I have a couple of options integrating here:
@henrique-borba commented on GitHub (May 20, 2021):
Far as I know my friends, JMAP is just a mirror of the HM-IMAP module. That said, supporting it will not he a hard work.
@marclaporte commented on GitHub (Oct 3, 2022):
@josaphatim will soon test Cypht's JMAP against https://stalw.art/jmap/ via https://jmap.cloud/
@landryb commented on GitHub (Oct 30, 2022):
fwiw, i've tested current HEAD against a selfhosted stalwart-jmap instance, and it doesnt look good (but i might have done something wrong in my setup):
.well-known/jmapinstead of using the value ofapiUrlgiven by the capabilities document (which seems to be/jmap/in stalwart-jmap's case, at least that's what i understand from comparing a working JMAP session done by meli)because all calls are GET to
.well-known/jmapare without args, of course the capabilities document don't have those values in the returned json.here's the complete json reply for
.well-known/jmap:i'll poke at various things to understand better how it's supposed to work
@landryb commented on GitHub (Oct 30, 2022):
well reading through
hm-jmap.php, it seemsapiUrlis correctly found and a var is initialized with its value, so ... will dig more. Maybe just php misconfiguration from my side with php-fpm & nginx...edit: seems
api_urlassignation is wrong, the line in https://github.com/jasonmunro/cypht/blob/master/modules/imap/hm-jmap.php#L859 seems to append something weird.$urlishttp://fqdn/$data[apiUrl]ishttp://fqdn/jmap/$this->api_urlend up withhttp://fqdnhttp://fqdn/jmap/which doesnt look right at allhardcoding
$this->api_urlto behttp://fqdn/jmap/i see the queries sent as POST but i'm getting a 400 code:@landryb commented on GitHub (Oct 30, 2022):
one of the things i found out from the above json & in testing (but maybe i should file a distinct issues for stalwart-jmap compatibility issues ?):
there's no
isPrimarykey inaccountshash, but a toplevel
primaryAccountshash with capabilities which each list account names.Thus
account_idends up null.. hacking it to not check if accounts are primary, the server still replies with a 400 when listing folders (fromget_folder_list)here's the post sent as logged by nginx via
$request_body:@landryb commented on GitHub (Oct 30, 2022):
playing with curl, it seems what is sent is escaped too much ? "Method/get" works, but cypht sends "Method\/get" which is what triggers a 400 from the server.
@landryb commented on GitHub (Oct 30, 2022):
Fixed the above by adding
JSON_UNESCAPED_SLASHESflag to thejson_encodecall informat_request... now the requests sent to the jmap server all return 200 codes, but cypht returns a 500 to the user. making progress slowly :)@marclaporte commented on GitHub (Sep 12, 2023):
@Yannick243 will resume work here, with some guidance from @josaphatim
@marclaporte commented on GitHub (Oct 24, 2023):
@Shadow243 now joins @Yannick243
@marclaporte commented on GitHub (Feb 23, 2024):
https://github.com/cypht-org/cypht/pull/909
@marclaporte commented on GitHub (Mar 22, 2024):
Progress:
@marclaporte commented on GitHub (Mar 22, 2024):
Spin off: https://github.com/cypht-org/cypht/issues/931
@marclaporte commented on GitHub (Mar 23, 2024):
@Shadow243 @Yannick243 What is the status here? It's time to release Cypht 2.0 :-)
@Shadow243 commented on GitHub (Mar 26, 2024):
Working on this task, we encountered some issues that we had to fix first: https://github.com/cypht-org/cypht/pull/936
We are currently working on the second PR, which will only cover JMAP.
@landryb commented on GitHub (Mar 30, 2024):
i'm feeling a bit dumb, but resuming testing on this against stalwart 0.6.0, in the current cypht master i haven't found a way to setup a new jmap account, there's only IMAP and SMTP listed once you've specified your email/passwd. Is there some hidden setting somewhere ? And i dont feel like #939 is fixing it, since it seems to be for an already configured jmap account. the jmap module is correctly listed in CYPHT_MODULES in
.env@Shadow243 commented on GitHub (Mar 30, 2024):
@landryb commented on GitHub (Mar 30, 2024):
ah, thanks. i dont have that checkbox, but now i see where it comes from in the code, so i'll try to figure out which bit is missing.
edit bah, nevermind, found it.
@marclaporte commented on GitHub (Mar 30, 2024):
@landryb I am super happy you are helping with JMAP testing.
General dev activity level on Cypht has been good recently: https://openhub.net/p/cypht
We have many new features and enhancements coming to Cypht 2.0, and JMAP support is the last one I am hoping we can add: https://github.com/cypht-org/cypht/issues/879
All the active devs are new to JMAP so it's much more challenging than other enhancements.
Thanks!
@landryb commented on GitHub (Mar 30, 2024):
my testing has been limited so far, but im mostly struggling with stalwart, not that much with cypht after all :) Been also testing jmap with meli...
@marclaporte commented on GitHub (Apr 3, 2024):
@landryb Are you saying that JMAP-Cypht is mostly working and we can release Cypht 2.0?
@landryb commented on GitHub (Apr 3, 2024):
i've been able to connect to a JMAP testing account without specific subfolders (eg only default folders), and go to folders that contained mails (#939 was needed for that). The number next to the folder name was always zero, regardless of the folders content.
i havent tested folders/subfolders nor large volumes of e-mail, nor sending an e-mail.
@marclaporte commented on GitHub (Apr 13, 2024):
Cypht 2.0 will ship with JMAP support, with at least one thing missing (will be done after 2.0): https://github.com/cypht-org/cypht/issues/931
@marclaporte commented on GitHub (May 1, 2024):
Cypht 2.0 has been released!
https://github.com/cypht-org/cypht/releases/tag/v2.0.0
@marclaporte commented on GitHub (May 16, 2024):
As of now, we only have one open issue related to JMAP within Cypht: https://github.com/cypht-org/cypht/issues/931
In theory, everything else works. Since this is so new, I do expect we'll encounter issues and we will work in priority on all issues related to JMAP. So please test!
Our reference dev/test server is Stalwart JMAP, and we'd love to collaborate with the Cyrus and Apache James communities, and anyone else working on JMAP!
@marclaporte commented on GitHub (May 16, 2024):
Here is a somewhat related thread:
https://forum.virtualmin.com/t/a-mail-server-that-supports-jmap-json-mail-access-protocol-to-replace-imap-smtp-carddav-and-caldav/126214/3
@marclaporte commented on GitHub (Oct 21, 2024):
As of now, there are 3 webmail clients listed here: https://jmap.io/software.html. If others exist, please update https://github.com/jmapio/jmap/blob/master/software/software.mdown
Of the 3:
So I conclude Cypht is the first/only Open Source standalone webmail client supporting JMAP. While it’s nice to be the trailblazer, I hope many others will join the club 😊. JMAP is an open standard. We can all work together to make it better. Specifically, I hope Roundcube will join the JMAP club soon (I made the original feature request in 2017): https://github.com/roundcube/roundcubemail/issues/5918
Here are other requests I am tracking:
Beyond JMAP, we are now working on EWS (Exchange Web Services) support: https://github.com/cypht-org/cypht/pull/1278
On a related note, here is a page to mainly compare all active and mature Open Source (thus self-hostable) webmail solutions to manage email from any server with standard email protocols (IMAP, SMTP, JMAP) and EWS (Exchange Web Services):
https://wikisuite.org/Webmail-and-groupware-comparison
Cypht permits to have all emails in a unified inbox, and even copy or move emails between accounts.