mirror of
https://github.com/cypht-org/cypht.git
synced 2026-04-26 05:26:00 +03:00
[GH-ISSUE #238] Email header issues #199
Labels
No labels
2fa
I18N
PGP
Security
Security
account
advanced_search
advanced_search
announcement
api_login
authentication
awaiting feedback
blocker
bug
bug
bug
calendar
config
contacts
core
core
devops
docker
docs
duplicate
dynamic_login
enhancement
epic
feature
feeds
framework
github
github
gmail_contacts
good first issue
help wanted
history
history
imap
imap_folders
inline_message
installation
keyboard_shortcuts
keyboard_shortcuts
ldap_contacts
mobile
need-ssh-access
new module set
nux
pop3
profiles
pull-request
question
refactor
release
research
saved_searches
smtp
strategic
tags
tests
themes
website
wordpress
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/cypht#199
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @szilardx on GitHub (Dec 8, 2017).
Original GitHub issue: https://github.com/cypht-org/cypht/issues/238
Originally assigned to: @jasonmunro on GitHub.
I received feedback that there are some issues with the address fields of the email I send, or forward, or reply to, and it is causing problems for others when answering to me, or forwarding my messages.
So I looked into it, checked the results in Thunderbird. I tested this with gmail accounts.
The problems seems like to be caused by missing or unnecessary quotation signs.
X-Google-Original-From gets an unnecessary " at the beginning?
Sorry for the long post, here is my test in detail (I use no quotation mark on my own here, the marks are where the UI, or the source show them:
I send all mail with a fully set up profile. The profile display in the Compose view bottom is in "My Sender Name" myaddress@gmail.com smtpservername-in-cypht format.
Receiving that mail in Thunderbird results in the UI displaying:
Sender: My Sender Name" myaddress@gmail.com myaddress@gmail.com
To: =?UTF-8?B?S8.......................................................HU+?= <>
Looking at the source shows:
From: "=?UTF-8?B?U.........................................T4=?=" myaddress@gmail.com
X-Google-Original-From: "=?UTF-8?B?........................................T4=?=
X-Mailer: Cypht
MIME-Version: 1.0
Reply-To: myaddress@gmail.com
To: "=?UTF-8?B?S8.............................................HU+?=
Cypht Sent UI shows:
From "My Sender Name" myaddress@gmail.com" myaddress@gmail.com
To "My Contact Name" mycontact@address.com
Cypht Sent RAW shows:
From: "=?UTF-8?B?...............................T4=?=" myaddress@gmail.com
X-Google-Original-From: "=?UTF-8?B?U3.................................T4=?=
X-Mailer: Cypht
MIME-Version: 1.0
Reply-To: myaddress@gmail.com
To: "=?UTF-8?B?U3.............................Pg==?=
Subject: =?UTF-8?B?d...............A==?=
Results the same in Thunderbird, in UI and in source too.
In Cypht Sent UI the To field is the following:
TO "My Contact Name" mycontact@address.com
Cypht sent RAW is the same as with the previous case.
From "My Sender Name" myaddress@gmail.com
To My Contact Name mycontact@address.com
RAW:
From: "=?UTF-8?B?U3....................Q==?=" myaddress@gmail.com
X-Google-Original-From: =?UTF-8?B?U3................Q==?= myaddress@gmail.com
To: =?UTF-8?Q?....somewhat....readable.....?=
mycontact@address.com
Subject: =?UTF-8?Q?test..somewhat-readable....?=
@jasonmunro commented on GitHub (Dec 8, 2017):
No idea whats going on here. We don't actually set X-Google-Original-From, so I don't know why it's got a leading quote. also this is concerning:
Can't explain that either, but it reminds me of this issue:
https://github.com/jasonmunro/cypht/issues/218
I will see what I can figure out and let you know
@szilardx commented on GitHub (Dec 9, 2017):
I used base64 decoder to further analyze and compare the source of email sent from Cypht and Thunderbird.
Here is my comparison:
For the From: header field, the format is the same for Thunderbird and Cypht too:
From: "=?UTF-8?B?.........................=?=" <myaddress@gmail.com>The Cypht base64 encoded part of this field also has quotes in the encoded text too, and contains the email address too., I think this is causing one of the problems. This quotes are also visible in Cypht UI when choosing sender profile. This is how the dotted out part looks when decoded:
My Senderame" <mymail@gmail.com>The To: field in the email source header is different for Cypht and Thunderbird:
Cypht is like this:
To: "=?UTF-8?B?....................................==?=Decoded to:
My Contact" <mycontact@gmail.com>Thunderbird is like this:
To: =?UTF-8?Q?..................................?= <mycontact@gmail.com>The dotted out part is decoded from quoted-printable:
My ContactnameSo my idea is that this issue could be solved by removing all the quotation marks from the Contact names that are sent to the utf-8 encoder from the Compose view.
The From: field is in good format otherwise, only the extra email address is not needed after the sender's name.
For the To: field, the email address should be left out from the encoded part I think.
@szilardx commented on GitHub (Dec 12, 2017):
I tried to fix this issue.
It seems to work for me, but I am a novice in php, and totally new to email clients :)
https://github.com/jasonmunro/cypht/compare/master...szilardx:master
Edit: I did a fix for the "reply", and "reply all" path too.
The main cause for this issue seems to be caused by two things:
the email address is getting encoded with the contact name, instead of being put after the encoded part between <> marks.
the encode_header_fld function is putting a leading quote, but only a leading quote before the encoded header fields in the case of a Contact name, and email address, because the name and the email address is treated as a single string.
@jasonmunro commented on GitHub (Dec 15, 2017):
@szilardx thanks for digging into this! It was my understanding that the encoding of header values does not need to be restricted to just a portion of the address, but I can see how that could be problematic for other clients. Looking at your fork I'm a bit concerned about removing quotes around address names, but if enchode_header_fld is inserting a leading quote, that is definitely an issue. I know that method is a mess from some recent changes. My first step to solving these issues is going to be to rewrite that bit and see how that helps. I will let you know when I have something for you to test!
@szilardx commented on GitHub (Dec 16, 2017):
Thanks! I just made the fork to temporarily fix the issue for myself.
I agree that quotes should not be removed entirely, though I prefer that sender and recipient name would not have quotes generally, but quotes are okay if the email address is the sender name also.
@jasonmunro commented on GitHub (Dec 18, 2017):
Just pushed an update to the encode_header_fld method, would love to know if this makes things look a bit better for you @szilardx.
@szilardx commented on GitHub (Dec 19, 2017):
Getting addresses from the Composer autocomplete results in a strange format, that caused the problems previously too:
Thunderbird UI To, and CC fields shows: "Contact Name address"@domain.com
From field: MySenderName myaddress@domain.com --> The problem here is that the sender name is missing a space.
Looking at the mail source view shows, the following:
From: "=?UTF-8?B?..................==?= =?UTF-8?B?..................=?= PartsWithNoInternatinalCharacter" myaddress@domain.com
To: "=?UTF-8?B?.............==?= =?UTF-8?B?.................=?=" address@domain.com
When Thunderbird sends an email, the resulting source is like this:
To: =?UTF-8?Q?..................................?= address@domain.com
Thank you for the work!
@szilardx commented on GitHub (Dec 19, 2017):
If this helps, here is a test case:
Contact or sender name (which we can set in a profile) could be:
Kovács Béla | Company Co.
This shold result in:
From: =?UTF-8?B?S292w6FjcyBCw6lsYSB8IENvbXBhbnk=?= kovacs@company.com
Same format for the To: field.
For the sender name, and contact name used in Cypht I would prefer not having quotes around the names as most of my emails does not have them, and also Thunderbird seems to be confused by quotation marks.
I looked through my mail, and there are some cases when the sender used quotation marks around his/her name, but in the source it shows that the quotation marks are inside the encoded part.
@jasonmunro commented on GitHub (Dec 19, 2017):
honestly I think I need to review the RFC specifications around address formats :) Both around quoting and "mime word" encoding. I would love to NOT have quotes when we don't need them, they tend to be a pain, but I want to see about compliance rules. With regards to the "mime word" encoding. it's my understanding those can appear anywhere in an address, even just a portion of the name part, but again I need to review it. I will update again when I know what the heck I'm talking about! :)
@jasonmunro commented on GitHub (Dec 19, 2017):
Ok, I think I have a bit more to go on now, and we are definitely doing some things wrong here :)
I'm going to work on this and let you know when I have it pushed.
@jasonmunro commented on GitHub (Dec 20, 2017):
So it turns out we actually DO need quotes on some display names, like if they have a comma. I think I have a handle on what we need to do to be compliant with the specifications, just need to figure out how to code it up :)
@szilardx commented on GitHub (Dec 20, 2017):
Oh, okay. That seems like a valid reason.
Thanks for the work.
@jasonmunro commented on GitHub (Dec 20, 2017):
Like everything to do with E-mail, it's painfully over-complicated :) The display name in an address is of type "phrase", which is one or more "words". "words" are an "atom" (or a quoted string). "atoms" are one or more "atext":
So if a display name only contains these characters we don't need to quote it. The trick now is, what if a display name contains something not in the list, AND needs to be "mime-word" encoded because it has non-ASCII characters?
I think the right answer there is to quote it, then mime-word encode the whole darn thing (in chunks because of the 75 char rule for mime-word encoding), including the quotes! :)
http://www.rfc-editor.org/rfc/rfc2822.txt
@dumblob commented on GitHub (Dec 21, 2017):
That was and still is my understanding. But I didn't read all the RFCs carefully, so there still might be an issue I could not spot.
@jasonmunro commented on GitHub (Dec 21, 2017):
@szilardx fixes pushed. :) This change set:
I tested with a bunch of various address formats, and I think this is working correctly.
@szilardx commented on GitHub (Dec 21, 2017):
@jasonmunro Thanks for the fixes :)
It is getting better definitely: Thunderbird recognizes the email addresses properly, although there are small issues still:
example:
Kovács Bélabecomes"KovácsBéla"-->space missingKovács Béla | Companybecomes"KovácsBéla | Company--> missing space between First and Family name, and also the leading quote can confuse mail clients.Maybe the missing space issue could be caused by the separated encoding of the words?
@jasonmunro commented on GitHub (Dec 21, 2017):
@szilardx thanks for the quick follow up, I should be able to recreate this and figure it out. Looking now!
@jasonmunro commented on GitHub (Dec 21, 2017):
@szilardx I think I found the problem. We were not treating the display name as a single entity, and looping over the words inside it and encoding those, which was incorrect. Just pushed a fix that seems to correct the issue here (both the missing spaces and the leading quote)
@szilardx commented on GitHub (Dec 21, 2017):
@jasonmunro seems to be okay now 👍
Strange issue: no mail show up in combined views? Everything, Sent, etc.?
@jasonmunro commented on GitHub (Dec 21, 2017):
oh dear. I wonder if it's related to this
github.com/jasonmunro/cypht@38ebdc9d4eI'm not seeing the same issue however, so weird@szilardx commented on GitHub (Dec 21, 2017):
I tested it.
Going back to
github.com/jasonmunro/cypht@162f823ae8works with combined views.Then updating to
github.com/jasonmunro/cypht@38ebdc9d4ebroke it.@jasonmunro commented on GitHub (Dec 21, 2017):
@szilardx thanks for testing. Looks like without the search charset included we had a "double space" bug in the search command (you can't have extra spaces in IMAP commands). Could you update to the latest and see if it's resolved?
@szilardx commented on GitHub (Dec 21, 2017):
Yes, issue resolved, Thank you very much! :)
@jasonmunro commented on GitHub (Dec 22, 2017):
As always, thanks @szilardx for the help on this. Closing for now. Re-open if it's not resolved!
@szilardx commented on GitHub (Dec 28, 2017):
@jasonmunro Happy holidays and happy new year! :)
I realized, that the header still have some issue:
for example:
A levél subjectis fine, butLevél ÖnnekbecomeslevélÖnnek@jasonmunro commented on GitHub (Dec 30, 2017):
@szilardx happy holidays to you, and thanks! I will look into this, hopefully tomorrow. I'm on vacation this week, so trying to stay offline as much as possible :) I did not see this in my testing, but I'm not surprised. Having two "mime encoded" words in a row separated by a space should be concatenated without the space (it needs to be encoded in one of the "mime words").
@dumblob commented on GitHub (Dec 30, 2017):
Don't forget about multiple spaces between such words 😉.
@szilardx commented on GitHub (Jan 4, 2018):
Sorry to bother, I just realized that there are some inconsistency with the quotes.
Thunderbird shows no problems in the UI though, but Cypht shows it.
And also Gmail UI is having some issues.
If the To: address has only an email address, the header filed becomes
"" <mail@something.com>This is visible in Cypht.If quotes are added to the To: field, in Gmail the recipient filed shows
"\"Name with National Chars\"" <mail@address.com>I fixed this temporarily with commenting out
github.com/jasonmunro/cypht@24a96747bf/modules/smtp/hm-mime-message.php (L118)Line 118, 119, 120 --> so disabling quoting.
I checked, Thunderbird uses base64 without quotes, Gmail uses quoted-printable without quotes.
Thanks!
@jasonmunro commented on GitHub (Jan 4, 2018):
@szilardx interesting, thanks for the feedback. I think my misconception here is that non-ascii chars must be in quotes. However, once they are mime-word encoded, they don't contain any characters that the RFC requires be quoted - so they are not needed. I will revisit this.
@jasonmunro commented on GitHub (Jan 5, 2018):
@szilardx Just a pushed a fix in
github.com/jasonmunro/cypht@ff17d92616This encodes a display name first, then decides if it needs quotes. If it has national chars and is then encoded, it does not need to be quoted. Let me know if this is any better!@szilardx commented on GitHub (Jan 6, 2018):
Thanks it works fine now!
The only issue remaining in this thread is that the spaces are getting left out if the words in the subject are encoded:
https://github.com/jasonmunro/cypht/issues/238#issuecomment-354272522
@dumblob commented on GitHub (Jan 6, 2018):
We shall also test it with multiple spaces between two words (e.g.
Levél Önnekor more likeLevél Önnek).@jasonmunro commented on GitHub (Jan 6, 2018):
@szilardx excellent, thanks for the feedback.
@dumblob Can do, my understanding is one or more spaces between mime encoded words should be ignored (which is why our incorrect encoding results in the loss the space when decoded).
@jasonmunro commented on GitHub (Jan 11, 2018):
@szilardx final (hopefully) fix is pushed. Let me know if you seem more trouble!
@szilardx commented on GitHub (Jan 11, 2018):
@jasonmunro Yes, I tested it and it works okay.
Thank you very much for your patience and work!