starred/papermerge

Fork 0

mirror of https://github.com/ciur/papermerge.git synced 2026-04-25 12:05:58 +03:00

[GH-ISSUE #339] IMAP import from gmail isn't working #268

New issue

Closed

opened 2026-02-25 21:31:34 +03:00 by kerem · 9 comments

kerem commented

2026-02-25 21:31:34 +03:00

Owner

Originally created by @DrVanSteiner on GitHub (Mar 8, 2021).
Original GitHub issue: https://github.com/ciur/papermerge/issues/339

Originally assigned to: @ciur on GitHub.

Description
IMAP import from gmail isn't working in Docker environment running version 2.0.0rc45 and before.

Expected
Attachements of Emails get imported

Actual
Papermerge can connect the IMAP Server and reports the count of UNSEEN messages but afterwards nothing happens not even an error. (Even with logging enabled as described here: https://github.com/ciur/papermerge/issues/118#issuecomment-693249651) i see only the following in the logfile:

MAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 2 count
IMAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 1 count
IMAP Import: UNSEEN messages 2 count
IMAP Import: UNSEEN messages 2 count

Configuration of IMAP:
IMPORT_MAIL_HOST = "imap.gmail.com"
IMPORT_MAIL_USER = "blabla"
IMPORT_MAIL_PASS = "blabla"
IMPORT_MAIL_BY_USER = False
IMPORT_MAIL_BY_SECRET = False
IMPORT_MAIL_DELETE = False

(Importing from local folder is working...)

Info:

Papermerge Version [2.0.0rc45

Originally created by @DrVanSteiner on GitHub (Mar 8, 2021). Original GitHub issue: https://github.com/ciur/papermerge/issues/339 Originally assigned to: @ciur on GitHub. **Description** IMAP import from gmail isn't working in Docker environment running version 2.0.0rc45 and before. **Expected** Attachements of Emails get imported **Actual** Papermerge can connect the IMAP Server and reports the count of UNSEEN messages but afterwards nothing happens not even an error. (Even with logging enabled as described here: https://github.com/ciur/papermerge/issues/118#issuecomment-693249651) i see only the following in the logfile: MAP Import: UNSEEN messages 1 count IMAP Import: UNSEEN messages 1 count IMAP Import: UNSEEN messages 2 count IMAP Import: UNSEEN messages 1 count IMAP Import: UNSEEN messages 1 count IMAP Import: UNSEEN messages 2 count IMAP Import: UNSEEN messages 2 count Configuration of IMAP: IMPORT_MAIL_HOST = "imap.gmail.com" IMPORT_MAIL_USER = "blabla" IMPORT_MAIL_PASS = "blabla" IMPORT_MAIL_BY_USER = False IMPORT_MAIL_BY_SECRET = False IMPORT_MAIL_DELETE = False (Importing from local folder is working...) **Info:** - Papermerge Version [2.0.0rc45

kerem

2026-02-25 21:31:34 +03:00

closed this issue
added the
bug
label

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@DrVanSteiner commented on GitHub (Mar 9, 2021):

What i see so far is that contains_attachments always returns false.

@DrVanSteiner commented on GitHub (Mar 9, 2021): What i see so far is that contains_attachments always returns false.

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@ciur commented on GitHub (Mar 12, 2021):

@DrVanSteiner, can you please check if release 2.0.0rc48 fixes this issue ?

@ciur commented on GitHub (Mar 12, 2021): @DrVanSteiner, can you please check if release [2.0.0rc48](https://github.com/ciur/papermerge/releases/tag/v2.0.0rc48) fixes this issue ?

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@DrVanSteiner commented on GitHub (Mar 15, 2021):

No it's not fixed with that version. The worker still only reports the number of unseen messages and nothing happens.

@DrVanSteiner commented on GitHub (Mar 15, 2021): No it's not fixed with that version. The worker still only reports the number of unseen messages and nothing happens.

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@l4rm4nd commented on GitHub (Mar 15, 2021):

Can confirm the issue.

Mails are correctly identified, but not imported. Should be a bug in import_attachment() of papermerge-core's imap.py.

Did some basic testing.

IMAP login, inbox selection and e-mail fetching work without an issue:

from papermerge.core.importers.imap import *
imap_client = login("imap.gmail.com","test@gmail.com","MySecurePassword")
imap_client = select_inbox(imap_client, "INBOX")
messages = imap_client.search(['UNSEEN'])
print("IMAP Import: UNSEEN messages %s" % (messages))

However, fetched e-mails don't seem to have an attachement. This is weird, since my sent e-mails clearly have!

messages_structure = imap_client.fetch(messages, ['BODYSTRUCTURE'])
for uid, structure in messages_structure.items():
  if not contains_attachments(structure[b'BODYSTRUCTURE']):
    print('no attach - weird, I did sent a file!')

@ciur: So this seems to be a bug in your validation contains_attachments(). The mails are removed from the messages object and therefore not processed - although having an attachement!

Note: The IMAPClient library provides a special gmail search function. Maybe we can utilize this instead of contains_attachments() for gmail specifically to fix the issue? Haven't had a look at your contains_attachments() implementation yet.

# This method only works for IMAP servers that support X-GM-RAW, which is only likely to be Gmail
messages = imap_client.gmail_search("has:attachment in:unread")

@l4rm4nd commented on GitHub (Mar 15, 2021): Can confirm the issue. Mails are correctly identified, but not imported. Should be a bug in `import_attachment()` of papermerge-core's [imap.py](https://github.com/papermerge/papermerge-core/blob/f22fe0b2b080cbe78b48dfaf02da0e638d23266f/papermerge/core/importers/imap.py#L269). Did some basic testing. - IMAP login, inbox selection and e-mail fetching work without an issue: ```` from papermerge.core.importers.imap import * imap_client = login("imap.gmail.com","test@gmail.com","MySecurePassword") imap_client = select_inbox(imap_client, "INBOX") messages = imap_client.search(['UNSEEN']) print("IMAP Import: UNSEEN messages %s" % (messages)) ```` ![image](https://user-images.githubusercontent.com/21357789/111165454-535f9780-859f-11eb-9997-134fcedf5f7e.png) - However, fetched e-mails don't seem to have an attachement. This is weird, since my sent e-mails clearly have! ```` messages_structure = imap_client.fetch(messages, ['BODYSTRUCTURE']) for uid, structure in messages_structure.items(): if not contains_attachments(structure[b'BODYSTRUCTURE']): print('no attach - weird, I did sent a file!') ```` ![image](https://user-images.githubusercontent.com/21357789/111165968-ed274480-859f-11eb-8225-e14cfc7169f9.png) @ciur: So this seems to be a bug in your validation `contains_attachments()`. The mails are removed from the `messages` object and therefore not processed - although having an attachement! **Note**: The IMAPClient library provides a [special gmail search function](https://imapclient.readthedocs.io/en/2.1.0/_modules/imapclient/imapclient.html#IMAPClient.gmail_search). Maybe we can utilize this instead of `contains_attachments()` **for gmail specifically** to fix the issue? Haven't had a look at your `contains_attachments()` implementation yet. ```` # This method only works for IMAP servers that support X-GM-RAW, which is only likely to be Gmail messages = imap_client.gmail_search("has:attachment in:unread") ````

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@ciur commented on GitHub (Mar 16, 2021):

@l4rm4nd, thanks for investigation; I will have a look as well.

@ciur commented on GitHub (Mar 16, 2021): @l4rm4nd, thanks for investigation; I will have a look as well.

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@ciur commented on GitHub (Mar 20, 2021):

Maybe we can utilize this instead of contains_attachments() for gmail specifically to fix the issue? Haven't had a look at your contains_attachments() implementation yet.

No. By adding gmail, gmx... etc specific code is simple no go. Doing so, long term will lead to very brittle code base.

I think the problem is in contains_attachments method - it just looks wrong and difficult to understand how it does what it is supposed to do.

I will refactor contains_attachments function which hopefully will fix this issue.

@ciur commented on GitHub (Mar 20, 2021): > Maybe we can utilize this instead of contains_attachments() for gmail specifically to fix the issue? Haven't had a look at your contains_attachments() implementation yet. No. By adding gmail, gmx... etc specific code is simple no go. Doing so, long term will lead to very brittle code base. I think the problem is in [contains_attachments](https://github.com/papermerge/papermerge-core/blob/a82afd6ffd916dba8d94d1b16ec5ce111c18bdb8/papermerge/core/importers/imap.py#L82) method - it just looks wrong and difficult to understand how it does what it is supposed to do. ![contains-attachments](https://user-images.githubusercontent.com/24827601/111862325-f07e4f80-8954-11eb-8ca0-4d10da800818.png) I will refactor ``contains_attachments`` function which hopefully will fix this issue.

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@ciur commented on GitHub (Apr 6, 2021):

Fix is now part of version 2.0

@ciur commented on GitHub (Apr 6, 2021): Fix is now part of [version 2.0](https://github.com/ciur/papermerge/releases/tag/v2.0.0)

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@guim31 commented on GitHub (Apr 24, 2021):

I post here because i configured my papermerge.conf like this :

IMPORT_MAIL_HOST ="imap.gmail.com"
IMPORT_MAIL_USER ="gu@gmail.com"
IMPORT_MAIL_PASS ="xxxx"
IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com"

The thing is : ALL my mails with attachments are imported by papermerge... Not only the ones coming from FromBrotherDevice@brother.com

Is there any way to fix this ?

@guim31 commented on GitHub (Apr 24, 2021): I post here because i configured my papermerge.conf like this : IMPORT_MAIL_HOST ="imap.gmail.com" IMPORT_MAIL_USER ="gu@gmail.com" IMPORT_MAIL_PASS ="xxxx" IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com" The thing is : ALL my mails with attachments are imported by papermerge... Not only the ones coming from FromBrotherDevice@brother.com Is there any way to fix this ?

kerem commented

2026-02-25 21:31:35 +03:00

Author

Owner

@citizen-17 commented on GitHub (Oct 23, 2021):

IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com"

You have to set IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com" to True, IMPORT_MAIL_BY_USER = True and then configure the rest in the papermerge webui

@citizen-17 commented on GitHub (Oct 23, 2021): > IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com" You have to set IMPORT_MAIL_BY_USER = "FromBrotherDevice@brother.com" to True, IMPORT_MAIL_BY_USER = True and then configure the rest in the papermerge webui