mirror of
https://github.com/d99kris/nmail.git
synced 2026-04-26 09:46:01 +03:00
[GH-ISSUE #45] [Question] Workflow to search content in cache using external tools #44
Labels
No labels
bug
enhancement
pull-request
question
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/nmail#44
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Kabouik on GitHub (Jun 7, 2020).
Original GitHub issue: https://github.com/d99kris/nmail/issues/45
Originally assigned to: @d99kris on GitHub.
I was investigating how I could search for strings in my cached emails using third party tools until
nmailhas a built-in search. Of course this wouldn't allow using the search result to do any actions like forward, reply, move or delete, but at least that would be enough to find some information I'm looking for in a message.With some scripting-fu, it should be possible to combine
fzffuzzy search withgreporripgrepand open a new terminal to search for a string in all files of a given directory. This could go well with #42 with a custom key to call this script directly fromnmailand show the new terminal to enter the string and see the results. Another way in my case (but that would be very specific to my workflow and the softwares I use) would be to open a new terminal runningnnnfile manager in theimapfolder, and then use a plugin script to do the fuzzy search on content directly innnnwith instant text previsualization:Here I just selected the .eml file manually, the idea would be to fuzzy search on content while in the folder.
nnnalready has fuzzy search built-in, but only on file names. But anyway, just an example, it's way too specific to make it the normal workflow.There's an extra difficulty if cache encrypting is set to
1, because then we need also something likeopenssl enc -d -aes-256-cbc -md sha1 -in ~/.nmail/cache/imap/*prior to the string search.This would be very hacky and shouldn't by any mean be something recommended or endorsed by
nmail, but I'm just asking candidly in case you would have some ideas on how to do such a thing. While being unofficial, hacky, and dependent on other tools, it could have a use untilnmailcan search and show emails by itself.@Kabouik commented on GitHub (Jun 7, 2020):
rgadoesn't sound too bad, and it can be integrated withfzf, which avoids cluttering the terminal when there are many hits.@Kabouik commented on GitHub (Jun 8, 2020):
See a short, basic, example with
nmailhere. Here I opened thetmuxpane in advance for the Asciinema cast, but the idea would be to open a new terminal or pane only when searching.Not ideal, of course, but useable until a more elegant solution is implemented. Also, it is about 27294 faster than my other email clients, and the ability to open the file in
$EDITORis not bad.Problems I can think of:
ripgrepandfzfdependencies; they are widely available andripgrepspeed should make it a standard at some point, but that contradictsnmailminimal style.sedblack magic but I don't know how to do it.passcould be used to convey the password?).Anyway, that is not meant to be an official solution, but thought it might be useful, and/or give ideas for the official implementation (no pressure).
For reference, here are the two commands I set up in
~/.bashrcto search in messages or just in headers:@d99kris commented on GitHub (Jun 9, 2020):
Thanks for sharing, this is very cool 👍
Yeah, email search is the one major feature that needs to be added next. I would not rule out using an external tool (they are likely more performant than I can implement), however - I'd like to also support searching encrypted cache, as well as IMAP search (i.e. not cached messages) - and optimally a solution would seamlessly encompass all these.
But yeah it's good inspiration to see this, and possibly it can be enabled within nmail with #42. I generally don't like navigating emails using panes, but I can see the usefulness when navigating search results. Will see if I can incorporate these concepts in the internal search support.
I've started a little on the search functionality, but so far only the IMAP side. Next is the UI, and lastly I was planning to add the local cache search.
@Kabouik commented on GitHub (Jun 9, 2020):
Totally, like the simplicity of
nmailand using panes would turn it into something else really. It doesn't bother me for searches because they are only punctual, I only call the extra pane/window when necessary (which the asciicast doesn't show because it cannot record new panes, I had to open it in advance). But I suppose if it was directly embedded innmail, it would work too. Note that the functions I used above are not perfect, they don't search for new keywords if you delete one and then type a new one in the fzf interface. Yet this should, fzf definitely supports that; this is just me failing at coding the functions properly.If the search occurs in the main window, it is important however to provide a way to get back to the current screen afterwards when closing the search. One reason for that is a frequent reason to search for old emails is we are searching for detailed information in a previous discussion when composing a new message, so it should be made easy to multitask between the current screen and the search screen/search results.
Regarding the underlying tools,
ripgrepis blazzing fast andfzfallows some nice, dynamic display as you type. Their combination makes for something that is user friendly, visually appealing (and configurable) and very efficient at the same time. Those who master regex could even use their mighty powers.rgawould allow searching in attachments too, but I suppose this would span a bit beyond the scope ofnmail, and it would only work if people store their attachments in thenmailfolder anyway because we I don't think we wantnmailsearch to drive into other folders.I don't know how that could integrate with IMAP search, but that's a good point I forgot indeed. I set
nmailto automatically fetch headers by default and.emlwhen opening the messages, which is why I came up with two different search commands. Searching headers in my cache allows searching in all my emails, but of course it won't find matches in message bodies. I also used to use cache encryption but disabled it for the tests. However I don't think this would be incompatible, the decrypt command could probably be nested in the search function and the passwoord fed to it whilenmailis running bypassor some other tools (?).I'm looking forward to seeing how this will work in
nmail, but I understand that it's a huge milestone which may take some time (which is why I went into local hacks in the mean time).@d99kris commented on GitHub (Aug 2, 2020):
FYI - I've now implemented basic email search functionality within
nmailin commit6b37a3b.I have not looked any further into enabling search via external tools, but I just wanted to let you know about the built-in search added.
@Kabouik commented on GitHub (Aug 4, 2020):
Awesome. Great work, that's a huge milestone. I tried it a bit with some dummy searches and it seems to work pretty well. It can only search emails in messages that are already fetched at the moment, not from IMAP yet, is that correct?
I see there are multiple new features and options in
main.conftoo, I need to add them to my configuration files. Cool stuff.@d99kris commented on GitHub (Aug 4, 2020):
Yes, it's only searching locally cached emails at the moment. I've added a command for full sync
sso the user can trigger a syncing of all emails at run-time.I have IMAP search implementation in progress, but I haven't yet figured out how to present it to users. Local search is magnitudes faster so I don't want to slow that down. I guess one option is to just have different keyboard shortcut for local vs. IMAP search. I'll need to think about this and do some testing before I can add IMAP search.
On 2020-08-04 17:28 Kabouik notifications@github.com wrote:
@Kabouik commented on GitHub (Aug 4, 2020):
I thought that new command was related to this, yes. Already triggered a full-sync after noticing search was limited to cached emails.
Sounds good. Maybe you could use some kind of decision tree if it is detected that not all emails have been cached? For instance, user initiating a search with an account with partially cached messages would get this in the status bar: "1234 emails are not cached yet. Search [o]nline with IMAP (slow), update cache with a full [s]ynchronization, or [i]gnore and search in cached emails only?".
I am not sure how to deal with "prefetch_level=0|1" though, as getting this prompt all the time would get old really quick. Or there could be a "search_preset=online|offline|update-cache" option, but perhaps there is no need to clutter
main.confwith an additional option if IMAP search can be implemented in a different way.[Edit] Also not sure if the cache check should be per folder or per account. Maybe this is a question for a future version when/if the search function gets some folder options.
On 2020-08-04 16:49 Kristofer Berggren notifications@github.com wrote:
@Kabouik commented on GitHub (Aug 11, 2020):
Can you tell how emails matching the search term are sorted? I noticed it's not sorted by date, so I thought it could be by relevance first and date second. I tested it by searching for a keyword that appears in a list of automatic emails (all identical), and yet the search results are still ordered by something else than date despite the identical content. Anyway I don't think relevance would be possible if
xapianonly looks for exact matches.@d99kris commented on GitHub (Aug 16, 2020):
Currently
nmailuses default search result order of Xapian, which is sorting by relevance - according to https://xapian.org/docs/apidoc/html/classXapian_1_1Enquire.html at least.Admittedly I have not evaluated how Xapian determines relevance. For a simple two term search (hello world) I suspect it ranks matches of "hello world" first, and then emails containing both "hello" and "world" and lastly emails containing either "hello" or "world". But it's just my assumption.
I'm also starting to think that it perhaps makes more sense to sort the results by date, as that's what some other emails clients do.
@Kabouik commented on GitHub (Aug 18, 2020):
Thanks a lot. It's weird that Xapian finds a way to sort by relevance (other than date) when we use only one search term and that term is an exact match. Defaulting to date sorting could be more straightforward indeed, or alternatively add a keybinding in the search results to change the sorting (?).
Do you know if Xapian would support searching for statuses, like read/unread?
@d99kris commented on GitHub (Aug 18, 2020):
Hi @Kabouik - I've actually just now (few minutes ago) pushed a couple of changes (improvements I hope) to the search function. In essence they are:
ANDby defaultRegarding your question on searching for status, it's not very simple to support, but it's certainly possible. I understand you'd like to be able to filter to only see unread email for example?
@Kabouik commented on GitHub (Aug 18, 2020):
Nice changes, I'll pull the update on my devices tomorrow!
Yes, I was looking for a way to just show read or unread emails, and possibly restrict searches to either status, I was wondering if this could be done with Xapian, but maybe that would not be the simplest way. No hurry anyway,
nmailalready is already quite nice for every day use now!@Kabouik commented on GitHub (Sep 30, 2020):
Just wanted to say that I am impressed by how fast and efficient the local search is. My colleague looking over my shoulder when we were working on something together was impressed (and jealous) too. :]
@d99kris commented on GitHub (Oct 3, 2020):
Thanks, that's great to hear! :)
On 2020-09-30 23:16 Kabouik notifications@github.com wrote:
@d99kris commented on GitHub (Jan 12, 2021):
I've been considering the support for searching cache using external tools, and although the proof of concept you provided is very impressive, I've leaning towards that this lies outside the scope of nmail. And that other features / bug fixes are more important. So I will proceed to close this issue for now. Please let me know if you have any concerns.
Separately I will be working on IMAP search, for the case when user does not have all messages locally cached.
@d99kris commented on GitHub (Feb 27, 2021):
On a slightly related note - just wanted to share that support for finding text in message view was added in
4d4973c- the default keybindings are/forfindand?forfind next.If one is using the search function (
/in list view) to search for emails, and then opens up one of the matches, it is now possible to press?to perform find inside the email using the same search query, and effectively find the matching occurrences within the email text (assuming plain text word / phrase search - the "in-message find" does not deal with AND, OR, +, -, etc).@Kabouik commented on GitHub (Feb 27, 2021):
That's excellent, thanks. I was sofar using my $EDITOR to perform searches within the body of emails, but this will be convenient. I will just change the default "next" keybinding to match that of my usual editor (n for next, a-n for previous).
On 2021-02-27 12:44 Kristofer Berggren notifications@github.com wrote: