[PR #219] [MERGED] Introduce document pipelines #567

New issue

Closed

opened 2026-02-25 21:32:15 +03:00 by kerem · 0 comments

kerem commented

2026-02-25 21:32:15 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/ciur/papermerge/pull/219
Author: @francescocarzaniga
Created: 11/20/2020
Status: ✅ Merged
Merged: 11/22/2020
Merged by: @ciur

Base: master ← Head: document_pipeline

📝 Commits (9)

7f8a56a Added import pipeline
1ae379e Spacing
6568353 Add python-magic to requirements
54c96a8 Merge branch 'master' into document_pipeline
7f56f2d Refactored IMAP and LOCAL importer for pipeline
901922b Removed conflicting test
333d660 Stub for pluggable pipelines
4117e48 Improved pipeline apps compatibility
c08739e Fixed func signature

📊 Changes

7 files changed (+315 additions, -174 deletions)

View changed files

📝 config/settings/base.py (+13 -1)
➕ papermerge/core/import_pipeline.py (+195 -0)
📝 papermerge/core/importers/imap.py (+51 -69)
📝 papermerge/core/importers/local.py (+26 -3)
📝 papermerge/core/views/documents.py (+28 -39)
➖ papermerge/test/test_imap.py (+0 -61)
📝 requirements/base.txt (+2 -1)

📄 Description

This PR introduces the concept of document pipeline. Document parts are the way to add plugin compatibility to the document model, while document pipelines have the objective to standardise and make plugins possible in the phase of document ingestion.

Currently the three different importers (web, imap, and local) have a different codebase, this PR tries to unify them. Moreover it is possible to extend the DefaultPipeline class to add functionality as an app, e.g. different file extensions.

Right now I only changed the web upload to work as a proof of concept. If @ciur thinks this is a good way to go forward I will replace the imap and local code as well.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/ciur/papermerge/pull/219 **Author:** [@francescocarzaniga](https://github.com/francescocarzaniga) **Created:** 11/20/2020 **Status:** ✅ Merged **Merged:** 11/22/2020 **Merged by:** [@ciur](https://github.com/ciur) **Base:** `master` ← **Head:** `document_pipeline` --- ### 📝 Commits (9) - [`7f8a56a`](https://github.com/ciur/papermerge/commit/7f8a56a53c64d0997f62df7f003ca3c31f6e9ab2) Added import pipeline - [`1ae379e`](https://github.com/ciur/papermerge/commit/1ae379e4d3e7f61932bb4060722f88ab9ee01bf3) Spacing - [`6568353`](https://github.com/ciur/papermerge/commit/65683537273f9b60bb6509d52eadad86c8650bcb) Add python-magic to requirements - [`54c96a8`](https://github.com/ciur/papermerge/commit/54c96a8023924bc017dca33ce2727b1074e2e657) Merge branch 'master' into document_pipeline - [`7f56f2d`](https://github.com/ciur/papermerge/commit/7f56f2de2acc2c925dd54e5e4fa50346021aa41e) Refactored IMAP and LOCAL importer for pipeline - [`901922b`](https://github.com/ciur/papermerge/commit/901922b52e1d1f019fcc22c46c0395b272ac157e) Removed conflicting test - [`333d660`](https://github.com/ciur/papermerge/commit/333d6608c62d31608d9d5918846e93dd1d209483) Stub for pluggable pipelines - [`4117e48`](https://github.com/ciur/papermerge/commit/4117e48d5d5fa5fbd0bf79c05de3c634d65fecdc) Improved pipeline apps compatibility - [`c08739e`](https://github.com/ciur/papermerge/commit/c08739eef73ebea3ace96d113bc41ba123d12b83) Fixed func signature ### 📊 Changes **7 files changed** (+315 additions, -174 deletions) <details> <summary>View changed files</summary> 📝 `config/settings/base.py` (+13 -1) ➕ `papermerge/core/import_pipeline.py` (+195 -0) 📝 `papermerge/core/importers/imap.py` (+51 -69) 📝 `papermerge/core/importers/local.py` (+26 -3) 📝 `papermerge/core/views/documents.py` (+28 -39) ➖ `papermerge/test/test_imap.py` (+0 -61) 📝 `requirements/base.txt` (+2 -1) </details> ### 📄 Description This PR introduces the concept of document pipeline. Document parts are the way to add plugin compatibility to the document model, while document pipelines have the objective to standardise and make plugins possible in the phase of document ingestion. Currently the three different importers (web, imap, and local) have a different codebase, this PR tries to unify them. Moreover it is possible to extend the DefaultPipeline class to add functionality as an app, e.g. different file extensions. Right now I only changed the web upload to work as a proof of concept. If @ciur thinks this is a good way to go forward I will replace the imap and local code as well. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>