kerem commented

Owner

Originally created by @Monirzadeh on GitHub (Oct 29, 2023).
Original GitHub issue: https://github.com/go-shiori/shiori/issues/771

this feature request a better way to import from other service as we talk about them in different issue and discussion like:
#768 #762

What is the problem?

currently shiori just can import some service like wallabag and pocket from terminal.
we need a better solution that solve multiple problem

we should done that in webui too.
currently command-line import them from specific functions inside https://github.com/go-shiori/shiori/tree/master/internal/cmd that is too more specific. we should combine all of them in one general function (terminal and webui) for easier maintain
in webui we should have a system like notification to inform user from import progress
maintaining of this should not be complicated

What we can do?

user get export from any service
convert that to specific format shiori understand (community can help to create script for convert different service to the same format)
shiori import that format
with this workflow we don't change code each time to support new service.

What about UI?

when user upload file and press import in settings menu. while it importing a bell icon show up above logout buttom.
maybe it can show progress as a ring around that. when it is finish bell icon disappear.

What should be consider?

some service has reading status and tag. how should we handle that? (as shiori don't have reading status yet)

currently showing progress percentage in UI is the problem that i can't imperilment yet.
is there anybody better idea for import functionality?

Originally created by @Monirzadeh on GitHub (Oct 29, 2023). Original GitHub issue: https://github.com/go-shiori/shiori/issues/771 this feature request a better way to import from other service as we talk about them in different issue and discussion like: #768 #762 # What is the problem? currently shiori just can import some service like wallabag and pocket from terminal. we need a better solution that solve multiple problem 1. we should done that in webui too. 2. currently command-line import them from specific functions inside `https://github.com/go-shiori/shiori/tree/master/internal/cmd` that is too more specific. we should combine all of them in one general function (terminal and webui) for easier maintain 3. in webui we should have a system like notification to inform user from import progress 4. maintaining of this should not be complicated # What we can do? 1. user get export from any service 2. convert that to specific format shiori understand (community can help to create script for convert different service to the same format) 3. shiori import that format with this workflow we don't change code each time to support new service. # What about UI? when user upload file and press import in settings menu. while it importing a bell icon show up above logout buttom. maybe it can show progress as a ring around that. when it is finish bell icon disappear. ![bell](https://github.com/go-shiori/shiori/assets/25131576/1b6649b9-1464-4578-a2f8-785f67771dc2) # What should be consider? some service has reading status and tag. how should we handle that? (as shiori don't have reading status yet) currently showing progress percentage in UI is the problem that i can't imperilment yet. is there anybody better idea for import functionality?

kerem added the

type:enhancement

label

2026-02-25 23:34:04 +03:00

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@fmartingr commented on GitHub (Nov 25, 2023):

I'm going to do a brain dump here on my thoughts on this issue:

The scope of the import/export for us (Shiori) should be allowing users to export and import data in a Shiori format can understand. This is useful when someone is switching provider or database engine. We need to pave a way for users to create a bundle of some structured data format file that we can read and import into the database. It would be nice if the user could toggle the export of files (allowing a way smaller file size if we only want the bookmarks and not the archives/thumbs/etc). if we do this, we need to take into account #675.
- We should isolate the logic outside the CLI/API, potentially in a domain or even a separate package.
If we want to allow users to do these actions via the Web UI we need to implement some kind of job system that shiori can use to run tasks in the background. This could be leveraged to create archives/ebooks/download content as well, instead of just firing a goroutine and call it a day. This is more complex than it seems, since this are jobs and we need to display information from them to the user we need to store data in the database and add a new way in the UI to show when a job is running.
- I like your idea of having that in the UI, but it would need to be expanded to display how many jobs are running and potentially adding a new page that shows the list of jobs and their status.
Once we have an import/export system that Shiori understands in place, we can open up to the community the creation of the conversions from other services to Shiori. We can participate in some of them or migrate code we already have, but maintaining them long-term is not feasible for us.

@fmartingr commented on GitHub (Nov 25, 2023): I'm going to do a brain dump here on my thoughts on this issue: - The scope of the import/export for us (Shiori) should be allowing users to export and import data in a Shiori format can understand. This is useful when someone is switching provider or database engine. We need to pave a way for users to create a bundle of some structured data format file that we can read and import into the database. It would be nice if the user could toggle the export of files (allowing a way smaller file size if we only want the bookmarks and not the archives/thumbs/etc). if we do this, we need to take into account #675. - We should isolate the logic outside the CLI/API, potentially in a domain or even a separate package. - If we want to allow users to do these actions via the Web UI we need to implement some kind of job system that shiori can use to run tasks in the background. This could be leveraged to create archives/ebooks/download content as well, instead of just firing a goroutine and call it a day. This is more complex than it seems, since this are jobs and we need to display information from them to the user we need to store data in the database and add a new way in the UI to show when a job is running. - I like your idea of having that in the UI, but it would need to be expanded to display how many jobs are running and potentially adding a new page that shows the list of jobs and their status. - Once we have an import/export system that Shiori understands in place, we can open up to the community the creation of the conversions from other services to Shiori. We can participate in some of them or migrate code we already have, but maintaining them long-term is not feasible for us.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@LLKoder commented on GitHub (Nov 26, 2023):

@Monirzadeh I know about that. My question about how I can use Shiori now. How I can move Chrome and Firefox Bookmarks to Shiori now any cases.

For example there is converter Netscape Bookmark File<->Chrome/Firefox Bookmark File or I have to use other steps. Or I have to write this converter. What would my steps to get temporary solution during you decide how it'll work? I do not know Go but I know other laguages include SQL.

I'm interested in Shiori 'cause it's bookmarks keeper in local cloud. But it'll become unuseful for me and for other users if we cannot move our bookmarks to Shiory from top browsers. I have no opportunity move ~100 thousand bookmarks by hand.

@LLKoder commented on GitHub (Nov 26, 2023): @Monirzadeh I know about that. My question about how I can use Shiori **now**. How I can move Chrome and Firefox Bookmarks to Shiori now any cases. For example there is converter Netscape Bookmark File<->Chrome/Firefox Bookmark File or I have to use other steps. Or I have to write this converter. What would my steps to get **temporary** solution during you decide how it'll work? I do not know Go but I know other laguages include SQL. I'm interested in Shiori 'cause it's bookmarks keeper in **local** cloud. But it'll become unuseful for me and for other users if we cannot move our bookmarks to Shiory from top browsers. I have no opportunity move ~100 thousand bookmarks by hand.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@Monirzadeh commented on GitHub (Nov 26, 2023):

@LLKoder you can use and modify scripts that we have in documentation.

for example

export your firefox bookmarks as html and move thet file to the shiori server
use bellow script to import your bookmarks to shiori

#!/bin/bash
grep -Eoi '<a [^>]+>' $1 | grep -Eo 'HREF="[^\"]+"' | cut -d'=' -f 2- | tr -d '"' | tac > firefox2shiori.txt

# Reading the URLs one by one and adding to shiori
while IFS= read -r line; do
    shiori add $line
done < firefox2shiori.txt

rm firefox2shiori.txt
exit 0

i don't have chrome base browser(so i don't have accesses to the bookmark export sample) to update script for that too but general concept is simple.
extract list of URLs from any backup file and loop them in shiori add command

@Monirzadeh commented on GitHub (Nov 26, 2023): @LLKoder you can use and modify scripts that we have in [documentation](https://github.com/go-shiori/shiori/blob/00582090e4ab0bb15ae0669a2a7e009747c21276/docs/Usage.md#improved-import-from-pocket). for example 1. export your firefox bookmarks as html and move thet file to the shiori server 2. use bellow script to import your bookmarks to shiori ```bash #!/bin/bash grep -Eoi '<a [^>]+>' $1 | grep -Eo 'HREF="[^\"]+"' | cut -d'=' -f 2- | tr -d '"' | tac > firefox2shiori.txt # Reading the URLs one by one and adding to shiori while IFS= read -r line; do shiori add $line done < firefox2shiori.txt rm firefox2shiori.txt exit 0 ``` i don't have chrome base browser(so i don't have accesses to the bookmark export sample) to update script for that too but general concept is simple. extract list of URLs from any backup file and loop them in `shiori add ` command

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@LLKoder commented on GitHub (Nov 27, 2023):

@Monirzadeh Thanks.
Can shiori add get not only link? Lets say title or other Shiori's parameters.
Or can Shiori get title for url automatically itself?

@LLKoder commented on GitHub (Nov 27, 2023): @Monirzadeh Thanks. Can `shiori add` get not only link? Lets say title or other Shiori's parameters. Or can Shiori get title for url automatically itself?

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@Monirzadeh commented on GitHub (Nov 27, 2023):

@Monirzadeh Thanks. Can shiori add get not only link? Lets say title or other Shiori's parameters. Or can Shiori get title for url automatically itself?

yes.
You can use -i flag shiori add $url -i $customName.
You can see more options here.
Unfortunately --help does not show this option, it may be helpful if you can send a PR for adding it to --help or at least to shiori documentation in docs.

@Monirzadeh commented on GitHub (Nov 27, 2023): > @Monirzadeh Thanks. Can `shiori add` get not only link? Lets say title or other Shiori's parameters. Or can Shiori get title for url automatically itself? yes. You can use `-i` flag `shiori add $url -i $customName`. You can see more options [here](https://github.com/go-shiori/shiori/blob/00582090e4ab0bb15ae0669a2a7e009747c21276/internal/cmd/add.go#L13C1-L29). Unfortunately `--help` does not show this option, it may be helpful if you can send a PR for adding it to `--help` or at least to shiori documentation in `docs`.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@mpr1255 commented on GitHub (Nov 30, 2023):

Major support for either an export functionality, or saving the files in some kind of format that ANY other software can read. I thought they might have been warc or mhtml, but nope. They are some kind of binary that only shiori can access?? And yet there is no API to use shiori from the CLI to push them out into anything that any other application can access? This makes Shiori a walled garden -- we can easily add things, but cannot get anything out except manually opening it and saving it as a file in the browser. DownThemAll! kinda works -- but they lose all the css, so not true archives.

This has come up before: https://github.com/go-shiori/shiori/discussions/337

I'm puzzled by the choice to save the files as (apparently?) a unique binary format? Why not just save them in the standard warc format, or perhaps mhtml? That is still a single file. PDFs are stored not as pdf either. I'm very curious about these design choices. Anyway, sadly cannot use shiori because of this. It's far better than the other tools I've been trying today but the fact that one cannot get one's data out makes it unusable for the main purpose: archiving stuff for reliably accessing later.

@mpr1255 commented on GitHub (Nov 30, 2023): Major support for either an export functionality, or saving the files in some kind of format that ANY other software can read. I thought they might have been warc or mhtml, but nope. They are some kind of binary that _only_ shiori can access?? And yet there is no API to use shiori from the CLI to push them out into anything that any other application can access? This makes Shiori a walled garden -- we can easily add things, but cannot get anything out except manually opening it and saving it as a file in the browser. DownThemAll! kinda works -- but they lose all the css, so not true archives. This has come up before: https://github.com/go-shiori/shiori/discussions/337 I'm puzzled by the choice to save the files as (apparently?) a unique binary format? Why not just save them in the standard warc format, or perhaps mhtml? That is still a single file. PDFs are stored not as pdf either. I'm very curious about these design choices. Anyway, sadly cannot use shiori because of this. It's far better than the other tools I've been trying today but the fact that one cannot get one's data out makes it unusable for the main purpose: archiving stuff for reliably accessing later.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@LLKoder commented on GitHub (Dec 2, 2023):

@Monirzadeh I add PR (https://github.com/go-shiori/shiori/pull/794) but it was blocked.

@LLKoder commented on GitHub (Dec 2, 2023): @Monirzadeh I add PR (https://github.com/go-shiori/shiori/pull/794) but it was blocked.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@LLKoder commented on GitHub (Dec 5, 2023):

@fmartingr @Monirzadeh I'm writing Python script for Import Chrome/Firefox Bookmarks to Shiori. Script will convert folder structure to tags structure. If it need I can do PR as temparary solve.

@LLKoder commented on GitHub (Dec 5, 2023): @fmartingr @Monirzadeh I'm writing Python script for Import Chrome/Firefox Bookmarks to Shiori. Script will convert folder structure to tags structure. If it need I can do PR as temparary solve.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@perdakovich commented on GitHub (Dec 7, 2023):

@LLKoder

Script will convert folder structure to tags structure.

But what if I have both folder and tag structure in my Firefox?

@perdakovich commented on GitHub (Dec 7, 2023): @LLKoder > Script will convert folder structure to tags structure. But what if I have both folder and tag structure in my Firefox?

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@perdakovich commented on GitHub (Dec 7, 2023):

@mpr1255 Completely agree, that's a mess! I'm hoping that the concept will change in the future..

@perdakovich commented on GitHub (Dec 7, 2023): @mpr1255 Completely agree, that's a mess! I'm hoping that the concept will change in the future..

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@LLKoder commented on GitHub (Dec 9, 2023):

But what if I have both folder and tag structure in my Firefox?

@Goosegit11 How do you have tags in Firefox? Chrome/Firefox don't support tags. Or am I wrong?

@LLKoder commented on GitHub (Dec 9, 2023): > But what if I have both folder and tag structure in my Firefox? @Goosegit11 How do you have tags in Firefox? Chrome/Firefox don't support tags. Or am I wrong?

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@LLKoder commented on GitHub (Dec 9, 2023):

@Goosegit11 I'll able to post a script here when I have a time to finish it, if need. Just developers not say merge the script to main branch or not. I not want to do waste PR.

@LLKoder commented on GitHub (Dec 9, 2023): @Goosegit11 I'll able to post a script here when I have a time to finish it, if need. Just developers not say merge the script to main branch or not. I not want to do waste PR.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@perdakovich commented on GitHub (Dec 9, 2023):

@LLKoder

@perdakovich commented on GitHub (Dec 9, 2023): @LLKoder ![image](https://github.com/go-shiori/shiori/assets/89806596/872e067a-195d-4079-b79c-c6fde1f8428f)

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@fmartingr commented on GitHub (Dec 10, 2023):

Hey people, trying to address all of the comments here. Thanks a lot to @Monirzadeh for chiming in and helping people move forward.

When I started using Shiori I noticed this "walled garden" problem, but for me at the time supposed no issue (I could just write a script to move things in and out), but now that I am here, the steps to move out of that problem are:

Improve the API. There has been problems with the API for some time, it was usable but suboptimal. The new API won't have session problems (thus, the extension will benefit from this) and will hopefully allow people to write it's own things on top of Shiori. (#658 #676 #657 #659 and probably others).
Move from WARC to Obelisk (#353). Obelisk will create archives in a single HTML file rather than a binary file. Still needs to be investigated if we can easily migrate all current WARC files to Obelisk readable files easily, but the idea would be to add support to Obelisk while keeping WARC, to allow backwards compatibility and make archiving modular, allowing to add more archiving options (like, for example, sending a webhook to archivebox).
Overhaul the import/export (this issue). Support exporting and importing to a machine-readable format that Shiori can use and others can convert to/from other bookmarking systems.

This is the current roardmap, and along the way some other issues had been found. We would love support in those areas if people are interested, I can provide guidance in issues for contributors.

If you have any more concerns about this please raise them here, the more we know before doing changes the better.

@fmartingr commented on GitHub (Dec 10, 2023): Hey people, trying to address all of the comments here. Thanks a lot to @Monirzadeh for chiming in and helping people move forward. When I started using Shiori I noticed this "walled garden" problem, but for me at the time supposed no issue (I could just write a script to move things in and out), but now that I am here, the steps to move out of that problem are: - Improve the API. There has been problems with the API for some time, it was usable but suboptimal. The new API won't have session problems (thus, the extension will benefit from this) and will _hopefully_ allow people to write it's own things on top of Shiori. (#658 #676 #657 #659 and probably others). - Move from WARC to Obelisk (#353). Obelisk will create archives in a single HTML file rather than a binary file. Still needs to be investigated if we can easily migrate all current WARC files to Obelisk readable files easily, but the idea would be to add support to Obelisk while keeping WARC, to allow backwards compatibility and make archiving modular, allowing to add more archiving options (like, for example, sending a webhook to archivebox). - Overhaul the import/export (this issue). Support exporting and importing to a machine-readable format that Shiori can use and others can convert to/from other bookmarking systems. This is the current roardmap, and along the way some other issues had been found. We would love support in those areas if people are interested, I can provide guidance in issues for contributors. If you have any more concerns about this please raise them here, the more we know before doing changes the better.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@LLKoder commented on GitHub (Dec 11, 2023):

@Goosegit11 Ahahaha... I have not known it till today. I don't have tags field when I save bookmarks, 'cause I have no active tags. But there is tags in bookmarks manager as I have seen. :) Thanks. I try add Firefox tags to my script.

@fmartingr Thanks to you. Say me pls how I understant, it all in plans. I.e. it is not released for now. And so as you know I write Python script. It'll export from Firefox/Crhome Bookmarks to Shiori CLI (with convert folder structure to tags) this is temporary solve as I see. Can I make PR or put it here is better?

@LLKoder commented on GitHub (Dec 11, 2023): @Goosegit11 Ahahaha... I have not known it till today. I don't have tags field when I save bookmarks, 'cause I have no active tags. But there is tags in bookmarks manager as I have seen. :) Thanks. I try add Firefox tags to my script. @fmartingr Thanks to you. Say me pls how I understant, it all in plans. I.e. it is not released for now. And so as you know I write Python script. It'll export from Firefox/Crhome Bookmarks to Shiori CLI (with convert folder structure to tags) this is temporary solve as I see. Can I make PR or put it here is better?

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@fmartingr commented on GitHub (Jan 13, 2024):

@fmartingr Thanks to you. Say me pls how I understant, it all in plans. I.e. it is not released for now. And so as you know I write Python script. It'll export from Firefox/Crhome Bookmarks to Shiori CLI (with convert folder structure to tags) this is temporary solve as I see. Can I make PR or put it here is better?

If you make a script to work with Shiori bookmarks you can create a repository/gist for it and we can link it from the documentation under community supported tooling.

@fmartingr commented on GitHub (Jan 13, 2024): > @fmartingr Thanks to you. Say me pls how I understant, it all in plans. I.e. it is not released for now. And so as you know I write Python script. It'll export from Firefox/Crhome Bookmarks to Shiori CLI (with convert folder structure to tags) this is temporary solve as I see. Can I make PR or put it here is better? If you make a script to work with Shiori bookmarks you can create a repository/gist for it and we can link it from the documentation under community supported tooling.

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@fmartingr commented on GitHub (Oct 17, 2024):

Coming back to this after a while. Does anyone have any hints into a open format for bookmarks interoperable between different projects? Or do we have to write our own? Is there something defined out there?

@fmartingr commented on GitHub (Oct 17, 2024): Coming back to this after a while. Does anyone have any hints into a open format for bookmarks interoperable between different projects? Or do we have to write our own? Is there something defined out there?

kerem commented

2026-02-25 23:34:04 +03:00

Author

Owner

@Monirzadeh commented on GitHub (Oct 18, 2024):

it is not exactly what you want but can be helpful (support range of device and programs)
floccus has an F-Droid version too but if i be honest i am not user of floccus myself
as it floccus is opensource you can look at how it handle different device and programs

don't forget to look at usage of WebDAV for sync bookmarks too.

@Monirzadeh commented on GitHub (Oct 18, 2024): it is not exactly what you want but can be helpful (support range of device and programs) [floccus](https://github.com/floccusaddon/floccus) has an [F-Droid version](https://f-droid.org/packages/org.handmadeideas.floccus/) too but if i be honest i am not user of floccus myself as it floccus is opensource you can look at how it handle different device and programs don't forget to look at usage of WebDAV for sync bookmarks too.

kerem referenced this issue

2026-02-25 23:34:47 +03:00

[PR #368] [MERGED] feat: async content download when creating via api #577

Rows
Columns

[GH-ISSUE #771] [meta/discussion] Import/Export functionality #368

What is the problem?

What we can do?

What about UI?

What should be consider?