[GH-ISSUE #120] WARC file support? #86

New issue

Closed

opened 2026-02-25 23:33:25 +03:00 by kerem · 2 comments

kerem commented

2026-02-25 23:33:25 +03:00

Owner

Originally created by @anarcat on GitHub (Nov 26, 2018).
Original GitHub issue: https://github.com/go-shiori/shiori/issues/120

it would be nice if this project would save files in WARC format. This allows collecting multiple page elements (JS, CSS, images, etc) in a single file. It's the standard archival format for web pages that is used by archive.org and many other programs. wget has support for it, for example.

This allows more faithful playback of pages than raw HTML because it includes HTTP headers which are sometimes capital in proper playback (if only, for example, for Content-type headers).

Originally created by @anarcat on GitHub (Nov 26, 2018). Original GitHub issue: https://github.com/go-shiori/shiori/issues/120 it would be nice if this project would save files in [WARC format](https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/). This allows collecting multiple page elements (JS, CSS, images, etc) in a single file. It's the standard archival format for web pages that is used by archive.org and many other programs. wget has support for it, for example. This allows more faithful playback of pages than raw HTML because it includes HTTP headers which are sometimes capital in proper playback (if only, for example, for `Content-type` headers).

kerem closed this issue

2026-02-25 23:33:25 +03:00

kerem commented

2026-02-25 23:33:25 +03:00

Author

Owner

@RadhiFadlillah commented on GitHub (Aug 27, 2019):

This issue is related with #108. So, archival is supported now, but not in warc format. This is done because IMHO warc is not really suitable for Shiori.

Correct me if I'm wrong but as far as I understand warc is simply a file that contains all of web page resources in text format. Since it's text based, it's not really easy to parse, and a bit slow to access a specific resource. I also not sure I could make the parser thread-safe.

Instead I just use bbolt which is key-value database. Thanks to this, read and write is easy and thread-safe. Not to mention the bbolt API is now fixed, so I believe it's safe enough to use for archival.

With that said, I believe the archival has been implemented albeit not in warc format, so I will close this issue. Feel free to reopen if needed.

@RadhiFadlillah commented on GitHub (Aug 27, 2019): This issue is related with #108. So, archival is supported now, but not in warc format. This is done because IMHO warc is not really suitable for Shiori. Correct me if I'm wrong but as far as I understand warc is simply a file that contains all of web page resources in text format. Since it's text based, it's not really easy to parse, and a bit slow to access a specific resource. I also not sure I could make the parser thread-safe. Instead I just use `bbolt` which is key-value database. Thanks to this, read and write is easy and thread-safe. Not to mention the `bbolt` API is now fixed, so I believe it's safe enough to use for archival. With that said, I believe the archival has been implemented albeit not in warc format, so I will close this issue. Feel free to reopen if needed.

kerem commented

2026-02-25 23:33:25 +03:00

Author

Owner

@neezer commented on GitHub (Jan 21, 2020):

If not using WARC proper, why not just use a file dump to a folder with the HTML, CSS, images, etc.? I'd like to make changes to the archive file because Shiori isn't correctly handling some things yet (#230, for example), but at the moment the archive exists as an opaque file—I don't have much of an idea about how to go about editing it manually.

I don't need wget replay or interoperability with existing WARC-processing tools like @anarcat, so I'm struggling to see the end-user value of a single-file archive: what does it deliver that a folder of assets wouldn't?

@neezer commented on GitHub (Jan 21, 2020): If not using WARC proper, why not just use a file dump to a folder with the HTML, CSS, images, etc.? I'd like to make changes to the archive file because Shiori isn't correctly handling some things yet (#230, for example), but at the moment the archive exists as an opaque file—I don't have much of an idea about how to go about editing it manually. I don't need `wget` replay or interoperability with existing WARC-processing tools like @anarcat, so I'm struggling to see the end-user value of a single-file archive: what does it deliver that a folder of assets wouldn't?