[GH-ISSUE #176] panic: sync: WaitGroup is reused before previous Wait has returned #131

New issue

Closed

opened 2026-02-25 23:33:31 +03:00 by kerem · 4 comments

kerem commented

2026-02-25 23:33:31 +03:00

Owner

Originally created by @makefu on GitHub (Sep 25, 2019).
Original GitHub issue: https://github.com/go-shiori/shiori/issues/176

I have a couple of thousand bookmarks i want to archive. When running SHIORI_DIR=/var/lib/shiori shiori update at some point in time i get the following error.

...
[2173/7974] Failed to download http://www.heise.de/newsticker/meldung/l-f-Praepariertes-Mini-Fladenbrot-spaeht-geheime-Krypto-Schluessel-aus-2721449.html?wt_mc=rss.ho.beitrag.atom: Get http://www.heise.de/newsticker/meldung/l-f-Praepariertes-Mini-Fladenbrot-spaeht-geheime-Krypto-Schluessel-aus-2721449.html?wt_mc=rss.ho.beitrag.atom: dial tcp [2a02:2e0:3fe:1001:7777:772e:2:85]:80: i/o timeout                                                                                              
[2174/7974] Downloaded http://www.fky.org/restaurierung/Klabunde/Lacke.htm                                                                          
[2175/7974] Downloaded http://yakamo.org/?p=1506                                                                                                    
[2176/7974] Downloaded http://www.kaspersky.com/antivirus-removal-tool?form=1
panic: sync: WaitGroup is reused before previous Wait has returned

goroutine 1237523 [running]:
sync.(*WaitGroup).Wait(0xc00586f578)                                                                                                                 
        /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.12.9/share/go/src/sync/waitgroup.go:132 +0xae                                                                   
github.com/go-shiori/shiori/pkg/warc/internal/archiver.(*Archiver).StartArchiver.func1(0xc00586f560)                                                                     
        /build/go/src/github.com/go-shiori/shiori/pkg/warc/internal/archiver/archiver.go:39 +0x3e                                                                        
created by github.com/go-shiori/shiori/pkg/warc/internal/archiver.(*Archiver).StartArchiver                                                                              
        /build/go/src/github.com/go-shiori/shiori/pkg/warc/internal/archiver/archiver.go:37 +0x54

shiori version: master
Git rev: c77b17caf8
go version: go-1.12.9

Thanks for your efforts, i just tried out shiori 2 days ago and i really like the capabilities. I am somewhat looking for a self-hosted full-text search through all my bookmarks and shiori is what comes closest to that.

Originally created by @makefu on GitHub (Sep 25, 2019). Original GitHub issue: https://github.com/go-shiori/shiori/issues/176 I have a couple of thousand bookmarks i want to archive. When running `SHIORI_DIR=/var/lib/shiori shiori update` at some point in time i get the following error. ``` ... [2173/7974] Failed to download http://www.heise.de/newsticker/meldung/l-f-Praepariertes-Mini-Fladenbrot-spaeht-geheime-Krypto-Schluessel-aus-2721449.html?wt_mc=rss.ho.beitrag.atom: Get http://www.heise.de/newsticker/meldung/l-f-Praepariertes-Mini-Fladenbrot-spaeht-geheime-Krypto-Schluessel-aus-2721449.html?wt_mc=rss.ho.beitrag.atom: dial tcp [2a02:2e0:3fe:1001:7777:772e:2:85]:80: i/o timeout [2174/7974] Downloaded http://www.fky.org/restaurierung/Klabunde/Lacke.htm [2175/7974] Downloaded http://yakamo.org/?p=1506 [2176/7974] Downloaded http://www.kaspersky.com/antivirus-removal-tool?form=1 panic: sync: WaitGroup is reused before previous Wait has returned goroutine 1237523 [running]: sync.(*WaitGroup).Wait(0xc00586f578) /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.12.9/share/go/src/sync/waitgroup.go:132 +0xae github.com/go-shiori/shiori/pkg/warc/internal/archiver.(*Archiver).StartArchiver.func1(0xc00586f560) /build/go/src/github.com/go-shiori/shiori/pkg/warc/internal/archiver/archiver.go:39 +0x3e created by github.com/go-shiori/shiori/pkg/warc/internal/archiver.(*Archiver).StartArchiver /build/go/src/github.com/go-shiori/shiori/pkg/warc/internal/archiver/archiver.go:37 +0x54 ``` shiori version: master Git rev: c77b17caf8fcdf336adea33d0e4ac7ab13c10bc5 go version: go-1.12.9 Thanks for your efforts, i just tried out shiori 2 days ago and i really like the capabilities. I am somewhat looking for a self-hosted full-text search through all my bookmarks and shiori is what comes closest to that.

kerem

2026-02-25 23:33:31 +03:00

closed this issue
added the
priority:low

tag:huge-data

note:fixed?
labels

kerem commented

2026-02-25 23:33:31 +03:00

Author

Owner

@RadhiFadlillah commented on GitHub (Oct 4, 2019):

Hi @makefu, sorry for late response.

After looking around, it seems this issue is happened because of this line :

github.com/go-shiori/shiori@598ea9476d/pkg/warc/internal/archiver/archiver.go (L36-L41)

As you can see, when archiver is started, the WaitGroup started to Wait 1 second after the function called. That one second is done to make sure the WaitGroup has been added before it waits.

This, in hindsight, obviously really bad because sometimes the process took longer than 1 second, which in turn makes the WaitGroup started to Wait before there are any counter added to that wait group. Thanks to this, there will be case when the WaitGroup is already closed (because it finished waiting) but another Goroutine still trying to add counter to this wait group, which will result in panic like yours.

For now, I've simplified and fixed the code for archival to make sure this issue not happened anymore. For simplicity, I've also moved the warc package to its own repository.

Unfortunately, I'm still not really sure that it works perfectly, so I decided to put the fix for this issue on its own warc branch. With that said, if you don't mind, could you build the warc branch and test it out ? Hopefully it should be fixed by now.

Thanks !

@RadhiFadlillah commented on GitHub (Oct 4, 2019): Hi @makefu, sorry for late response. After looking around, it seems this issue is happened because of this line : https://github.com/go-shiori/shiori/blob/598ea9476d9cdcc29eaf74725a41e9159376e113/pkg/warc/internal/archiver/archiver.go#L36-L41 As you can see, when archiver is started, the `WaitGroup` started to `Wait` 1 second after the function called. That one second is done to make sure the `WaitGroup` has been added before it waits. This, in hindsight, obviously really bad because sometimes the process took longer than 1 second, which in turn makes the `WaitGroup` started to `Wait` before there are any counter added to that wait group. Thanks to this, there will be case when the `WaitGroup` is already closed (because it finished waiting) but another Goroutine still trying to add counter to this wait group, which will result in panic [like yours](https://stackoverflow.com/questions/39800700/waitgroup-is-reused-before-previous-wait-has-returned). For now, I've simplified and fixed the code for archival to make sure this issue not happened anymore. For simplicity, I've also moved the `warc` package to its own [repository](https://github.com/go-shiori/warc). Unfortunately, I'm still not really sure that it works perfectly, so I decided to put the fix for this issue on its own [`warc`](https://github.com/go-shiori/shiori/tree/warc) branch. With that said, if you don't mind, could you build the `warc` branch and test it out ? Hopefully it should be fixed by now. Thanks !

kerem commented

2026-02-25 23:33:31 +03:00

Author

Owner

@makefu commented on GitHub (Oct 5, 2019):

Hey, thanks for the reply, i really apprechiate it. I will try out your code in the next days and give feedback :)

@makefu commented on GitHub (Oct 5, 2019): Hey, thanks for the reply, i really apprechiate it. I will try out your code in the next days and give feedback :)

kerem commented

2026-02-25 23:33:31 +03:00

Author

Owner

@makefu commented on GitHub (Oct 9, 2019):

@RadhiFadlillah just wanted to get back to you after testing your changes. All ~8000 bookmarks got processed and archived (if they are still available).

Looking forward for this change to land in master 👍

@makefu commented on GitHub (Oct 9, 2019): @RadhiFadlillah just wanted to get back to you after testing your changes. All ~8000 bookmarks got processed and archived (if they are still available). Looking forward for this change to land in master :+1:

kerem commented

2026-02-25 23:33:31 +03:00

Author

Owner

@RadhiFadlillah commented on GitHub (Oct 9, 2019):

Merged in master at a3e1563.

@RadhiFadlillah commented on GitHub (Oct 9, 2019): Merged in master at [a3e1563](https://github.com/go-shiori/shiori/commit/a3e1563dff7aeaba97c3984fc7a3bd66743423eb).