[GH-ISSUE #114] Thread 'test_server:udp:server' panic #60

Closed
opened 2026-03-07 22:18:26 +03:00 by kerem · 10 comments
Owner

Originally created by @liranringel on GitHub (Apr 9, 2017).
Original GitHub issue: https://github.com/hickory-dns/hickory-dns/issues/114

The test test_server_www_udp seems to leak the thread called test_server:udp:server and panic with that error:

thread 'test_server:udp:server' panicked at 'cannot create new TLS keys after the main thread has exited', C:\bot\slave\stable-dist-rustc-win-msvc-64\build\src\libstd\sys\windows\thread_local.rs:143

First occurred to me in appveyor, and I'm able to reproduce it on my local windows machine:
https://ci.appveyor.com/project/liran-ringel/trust-dns/build/1.0.4/job/33yskkn020b6ec0p

Originally created by @liranringel on GitHub (Apr 9, 2017). Original GitHub issue: https://github.com/hickory-dns/hickory-dns/issues/114 The test `test_server_www_udp` seems to leak the thread called `test_server:udp:server` and panic with that error: > thread 'test_server:udp:server' panicked at 'cannot create new TLS keys after the main thread has exited', C:\bot\slave\stable-dist-rustc-win-msvc-64\build\src\libstd\sys\windows\thread_local.rs:143 First occurred to me in appveyor, and I'm able to reproduce it on my local windows machine: https://ci.appveyor.com/project/liran-ringel/trust-dns/build/1.0.4/job/33yskkn020b6ec0p
kerem 2026-03-07 22:18:26 +03:00
  • closed this issue
  • added the
    bug:tests
    label
Author
Owner

@bluejekyll commented on GitHub (Apr 9, 2017):

Huh. I haven't noticed that anywhere else. Do you have a back trace possibly?

<!-- gh-comment-id:292820559 --> @bluejekyll commented on GitHub (Apr 9, 2017): Huh. I haven't noticed that anywhere else. Do you have a back trace possibly?
Author
Owner

@liranringel commented on GitHub (Apr 9, 2017):

When I don't set RUST_BACKTRACE=1, It stuck.
When I do set RUST_BACKTRACE=1, It just print that the panic happened, but does not display the backtrace.
Then I thought I need to pass --nocapture, but when I do that, It doesn't panic at all.

<!-- gh-comment-id:292822425 --> @liranringel commented on GitHub (Apr 9, 2017): When I don't set RUST_BACKTRACE=1, It stuck. When I do set RUST_BACKTRACE=1, It just print that the panic happened, but does not display the backtrace. Then I thought I need to pass `--nocapture`, but when I do that, It doesn't panic at all.
Author
Owner

@bluejekyll commented on GitHub (Apr 9, 2017):

Sounds like a race condition. I don't think I'm doing anything crazy there, so I wonder if there is a bug in the library.

<!-- gh-comment-id:292822800 --> @bluejekyll commented on GitHub (Apr 9, 2017): Sounds like a race condition. I don't think I'm doing anything crazy there, so I wonder if there is a bug in the library.
Author
Owner

@liranringel commented on GitHub (Apr 10, 2017):

Is that ok that the thread called test_server:udp:server is not joined? Threads are detached when JoinHandle is dropped, not terminated.

<!-- gh-comment-id:292824028 --> @liranringel commented on GitHub (Apr 10, 2017): Is that ok that the thread called `test_server:udp:server` is not joined? Threads are detached when `JoinHandle` is dropped, not terminated.
Author
Owner

@liranringel commented on GitHub (Apr 10, 2017):

Ok I set a breakpoint on panic_unwind::__rust_start_panic with windbg, that's the backtrace:
https://gist.github.com/liranringel/ac938f20e6beecf4b5182013460440dd

<!-- gh-comment-id:292825276 --> @liranringel commented on GitHub (Apr 10, 2017): Ok I set a breakpoint on `panic_unwind::__rust_start_panic` with windbg, that's the backtrace: https://gist.github.com/liranringel/ac938f20e6beecf4b5182013460440dd
Author
Owner

@bluejekyll commented on GitHub (Apr 10, 2017):

Woh. I totally misread the docs on that. That's a good catch. I should definitely be joining both threads.

<!-- gh-comment-id:292825401 --> @bluejekyll commented on GitHub (Apr 10, 2017): Woh. I totally misread the docs on that. That's a good catch. I should definitely be joining both threads.
Author
Owner

@bluejekyll commented on GitHub (Apr 10, 2017):

Not sure why it's a problem on windows though. It's only a few threads, and all should be cleaned up on process exit.

<!-- gh-comment-id:292825471 --> @bluejekyll commented on GitHub (Apr 10, 2017): Not sure why it's a problem on windows though. It's only a few threads, and all should be cleaned up on process exit.
Author
Owner

@bluejekyll commented on GitHub (Apr 10, 2017):

Never mind. Now I'm more confused as to why I have seen this on Linux or macOS. Anyway, I'll look at a patch tonight. Should I set up an app auto account to connect to this?

<!-- gh-comment-id:292826289 --> @bluejekyll commented on GitHub (Apr 10, 2017): Never mind. Now I'm more confused as to why I have seen this on Linux or macOS. Anyway, I'll look at a patch tonight. Should I set up an app auto account to connect to this?
Author
Owner

@liranringel commented on GitHub (Apr 10, 2017):

Sorry I didn't understand the last question.

According to the backtrace, the panic is raised here: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/windows/thread_local.rs#L69

It happens because TlsAlloc() returns TLS_OUT_OF_INDEXES (https://msdn.microsoft.com/en-us/library/windows/desktop/ms686801(v=vs.85).aspx).
That article says that TLS (thread-local storage) has a limit on windows:
https://msdn.microsoft.com/en-us/library/ms686749(VS.85).aspx
as opposed to linux: http://stackoverflow.com/a/1461048
So if there is a leak of the thread-local storage, it can explain why it happens only on windows.

<!-- gh-comment-id:292827507 --> @liranringel commented on GitHub (Apr 10, 2017): Sorry I didn't understand the last question. According to the backtrace, the panic is raised here: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/windows/thread_local.rs#L69 It happens because `TlsAlloc()` returns `TLS_OUT_OF_INDEXES` (https://msdn.microsoft.com/en-us/library/windows/desktop/ms686801(v=vs.85).aspx). That article says that TLS (thread-local storage) has a limit on windows: https://msdn.microsoft.com/en-us/library/ms686749(VS.85).aspx as opposed to linux: http://stackoverflow.com/a/1461048 So if there is a leak of the thread-local storage, it can explain why it happens only on windows.
Author
Owner

@bluejekyll commented on GitHub (Apr 10, 2017):

Thanks for the links. I guess the reason I'm confused is because the number of connections even with the leak (which we should fix) should only be 6 or so. But anyway, I'll get a fix in for the thread join.

<!-- gh-comment-id:292833414 --> @bluejekyll commented on GitHub (Apr 10, 2017): Thanks for the links. I guess the reason I'm confused is because the number of connections even with the leak (which we should fix) should only be 6 or so. But anyway, I'll get a fix in for the thread join.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/hickory-dns#60
No description provided.