[PR #1440] [MERGED] kernel: Rewrite pthread emulation #2110

New issue

Closed

opened 2026-02-27 21:15:13 +03:00 by kerem · 0 comments

kerem commented

2026-02-27 21:15:13 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/shadps4-emu/shadPS4/pull/1440
Author: @raphaelthegreat
Created: 10/23/2024
Status: ✅ Merged
Merged: 11/21/2024
Merged by: @raphaelthegreat

Base: main ← Head: more-kernel

📝 Commits (10+)

8860a0b libkernel: Cleanup some function places
c878e69 kernel: Refactor thread functions
00b84b2 kernel: It builds
d0d8b5e kernel: Fix a bunch of bugs, kernel thread heap
77ff429 kernel: File cleanup pt1
3771a65 File cleanup pt2
f635043 File cleanup pt3
ad8a202 File cleanup pt4
ef3341c kernel: Add missing funcs
cef37be kernel: Add basic exceptions for linux

📊 Changes

104 files changed (+5511 additions, -3936 deletions)

View changed files

📝 CMakeLists.txt (+33 -20)
📝 src/common/debug.h (+1 -1)
➕ src/common/slab_heap.h (+163 -0)
📝 src/common/slot_vector.h (+0 -3)
➕ src/common/spin_lock.cpp (+53 -0)
➕ src/common/spin_lock.h (+33 -0)
📝 src/core/address_space.cpp (+139 -69)
📝 src/core/debug_state.cpp (+1 -1)
➕ src/core/file_sys/file.cpp (+114 -0)
📝 src/core/libraries/audio3d/audio3d.cpp (+4 -4)
📝 src/core/libraries/audio3d/audio3d_impl.cpp (+1 -1)
📝 src/core/libraries/avplayer/avplayer.cpp (+3 -7)
📝 src/core/libraries/avplayer/avplayer.h (+2 -2)
📝 src/core/libraries/avplayer/avplayer_common.cpp (+5 -13)
📝 src/core/libraries/avplayer/avplayer_common.h (+4 -6)
📝 src/core/libraries/avplayer/avplayer_file_streamer.cpp (+3 -7)
📝 src/core/libraries/avplayer/avplayer_file_streamer.h (+2 -4)
📝 src/core/libraries/avplayer/avplayer_impl.cpp (+11 -26)
📝 src/core/libraries/avplayer/avplayer_impl.h (+2 -6)
📝 src/core/libraries/avplayer/avplayer_source.cpp (+13 -33)

...and 80 more files

📄 Description

The current implementation of pthreads uses winpthreads on windows and pthread lib on linux. While it has served its purpose for quick prototyping, the code at the moment is very unorganized and messy. The emulator does not perform a lot of necessary checks to validate state. Note: This is linux only and draft at the moment

This has been in works for a while but aims to fully rewrite pthread primitives such as mutexes, condvars, rwlocks and threads to be accurate to the real freebsd libthr library. Thus we dont depend on an old library for it and have more control.

Mutexes/Rwlocks now implement all static initializers and error checks as libthr. For mutexes recursion is handled manually and std::timed_mutex is used from STL as a sync primitive, while std::shared_timed_mutex for rwlocks. The PthreadMutex struct behaves like a lockable type so it can be passed to std::condition_variable directly to ensure correct ownership tracking. Adaptive mutexes have been implemented with spinloops.
Condition variables are implemented on top of std::condtion_variable_any. It's a very slim wrapper at the moment but it works okay from testing. Before this PR is merged I want to experiment with implementing sleepqueues from libthr and using waitable atomics for sleeping threads. This will allow us to implement scePhtreadCondSignalTo properly.
Thread local storage has been improved. A major bug in __tls_get_addr has been uncovered and fixed where an incorrect module TLS region could be loaded. On main this is masked because all DTVs are filled on thread creation but this isn't what real kernel does so we don't do that anymore either. May fix crashes on games that load custom prx modules with TLS regions. DTVs are now also freed when a thread is destoyed which may improve memory leaks on applications that create and destroy many threads.
Thread management has been completely reworked to be more accurate. Cleanup functions are accurate to original library and thread-safe, thread joining/exiting are also performed manually with cancellation points in mind. Pthread recycling and garbage collection is implemented as well. Thread objects are allocated with a slab heap using memory from the system reserved region, which mimics how actual kernel uses sceLibcInternal to allocate objects. This fixes Bloodborne on Linux freezes and removed the need for the annoying hack.

TLS allocation is now performed by the caller thread, while the target thread simply sets the guest tcb. This means that calling guest code from host threads is a bit more tricky as we can't easily setup TLS without the help of the kernel. For this a Thread class has been added in kernel which mimics an std::jthread like interface and uses pthread call underneath. This is useful for avplayer HLE.

Thread stack allocation is also implemented but not wired up yet as it needs to more testing. This may help games like TLOU2 that query thread stacks from the memory system.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/shadps4-emu/shadPS4/pull/1440 **Author:** [@raphaelthegreat](https://github.com/raphaelthegreat) **Created:** 10/23/2024 **Status:** ✅ Merged **Merged:** 11/21/2024 **Merged by:** [@raphaelthegreat](https://github.com/raphaelthegreat) **Base:** `main` ← **Head:** `more-kernel` --- ### 📝 Commits (10+) - [`8860a0b`](https://github.com/shadps4-emu/shadPS4/commit/8860a0bbd5210a08a06ef9a3337d7b8bccf3815e) libkernel: Cleanup some function places - [`c878e69`](https://github.com/shadps4-emu/shadPS4/commit/c878e692708fc10c2a8f754b03c766d4c1a3282e) kernel: Refactor thread functions - [`00b84b2`](https://github.com/shadps4-emu/shadPS4/commit/00b84b2c7f22c0ef4038d838400cfc0708aedd25) kernel: It builds - [`d0d8b5e`](https://github.com/shadps4-emu/shadPS4/commit/d0d8b5eee52f92412fac0b16da169f7f1ac6dfcb) kernel: Fix a bunch of bugs, kernel thread heap - [`77ff429`](https://github.com/shadps4-emu/shadPS4/commit/77ff4290d6178a356d92f0882d537bc488f1f46d) kernel: File cleanup pt1 - [`3771a65`](https://github.com/shadps4-emu/shadPS4/commit/3771a655bf66c1b5194eece88ee8b2762c510313) File cleanup pt2 - [`f635043`](https://github.com/shadps4-emu/shadPS4/commit/f6350433cca349f1332bcf1e21b3fe00ee075785) File cleanup pt3 - [`ad8a202`](https://github.com/shadps4-emu/shadPS4/commit/ad8a2022ec80102d4b3d5138a5ea7b1b07dbac6b) File cleanup pt4 - [`ef3341c`](https://github.com/shadps4-emu/shadPS4/commit/ef3341c78c7ce59478d2e780856f9cd7ffbb94ce) kernel: Add missing funcs - [`cef37be`](https://github.com/shadps4-emu/shadPS4/commit/cef37be33d31732730ca9aa7ff3b2880df476788) kernel: Add basic exceptions for linux ### 📊 Changes **104 files changed** (+5511 additions, -3936 deletions) <details> <summary>View changed files</summary> 📝 `CMakeLists.txt` (+33 -20) 📝 `src/common/debug.h` (+1 -1) ➕ `src/common/slab_heap.h` (+163 -0) 📝 `src/common/slot_vector.h` (+0 -3) ➕ `src/common/spin_lock.cpp` (+53 -0) ➕ `src/common/spin_lock.h` (+33 -0) 📝 `src/core/address_space.cpp` (+139 -69) 📝 `src/core/debug_state.cpp` (+1 -1) ➕ `src/core/file_sys/file.cpp` (+114 -0) 📝 `src/core/libraries/audio3d/audio3d.cpp` (+4 -4) 📝 `src/core/libraries/audio3d/audio3d_impl.cpp` (+1 -1) 📝 `src/core/libraries/avplayer/avplayer.cpp` (+3 -7) 📝 `src/core/libraries/avplayer/avplayer.h` (+2 -2) 📝 `src/core/libraries/avplayer/avplayer_common.cpp` (+5 -13) 📝 `src/core/libraries/avplayer/avplayer_common.h` (+4 -6) 📝 `src/core/libraries/avplayer/avplayer_file_streamer.cpp` (+3 -7) 📝 `src/core/libraries/avplayer/avplayer_file_streamer.h` (+2 -4) 📝 `src/core/libraries/avplayer/avplayer_impl.cpp` (+11 -26) 📝 `src/core/libraries/avplayer/avplayer_impl.h` (+2 -6) 📝 `src/core/libraries/avplayer/avplayer_source.cpp` (+13 -33) _...and 80 more files_ </details> ### 📄 Description The current implementation of pthreads uses winpthreads on windows and pthread lib on linux. While it has served its purpose for quick prototyping, the code at the moment is very unorganized and messy. The emulator does not perform a lot of necessary checks to validate state. Note: This is linux only and draft at the moment This has been in works for a while but aims to fully rewrite pthread primitives such as mutexes, condvars, rwlocks and threads to be accurate to the real freebsd libthr library. Thus we dont depend on an old library for it and have more control. * Mutexes/Rwlocks now implement all static initializers and error checks as libthr. For mutexes recursion is handled manually and std::timed_mutex is used from STL as a sync primitive, while std::shared_timed_mutex for rwlocks. The PthreadMutex struct behaves like a lockable type so it can be passed to std::condition_variable directly to ensure correct ownership tracking. Adaptive mutexes have been implemented with spinloops. * Condition variables are implemented on top of std::condtion_variable_any. It's a very slim wrapper at the moment but it works okay from testing. Before this PR is merged I want to experiment with implementing sleepqueues from libthr and using waitable atomics for sleeping threads. This will allow us to implement scePhtreadCondSignalTo properly. * Thread local storage has been improved. A major bug in __tls_get_addr has been uncovered and fixed where an incorrect module TLS region could be loaded. On main this is masked because all DTVs are filled on thread creation but this isn't what real kernel does so we don't do that anymore either. May fix crashes on games that load custom prx modules with TLS regions. DTVs are now also freed when a thread is destoyed which may improve memory leaks on applications that create and destroy many threads. * Thread management has been completely reworked to be more accurate. Cleanup functions are accurate to original library and thread-safe, thread joining/exiting are also performed manually with cancellation points in mind. Pthread recycling and garbage collection is implemented as well. Thread objects are allocated with a slab heap using memory from the system reserved region, which mimics how actual kernel uses sceLibcInternal to allocate objects. This fixes Bloodborne on Linux freezes and removed the need for the annoying hack. TLS allocation is now performed by the caller thread, while the target thread simply sets the guest tcb. This means that calling guest code from host threads is a bit more tricky as we can't easily setup TLS without the help of the kernel. For this a Thread class has been added in kernel which mimics an std::jthread like interface and uses pthread call underneath. This is useful for avplayer HLE. Thread stack allocation is also implemented but not wired up yet as it needs to more testing. This may help games like TLOU2 that query thread stacks from the memory system. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>