[GH-ISSUE #261] Allocation of virtual memory backing file fails with small page file, crashes program #38

Closed
opened 2026-02-27 21:04:16 +03:00 by kerem · 12 comments
Owner

Originally created by @xezrunner on GitHub (Jul 7, 2024).
Original GitHub issue: https://github.com/shadps4-emu/shadPS4/issues/261

CreateFileMapping2() returns null after numerous successful attempts at starting a game. Rebooting the system and trying again resolves the issue for a while, but it eventually happens again.

I am unsure whether this is an issue on the emulator or Windows side. My machine has 8GB of RAM and a 14GB page file.

github.com/shadps4-emu/shadPS4@379ec554a8/src/core/address_space.cpp (L54-L58)

I have been attempting to get a game running through debugging, so I have the code up. I inserted my own code to break if the backing pointer is zero:

0

That assert hits and crashes, even when running the executable without a debugger.

Originally created by @xezrunner on GitHub (Jul 7, 2024). Original GitHub issue: https://github.com/shadps4-emu/shadPS4/issues/261 `CreateFileMapping2()` returns null after numerous successful attempts at starting a game. Rebooting the system and trying again resolves the issue for a while, but it eventually happens again. ~~I am unsure whether this is an issue on the emulator or Windows side. My machine has 8GB of RAM and a 14GB page file.~~ https://github.com/shadps4-emu/shadPS4/blob/379ec554a8aa6e683a3a094a9f4ce8ec6da3f3e9/src/core/address_space.cpp#L54-L58 I have been attempting to get a game running through debugging, so I have the code up. I inserted my own code to break if the backing pointer is zero: <img width="704" alt="0" src="https://github.com/shadps4-emu/shadPS4/assets/8061077/1d2b9ab0-10b2-432d-9a44-fa71522a379a"> That assert hits and crashes, even when running the executable without a debugger.
kerem closed this issue 2026-02-27 21:04:16 +03:00
Author
Owner

@raphaelthegreat commented on GitHub (Jul 7, 2024):

Call Common::GetLastErrorMsg to get a more detailed description of the error from the OS

<!-- gh-comment-id:2212493275 --> @raphaelthegreat commented on GitHub (Jul 7, 2024): Call Common::GetLastErrorMsg to get a more detailed description of the error from the OS
Author
Owner

@xezrunner commented on GitHub (Jul 7, 2024):

Call Common::GetLastErrorMsg to get a more detailed description of the error from the OS

Thanks, that did point me in the right direction!

It is weird how it works initially upon reboot, until some time passes where it just starts returning null.

image

<!-- gh-comment-id:2212495533 --> @xezrunner commented on GitHub (Jul 7, 2024): > Call Common::GetLastErrorMsg to get a more detailed description of the error from the OS Thanks, that did point me in the right direction! It is weird how it works initially upon reboot, until some time passes where it just starts returning null. ![image](https://github.com/shadps4-emu/shadPS4/assets/8061077/b194787f-3cae-43d8-a02a-0931211361b7)
Author
Owner

@xezrunner commented on GitHub (Jul 7, 2024):

I am not familiar with virtual memory management, but I read from Win32 docs:

If hFile is INVALID_HANDLE_VALUE, the calling process must also specify a size for the file mapping object in the dwMaximumSizeHigh and dwMaximumSizeLow parameters.
In this scenario, CreateFileMapping creates a file mapping object of a specified size that is backed by the system paging file instead of by a file in the file system.

I manually allocated a larger page file from 4GB to 12GB and that resolved the issue. Noticed that I was sort-of low on disk space, so the OS probably did not allocate a larger page file to conserve space.

Perhaps it would be beneficial to let it map an actual file instead of using the page file in cases where the page file is held small by the OS (so when this "insufficient resources" error occurs)?

<!-- gh-comment-id:2212499129 --> @xezrunner commented on GitHub (Jul 7, 2024): I am not familiar with virtual memory management, but I read from [Win32 docs](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-createfilemapping2#:~:text=In%20this%20scenario%2C%20CreateFileMapping%20creates%20a%20file%20mapping%20object%20of%20a%20specified%20size%20that%20is%20backed%20by%20the%20system%20paging%20file%20instead%20of%20by%20a%20file%20in%20the%20file%20system.): > If hFile is INVALID_HANDLE_VALUE, the calling process must also specify a size for the file mapping object in the dwMaximumSizeHigh and dwMaximumSizeLow parameters. > **In this scenario, CreateFileMapping creates a file mapping object of a specified size that is backed by the system paging file instead of by a file in the file system.** I manually allocated a larger page file from 4GB to 12GB and that resolved the issue. Noticed that I was sort-of low on disk space, so the OS probably did not allocate a larger page file to conserve space. <img src="https://github.com/shadps4-emu/shadPS4/assets/8061077/c5535866-b73a-4d7a-bef8-5194fec9a5ee" height="300" /> Perhaps it would be beneficial to let it map an actual file instead of using the page file in cases where the page file is held small by the OS (so when this "insufficient resources" error occurs)?
Author
Owner

@raphaelthegreat commented on GitHub (Jul 7, 2024):

This is not supposed to be a file per sense, its a file mapping so we can map portions of it to the reserved virtual address space. Having it write to disk will not be ideal

<!-- gh-comment-id:2212499898 --> @raphaelthegreat commented on GitHub (Jul 7, 2024): This is not supposed to be a file per sense, its a file mapping so we can map portions of it to the reserved virtual address space. Having it write to disk will not be ideal
Author
Owner

@xezrunner commented on GitHub (Jul 7, 2024):

This is not supposed to be a file per sense, its a file mapping so we can map portions of it to the reserved virtual address space. Having it write to disk will not be ideal

That makes sense. In that case, at least additional logging/non-destructive error handling (useful error message to increase page size?) would be useful here.

This crash seems to be happening early enough that the logging doesn't appear to be initialized yet - the console doesn't show anything and the assertion crashes further when it tries to call Common::Log::Impl::Stop().

<!-- gh-comment-id:2212500783 --> @xezrunner commented on GitHub (Jul 7, 2024): > This is not supposed to be a file per sense, its a file mapping so we can map portions of it to the reserved virtual address space. Having it write to disk will not be ideal That makes sense. In that case, at least additional logging/non-destructive error handling (useful error message to increase page size?) would be useful here. This crash seems to be happening early enough that the logging doesn't appear to be initialized yet - the console doesn't show anything and the assertion crashes further when it tries to call `Common::Log::Impl::Stop()`.
Author
Owner

@dima-xd commented on GitHub (Jul 14, 2024):

Now I am having the same issue

<!-- gh-comment-id:2227340100 --> @dima-xd commented on GitHub (Jul 14, 2024): Now I am having the same issue
Author
Owner

@Hermiten commented on GitHub (Oct 4, 2024):

Can I have an update on this issue ?Is it still revelant ?

<!-- gh-comment-id:2393128880 --> @Hermiten commented on GitHub (Oct 4, 2024): Can I have an update on this issue ?Is it still revelant ?
Author
Owner

@xezrunner commented on GitHub (Oct 4, 2024):

Can I have an update on this issue ?Is it still revelant ?

This appears to have been resolved by making the assert for the allocation also throw an error message from the OS itself in case of failure, which should say

Insufficient system resources exist to complete the requested service.

when there isn't enough virtual memory to allocate the backing handle on Windows.

github.com/shadps4-emu/shadPS4@76644a0169/src/core/address_space.cpp (L101-L105)

Ideally, a friendlier message could be shown in this particular error case, so that the user would know that they have to increase the virtual memory size, but this varies across platforms.

<!-- gh-comment-id:2393461584 --> @xezrunner commented on GitHub (Oct 4, 2024): > Can I have an update on this issue ?Is it still revelant ? This appears to have been resolved by making the assert for the allocation also throw an error message from the OS itself in case of failure, which should say > Insufficient system resources exist to complete the requested service. when there isn't enough virtual memory to allocate the backing handle on Windows. https://github.com/shadps4-emu/shadPS4/blob/76644a0169ffbba4e6c5d27c4506f858d715f5dc/src/core/address_space.cpp#L101-L105 Ideally, a friendlier message could be shown in this particular error case, so that the user would know that they have to increase the virtual memory size, but this varies across platforms.
Author
Owner

@Hermiten commented on GitHub (Oct 4, 2024):

Good idea, maybe you can put a PR with this and you will see if it's accepted
Sometimes a good log can be enough for user to understand what to do and improve their tests

<!-- gh-comment-id:2393504721 --> @Hermiten commented on GitHub (Oct 4, 2024): Good idea, maybe you can put a PR with this and you will see if it's accepted Sometimes a good log can be enough for user to understand what to do and improve their tests
Author
Owner

@xezrunner commented on GitHub (Oct 4, 2024):

Good idea, maybe you can put a PR with this and you will see if it's accepted Sometimes a good log can be enough for user to understand what to do and improve their tests

I was considering opening a PR for this, but I'm not sure how platforms other than Win32 are handled. It seems like the whole address_space.cpp file is only doing Win32 stuff, so it might even be a better idea to consolidate main virtual memory allocations (perhaps there's only one) into a more obvious place for all platforms first and handling the errors there.

That would make the scope of the PR larger, which is probably best left to those that are more familiar with the memory allocation-side of things.

<!-- gh-comment-id:2393563154 --> @xezrunner commented on GitHub (Oct 4, 2024): > Good idea, maybe you can put a PR with this and you will see if it's accepted Sometimes a good log can be enough for user to understand what to do and improve their tests I was considering opening a PR for this, but I'm not sure how platforms other than Win32 are handled. It seems like the whole `address_space.cpp` file is only doing Win32 stuff, so it might even be a better idea to consolidate main virtual memory allocations (perhaps there's only one) into a more obvious place for all platforms first and handling the errors there. That would make the scope of the PR larger, which is probably best left to those that are more familiar with the memory allocation-side of things.
Author
Owner

@polybiusproxy commented on GitHub (Oct 4, 2024):

I was considering opening a PR for this, but I'm not sure how platforms other than Win32 are handled. It seems like the whole address_space.cpp file is only doing Win32 stuff

address_space.cpp includes implementations for both Win32 and Linux/macOS

<!-- gh-comment-id:2393605394 --> @polybiusproxy commented on GitHub (Oct 4, 2024): > I was considering opening a PR for this, but I'm not sure how platforms other than Win32 are handled. It seems like the whole `address_space.cpp` file is only doing Win32 stuff `address_space.cpp` includes implementations for both Win32 and Linux/macOS
Author
Owner

@xezrunner commented on GitHub (Oct 4, 2024):

address_space.cpp includes implementations for both Win32 and Linux/macOS

I had a look again. I see it now, my bad.

What confused me was that backing_handle is (seemingly) only assigned to with CreateFileMapping2() in this file, which is a Win32 API call. In fact, that whole part is inside an #ifdef _WIN32.
Did not see backing_handle being assigned by anything else, which is what lead me to believe that virtual memory is handled somewhere else for other OSes.

I am only looking at it on the surface from GitHub though, and I'm not too familiar with this part of the codebase.

I could probably figure out how a low amount of VM fails on Linux/Mac, but I don't have any of those set up for hacking at the moment.

<!-- gh-comment-id:2393855823 --> @xezrunner commented on GitHub (Oct 4, 2024): > `address_space.cpp` includes implementations for both Win32 and Linux/macOS I had a look again. I see it now, my bad. What confused me was that `backing_handle` is (seemingly) only assigned to with `CreateFileMapping2()` in this file, which is a Win32 API call. In fact, that whole part is inside an `#ifdef _WIN32`. Did not see `backing_handle` being assigned by anything else, which is what lead me to believe that virtual memory is handled somewhere else for other OSes. I am only looking at it on the surface from GitHub though, and I'm not too familiar with this part of the codebase. I could probably figure out how a low amount of VM fails on Linux/Mac, but I don't have any of those set up for hacking at the moment.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/shadPS4#38
No description provided.