[PR #3771] [Draft] Full filesystem emulation with host integration #3710

New issue

Open

opened 2026-02-27 22:04:42 +03:00 by kerem · 0 comments

kerem commented

2026-02-27 22:04:42 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/shadps4-emu/shadPS4/pull/3771
Author: @marecl
Created: 11/3/2025
Status: 🔄 Open

Base: main ← Head: quasifs

📝 Commits (10+)

8ea7ce6 qfs integration
b0f31ad initial integration
c16ba68 Implemented sceKernelTruncate (was missing)
b105ade added /dev/null properly
538ee93 Fixed incorrect size being read preventing actually reading anything
c328815 Separated virtual from "regular" files
cd49b6b Prepared Directory for normal and PFS
ab5a9db As above, forgor
6d02f5b Quieted some messages (i'm lazy)
60a32ab moar

📊 Changes

100 files changed (+6731 additions, -2514 deletions)

View changed files

📝 CMakeLists.txt (+67 -25)
📝 CMakeSettings.json (+2 -2)
📝 documents/building-macos.md (+36 -0)
📝 src/common/io_file.cpp (+0 -22)
📝 src/common/io_file.h (+0 -1)
➖ src/core/file_sys/devices/base_device.cpp (+0 -12)
➖ src/core/file_sys/devices/base_device.h (+0 -76)
📝 src/core/file_sys/devices/console_device.cpp (+3 -65)
📝 src/core/file_sys/devices/console_device.h (+28 -21)
📝 src/core/file_sys/devices/deci_tty6_device.cpp (+3 -65)
📝 src/core/file_sys/devices/deci_tty6_device.h (+28 -21)
📝 src/core/file_sys/devices/logger.cpp (+6 -19)
📝 src/core/file_sys/devices/logger.h (+25 -9)
📝 src/core/file_sys/devices/nop_device.h (+21 -54)
➕ src/core/file_sys/devices/null_device.cpp (+23 -0)
➕ src/core/file_sys/devices/null_device.h (+40 -0)
📝 src/core/file_sys/devices/random_device.cpp (+14 -65)
📝 src/core/file_sys/devices/random_device.h (+33 -21)
📝 src/core/file_sys/devices/rng_device.cpp (+4 -59)
📝 src/core/file_sys/devices/rng_device.h (+25 -18)

...and 80 more files

📄 Description

This PR implements full on filesystem implementation, superseeds #3368.
Backend resembles how kernel treats files, with generic POSIX inode prototypes.

Main goal: Accurate FS emulation
Side goal: fix games suffering from inconsistencies between POSIX/native FS calls and C++ streams (Catherine, Burnout IIRC).

What works

Compiles and mostly works on Linux
stat: nearly everything: fileno, nlink, (mostly) timestamps etc.
Virtual partitions like /dev
Virtual files (data is written and read, not committed to host)
Real-time mounting/unmounting partitions (DLC, savedata)
Links, symlinks
Host-bound partitions like /app0, /data
Host bound files/directories
File operations (open, link, etc.)
Partition permissions (RO/RW)
File permissions
Full path resolution
- Guards against jumping out of host's target directory
Compliant with all tested edge cases for file_system
Virtual devices (stdout, urandom etc.)
PFS and regular dirents
PFS filesystem
File descriptors of files with no links are valid

Known issues

Sockets / epoll not working
Desync - host may or may not r/w as many bytes as requested, which can throw off virtual descriptors
Occasional race conditions (found):
- Earliest available file descriptor can get overwritten
Long file indexing times on startup:
- ~10s for 25k files
- Fixable

TODO

Symlink on Windows hosts (partially "implemented")
macOS support (at all)
Case-insensitive path resolution
Measure execution speed
Port epoll
Port sockets (new inode derivative)
Thread safety overhaul (CUSA33241 crashing on launch)
Partition overlays for -UPDATE and -patch

Optional

(?) mount -o bind-equivalent
(?) Path lookup table/cache
(?) Pre-caching files in RAM
- I think some files are small enough to justify keeping them there
- Would increase RAM usage, but maybe speed up loading times
- IDK, if it worked that'd be cool since some people can benefit from an abundance of RAM
- My vision of effective implementation involves telemetry/trace calls, which is to be yet discussed
(?) stream API to preserve shad's internal calls
- There's a lot of them
- Raw u8 buffer read/writes may or may not be the most intuitive solution

What's not supposed to be a part of this PR

Disabled some stub warnings
Disabled SDL audio queue warnings
Changes to vulkan-headers which are not changes. No idea how to solve this o.o

Memory managements

Everything is based on shared_ptr. My reasoning is that OG pointers can (and will) run out of scope, making MM a living hell. This way inodes live on in their own, getting freed on their own. Maybe some of those can be substituted with weak_ptr but I want to make it work first.

Disclaimer

All lists are not exhaustive. This is subject to a lot of changes.
It's also not like there are a lot of things not working and few working, it's quite the contrary. I just don't really remember anymore.

I'm well aware that some features may be an overkill, but so far I need time to make it work, wait for feedback and then evaluate what requires optimization.

Based on my own lib built specifically as a PoC for this usecase
This lib includes tests and examples on how to use and navigate QFS.

How QFS works

Everything is based on inode having basic file prototypes
Basic types are morphable to implement, for example, multiple types of files being files, but handled in a different way (like regular and PFS directories, host-bound and virtual files)
File attributes can be read with POSIX-standard (or helper) functions (no external separation of files, devices etc.)
QFS owns a root partition "/"
Any partition may have (theoretically) unlimited amount of files, directories and mountpoints
QFS is completely isolated from host
Interacts with HostIO to access the host

There's a lot of inheritance, but I hope it's organized well enough.

HostIO

This is a proxy driver interacting with QFS and host's FS. All targets are independent.
Since this driver is completely separate, interacting with QFS requires setting up some context before file operations.
Hosts are not affected.

This lib is written with Linux as a target, since PS4 is running basically that. macOS is similar, but Windows needs to accomodate.

Disclaimer

So far it works, but my free time has been nerfed by a lot recently.
I hope I'll be able to push it out as far as I can.
If you have some spare time on your hands, feel free to review and question my design decisions.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/shadps4-emu/shadPS4/pull/3771 **Author:** [@marecl](https://github.com/marecl) **Created:** 11/3/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `quasifs` --- ### 📝 Commits (10+) - [`8ea7ce6`](https://github.com/shadps4-emu/shadPS4/commit/8ea7ce6147a8c4103736411bda13722b1e8d58c5) qfs integration - [`b0f31ad`](https://github.com/shadps4-emu/shadPS4/commit/b0f31ad23ae32b97d735d6f8962fa6f8bf00b769) initial integration - [`c16ba68`](https://github.com/shadps4-emu/shadPS4/commit/c16ba684222dbcd18fbcf6cc525991131af08316) Implemented sceKernelTruncate (was missing) - [`b105ade`](https://github.com/shadps4-emu/shadPS4/commit/b105adeb9e10fa4bbf271f36a1ce5427f22fcec8) added /dev/null properly - [`538ee93`](https://github.com/shadps4-emu/shadPS4/commit/538ee9361d109bd565213c97e21877e03fa8c527) Fixed incorrect size being read preventing actually reading anything - [`c328815`](https://github.com/shadps4-emu/shadPS4/commit/c3288155915bc52acae6209d4f078fac63108019) Separated virtual from "regular" files - [`cd49b6b`](https://github.com/shadps4-emu/shadPS4/commit/cd49b6b1e1f55a9ecb1c2665c920ba4b9965960b) Prepared Directory for normal and PFS - [`ab5a9db`](https://github.com/shadps4-emu/shadPS4/commit/ab5a9db46f78c39270efd3f7a1295e24e3b182df) As above, forgor - [`6d02f5b`](https://github.com/shadps4-emu/shadPS4/commit/6d02f5bb3b2604c51427e6355d5fa6bd6a7863a1) Quieted some messages (i'm lazy) - [`60a32ab`](https://github.com/shadps4-emu/shadPS4/commit/60a32ab7238336becedfa68929bc0937745a38b0) moar ### 📊 Changes **100 files changed** (+6731 additions, -2514 deletions) <details> <summary>View changed files</summary> 📝 `CMakeLists.txt` (+67 -25) 📝 `CMakeSettings.json` (+2 -2) 📝 `documents/building-macos.md` (+36 -0) 📝 `src/common/io_file.cpp` (+0 -22) 📝 `src/common/io_file.h` (+0 -1) ➖ `src/core/file_sys/devices/base_device.cpp` (+0 -12) ➖ `src/core/file_sys/devices/base_device.h` (+0 -76) 📝 `src/core/file_sys/devices/console_device.cpp` (+3 -65) 📝 `src/core/file_sys/devices/console_device.h` (+28 -21) 📝 `src/core/file_sys/devices/deci_tty6_device.cpp` (+3 -65) 📝 `src/core/file_sys/devices/deci_tty6_device.h` (+28 -21) 📝 `src/core/file_sys/devices/logger.cpp` (+6 -19) 📝 `src/core/file_sys/devices/logger.h` (+25 -9) 📝 `src/core/file_sys/devices/nop_device.h` (+21 -54) ➕ `src/core/file_sys/devices/null_device.cpp` (+23 -0) ➕ `src/core/file_sys/devices/null_device.h` (+40 -0) 📝 `src/core/file_sys/devices/random_device.cpp` (+14 -65) 📝 `src/core/file_sys/devices/random_device.h` (+33 -21) 📝 `src/core/file_sys/devices/rng_device.cpp` (+4 -59) 📝 `src/core/file_sys/devices/rng_device.h` (+25 -18) _...and 80 more files_ </details> ### 📄 Description This PR implements full on filesystem implementation, superseeds #3368. Backend resembles how kernel treats files, with generic POSIX `inode` prototypes. Main goal: Accurate FS emulation Side goal: fix games suffering from inconsistencies between POSIX/native FS calls and C++ streams (Catherine, Burnout IIRC). ### What works * Compiles and mostly works on Linux * `stat`: nearly everything: `fileno`, `nlink`, (mostly) timestamps etc. * Virtual partitions like `/dev` * Virtual files (data is written and read, not committed to host) * Real-time mounting/unmounting partitions (DLC, savedata) * Links, symlinks * Host-bound partitions like `/app0`, `/data` * Host bound files/directories * File operations (`open`, `link`, etc.) * Partition permissions (RO/RW) * File permissions * Full path resolution * Guards against jumping out of host's target directory * Compliant with all tested edge cases for `file_system` * Virtual devices (`stdout`, `urandom` etc.) * PFS and regular dirents * PFS filesystem * File descriptors of files with no links are valid ### Known issues * Sockets / epoll not working * Desync - host may or may not r/w as many bytes as requested, which can throw off virtual descriptors * Occasional race conditions (found): * Earliest available file descriptor can get overwritten * Long file indexing times on startup: * ~10s for 25k files * Fixable ### TODO * Symlink on Windows hosts (partially "implemented") * macOS support (at all) * Case-insensitive path resolution * Measure execution speed * Port epoll * Port sockets (new `inode` derivative) * Thread safety overhaul (CUSA33241 crashing on launch) * Partition overlays for `-UPDATE` and `-patch` ### Optional * (?) `mount -o bind`-equivalent * (?) Path lookup table/cache * (?) Pre-caching files in RAM * I think some files are small enough to justify keeping them there * Would increase RAM usage, but **maybe** speed up loading times * IDK, if it worked that'd be cool since some people can benefit from an abundance of RAM * My vision of effective implementation involves telemetry/trace calls, which is to be yet discussed * (?) stream API to preserve shad's internal calls * There's a lot of them * Raw `u8` buffer read/writes may or may not be the most intuitive solution ### What's not supposed to be a part of this PR * Disabled some stub warnings * Disabled SDL audio queue warnings * Changes to vulkan-headers which are not changes. No idea how to solve this o.o ### Memory managements Everything is based on `shared_ptr`. My reasoning is that OG pointers can (and will) run out of scope, making MM a living hell. This way `inodes` live on in their own, getting freed on their own. Maybe some of those can be substituted with `weak_ptr` but I want to make it work first. ### Disclaimer All lists are not exhaustive. This is subject to a lot of changes. It's also not like there are a lot of things not working and few working, it's quite the contrary. I just don't really remember anymore. I'm well aware that some features may be an overkill, but so far I need time to make it work, wait for feedback and then evaluate what requires optimization. Based on my own lib built specifically as a [PoC for this usecase](https://github.com/marecl/QuasiFS) This lib includes tests and examples on how to use and navigate QFS. ### How QFS works * Everything is based on `inode` having basic file prototypes * Basic types are morphable to implement, for example, multiple types of files being files, but handled in a different way (like regular and PFS directories, host-bound and virtual files) * File attributes can be read with POSIX-standard (or helper) functions (no external separation of files, devices etc.) * QFS owns a root partition `"/"` * Any partition may have (theoretically) unlimited amount of files, directories and mountpoints * QFS is completely isolated from host * Interacts with `HostIO` to access the host There's **a lot** of inheritance, but I hope it's organized well enough. ### HostIO This is a proxy driver interacting with QFS and host's FS. All targets are independent. Since this driver is completely separate, interacting with QFS requires setting up some context before file operations. Hosts are not affected. This lib is written with Linux as a target, since PS4 is running basically that. macOS is similar, but Windows needs to accomodate. ### Disclaimer So far it works, but my free time has been nerfed by a lot recently. I hope I'll be able to push it out as far as I can. If you have some spare time on your hands, feel free to review and question my design decisions. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>