mirror of
https://github.com/007revad/Synology_enable_Deduplication.git
synced 2026-04-25 12:46:03 +03:00
[GH-ISSUE #25] [Feedback] DS1618+ #276
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/Synology_enable_Deduplication#276
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Xeroxxx on GitHub (Aug 14, 2023).
Original GitHub issue: https://github.com/007revad/Synology_enable_Deduplication/issues/25
Synology DS1618+ with 7.2-64570 Update 3
Regular SSD EMTEC X150
Does not work. Deduplication menu does not show up. Scheduling is visible under global settings.
General Feedback:
--restore does not restore synoinfo.conf
@007revad commented on GitHub (Aug 16, 2023):
Do you have a Btrfs storage pool and a volume on the SSD?
Do you have usage detail analysis enabled for that Btrfs volume?
Does the SSD drive or volume show any warnings in storage manager?
Thanks for the feedback on restoring synoinfo.conf. I'll have a look at that.
Was it
/etc/synoinfo.confor/etc.defaults/synoinfo.confthat wasn't restored?@Xeroxxx commented on GitHub (Aug 16, 2023):
Yes BTRFS, Yes on SDD
Yes
No
Only /etc/synoinfo.conf, yes NAS has been restarted twice after restore.
No there is no cronjob.
@007revad commented on GitHub (Dec 5, 2023):
Do you have 16GB or more memory?
@007revad commented on GitHub (Dec 6, 2023):
I just tested with DSM 7.2.1 update 3 on my DS1821+ and it worked.
I can see your PCIe slot is empty.
Are these 2.5 inch SATA SSDs or NVMe drives?
If you run syno_enable_dedupe.sh with the --check option what does it return?
@007revad commented on GitHub (Dec 6, 2023):
Do both of the following commands return yes
synogetkeyvalue /etc.defaults/synoinfo.conf support_btrfs_dedupesynogetkeyvalue /etc/synoinfo.conf support_btrfs_dedupeI just tested with DSM 7.2.1 update 3 on my DS720+ and it also worked (even though it only has 2GB of memory!?!?)
I had previously run syno_enable_m2_volume on the DS720+ so /usr/lib/libhwcontrol.so.1 was already edited. What I did notice was:
After setting support_btrfs_dedupe to yes in /etc.defaults/synoinfo.conf and rebooting the deduplication option apperared in storage manager.
sudo -i synosetkeyvalue /etc.defaults/synoinfo.conf support_btrfs_dedupe yes@007revad commented on GitHub (Dec 6, 2023):
Both my DS1821+ and DS720+ are running DSM 7.2.1-69057 Update 3 and the deduplication option is visible on them. https://imgur.com/6JbKkab
I just noticed I previously wrote 7.2.1 update 1 (which was wrong).
@007revad commented on GitHub (Dec 6, 2023):
Can you check if there's a /var/log/synoinfo.conf.bad file and when it was created:
ls -l /var/log/synoinfo.conf.badCan I get a copy of these 2 log files:
@007revad commented on GitHub (Dec 6, 2023):
If you zip the logs you can attach the zip file to a comment. You can copy and paste, or drag and drop the zip file onto a comment, or use the "Paste, drop or click to add files" button/link below the comment when you're typing it.
@007revad commented on GitHub (Dec 7, 2023):
Hmm, no errors in those. Can you reply with the following logs:
/var/log/packages/StorageManager.log
/var/log/disk.log
/var/log/messages
@007revad commented on GitHub (Dec 7, 2023):
I just searched for files containing dedupe in DSM 7.2.1 for the DS1821+ and the DS1823xs+ and only 3 of those files are different between the 2 models.
Both models use the same Storage Manager package.
I haven't looked in btrfs.ko yet but I don't expect there to differences relating to deduplication.
The interesting differences in synoinfo.conf are:
I can't find any info on what support_fc does. support_uasp only relates to USB drives.
is_business_model, support_diffraid, support_synodrive_ability and support_synodrive_rule look interesting and need some investigation.
@Xeroxxx commented on GitHub (Dec 7, 2023):
Would you mind comparing DS1821+ to DS1618+. Two non business models 1821 works, 1618 doesnt.
1618+
support_synodrive_ability="no"
support_synodrive_rule="1"
support_uasp="no"
support_syno_hybrid_raid="yes"
support_auto_repair="no"
@007revad commented on GitHub (Dec 8, 2023):
@Xeroxxx
Unfortunately there's a lot missing in DSM 7.2.1 for the DS1618+ that prevents deduplication from working.
And there's 23 files that are needed for deduplication that don't exist on a DS1618+
One of the Xpenology devs informed me maybe 2 months ago that this script does not work with some CPU architectures including the Denverton CPU used in the DS1618+.
I've updated the readme to show which CPU types this script work for and which ones it doesn't work for. https://github.com/007revad/Synology_enable_Deduplication
@karthik-panchap commented on GitHub (Dec 14, 2023):
Interesting update. Adding an NVME pool and volume provides the option to enable deduplication.
To rule out any issues with content on the SSD pool/volume - erased and rebuilt them. While an option comes up to include the SSD pool/volume in global deduplication - the SSD volume still does not provide the option.
@007revad commented on GitHub (Dec 14, 2023):
@karthik-panchap
Interesting. So deduplication still works on a DS1823xs+, but just not for your SATA SSDs for some reason.
I wondered if it doesn't work on RAID 5 volumes as the officially supported Synology models have RAID F1 as a RAID 5 alternative for SSD storage pools. But I can't find anything that confirms that.
I do wonder if DSM knows your SATA SSDs are not Synology drives because the largest Synology branded SATA SSD is 7TB. I'll search in DSM for files containing SAT5210 and see if I can find a list of Synology SSDs I can edit.
Did you delete all your previous comments?
@007revad commented on GitHub (Dec 14, 2023):
I wasn't seeing the deduplication option for my SATA SSD but I incorrectly assumed it was because it needed to have at least 2 drives in RAID 1 or SHR. So I backed up then removed a SATA SSD from my PC and put it in my DS1821+ and created a 2 drive SATA SSD SHR volume with Btrfs and noticed a few things:
I though maybe it was because one of my SATA SSDs shows "Unknown" as the brand instead of Crucial. So I deleted the storage pool and created a new one on just 1 Intel SATA SSD but got the same result as point 3.
It looks like in DSM 7.2.1 (with Storage Manager now a package) you must enable Automatic Data Deduplication when creating the storage pool !?!?
I don't know if this is a bug, or is normal behaviour, or is only happening because the drives aren't Synology drives.
At least now that I'm seeing the same behaviour as you I can test things here to try to find a solution.
@karthik-panchap commented on GitHub (Dec 14, 2023):
This is exactly what is happening on the DS1823. However, dedupe options and metrics are only shown on the NVME volume - which suggests that even though dedupe seems enabled for the SSD volume - it's not actually working on the SSD volume.
BTW: Yes, I removed the older comments as they were misleading on the issue.
@007revad commented on GitHub (Dec 17, 2023):
This is proving to be harder than I hoped. The file I've been looking at is over 43,000 lines of javascript. Searching and reading a 43,000 line file is bad enough let alone javascript (which I have limited knowledge of).
While searching through that file I noticed some hard coded Synology drive model numbers and thought I'd come back to. Finding it again was time consuming because I was searching for SAT and HAT. When I did eventually find it it was for Synology NVMe drives that start with SVN (not SAT or HAT). I tried adding my SATA SSD's model number but that made no difference.
While viewing the volume ... button of my SATA SSD volume in the storage manager page (using Chrome's Inspect option) I did manage to click on something that made a window popup asking if I was sure I wanted to run deduplication. So somehow I had clicked on the Manual Data Deduplication Run Now button (even though it wasn't visible).
@007revad commented on GitHub (Dec 17, 2023):
I think I'm close.
The file I'm looking at contains 371 instances of dedup so I decided to test if I was on the right track by trying to enable creating a storage pool on NVMe drives in a PCIe adaptor card, which only has 2 instance in the file... and it worked!
@007revad commented on GitHub (Dec 19, 2023):
@karthik-panchap
I've got deduplication working on a SATA SSD by editing the file by hand.
This change also enabled deduplication on HDDs. I'm not sure if that's a good thing or a bad thing. There was no option to enable Automatic Data Deduplication when creating the volume but the ... menu allows me to enable or disable Automatic Data Deduplication or run Deduplication manually.
I just ran deduplication on a volume on a HDD and it worked.
I also noticed that all my drives (HDD, SATA SSD and NVME) have "Firmware Update" in their Action menu. This is obviously a side effect of making DSM think all the drives are Synology drives.
@007revad commented on GitHub (Dec 21, 2023):
@karthik-panchap
I've released a new version that solves your issue of the missing Configure Data Duplication menu option.
https://github.com/007revad/Synology_enable_Deduplication/releases/tag/v1.1.13
Even though it is working perfectly I did notice something strange. After running the script with the --restore option (and checking that the 4 files were actually restored to default) the Configure Data Duplication menu option still appeared for SATA SSDs and HDDs.
@karthik-panchap commented on GitHub (Dec 23, 2023):
This is a great thing assuming it works with in conjunction with SSD caching. REFS (Windows Server) has this feature and is well loved.
Good news: Installing the latest version enabled the option and performing a manual dedupe run seemed to work.
[Potentially unrelated] Bad news: After the test (manual dedupe run on a close to empty volume ~100mb), calling the syno_enable_dedupe.sh script with the --restore option prompted for a reboot that did not complete. After waiting for ~2 hours and performing a forced reboot via ssh - the DSM web GUI broke.
At this point, there seem few options but to reset the system. Fortunately, I have backed everything up. Again, the issue might be due to some unique corner case on my system.
Thanks for the progress and Happy Holidays!
@007revad commented on GitHub (Dec 23, 2023):
I've had that happen a few times. DSM really doesn't like some files being replaced while it's in use. Editing the files is okay.
I had a similar issue when I ran this script with the --restore option where it never rebooted and I couldn't access the UI (though SSH still worked). I only waited about 20 minutes before I pressed and held the power button on the NAS, expecting the NAS to shut down. The NAS beeped when I released the button... and then it shut down and booted up again. So it appeared it was stuck trying to shutdown until I forced it to shutdown.
After it came back online everything was normal... except deduplication was still enabled for HDDs and SATA SSDs. Even though running the script with the --check option shows it should not be enabled for HDDs and SATA SSDs.
I might change the restore process to edit the file instead of replacing the file with of copy form the backup.
@007revad commented on GitHub (Dec 24, 2023):
@karthik-panchap
I've nearly finishing changing the restore option. Instead of copying backups over the top of files that DSM is using it will edit the values in those files to restore the values from the backup files.
I've also discovered that one of the 3 files doesn't need to be edited in DSM 7.2.1. This file is the one that requires rebooting and causes issues in DSM if the file is replaced instead of being edited.
@007revad commented on GitHub (Dec 25, 2023):
@karthik-panchap
I've released a new version with a much better restore process, and some other improvements.
https://github.com/007revad/Synology_enable_Deduplication/releases/tag/v1.2.14
@planetrocky commented on GitHub (Oct 29, 2025):
I appreciate this is closed and old; besides many missing files on Denverton CPU based Synologys, is there any technical reason why dedupe couldn't be run?
I'm on an DS1819+ with 200TiB, and unlikely to be able to upgrade anytime soon.
@007revad commented on GitHub (Oct 29, 2025):
There are way too many files missing in DSM for older models to make it work reliably, and any DSM update could break it.