mirror of
https://github.com/1Remote/1Remote.git
synced 2026-04-25 13:36:03 +03:00
[GH-ISSUE #76] Syncing data #60
Labels
No labels
area-configuration
area-ct-app
area-ct-rdp
area-ct-remoteapp
area-ct-ssh
area-ct-vnc
area-launcher
area-list
area-tags
area-teamwork
bug
chore
dependencies
general-build/ci
general-performance
general-refactor
general-security
general-supportive
general-ux
meta-documentation
meta-enhancement
meta-enhancement
meta-feature
meta-help-wanted
meta-unknown-error
priority-hi
priority-low
pull-request
question
resolution-duplicate
resolution-invalid
resolution-wontfix
stale
task-put-off
task-still-considering
task-working-in-progress
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/1Remote#60
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @VShawn on GitHub (Feb 4, 2021).
Original GitHub issue: https://github.com/1Remote/1Remote/issues/76
I use 2 PCs at same time. Some time I have to copy PRM database from one to other manually to sync configuration.It is better to have some auto-sync feature to help me do this.
It may works as below:
01 Sync by cloud
In this approach
sqlite databasewould be upload/download by Drive automatically. What PRM needs to do is reload the database.advantage:
disadvantage:
02 Using MySql/MSSQL
In this approach PRM connect to the remote database so that we can get unified data in two PC.
advantage:
disadvantage:
discussion
the most importance things is design how to let PRM know that the data in the database is newer than the one in its cache and then update its cache by re-read from the database.
maybe we can add an update time fileld for every remote session config in the database to mark the data time. PRM reads the time in the database in the background periodically. Once the time in the database is later than the time in the cache, the PRM cache needs to be reloaded.
draft.zip
@VShawn commented on GitHub (Feb 4, 2021):
see here conflict comes, we edit issue at same time : )
@majkinetor commented on GitHub (Feb 4, 2021):
Ahahah, awesome :)
@majkinetor commented on GitHub (Feb 27, 2021):
Implemented or wontfix ?
@VShawn commented on GitHub (Feb 27, 2021):
？？？
Didn't remember I clicked the close button
@VShawn commented on GitHub (Jun 28, 2021):
I will not support sync database, I trying to design a subscription system to sync server data for business users on net5 version.
@majkinetor commented on GitHub (Dec 14, 2021):
I will reopen this as you told me you still want this.
I will look into this, initial plan is to see if this can be used: https://www.sqlite.org/sessionintro.html
Sync server might be great for enterprise, but probably bad for individual users that want PRM on several computers to be in sync. There is probably solution that targets both usecases.
@majkinetor commented on GitHub (Jan 17, 2022):
I don't like a lot when apps reinvent the thing when we can use better existing thing although here I also see a benefit of having it internally. So I thought to try available things.
I tried using Synthing first. Setup is fairly trivial, its done in couple of minutes on new machine and syncthing itself looks great. It syncs stuff, however PRemoteM must not be running cause it locks database file. Its not that easy to support such scenario on Windows (syncing locked files) so internal solution is in check if that stays this way. This makes syncthing and basically any other sync engine unusable (I am certainly not going to think about manually closing the app so it gets synced eventually)
One pretty easy solution comes to mind regarding this, and I bet a lot of people would appreciate if they could use whatever sync engine they are using currently:
This should be implementable in a day or two IMO, could be optional (config option [x] keep database unlocked) and would also be good for some helpers tools - those that would need to access and modify db directly in absence of PRM functionality.
With this, we can have sync in number of ways (I will make sure to document few of them if it gets developed). Internal functionality for this is certainly better but its far from easy to implement and then again you have that not-doing-one-thing-well problem. You can later always implement full blown internal solution but we will have working solution until then. Its also very cool not to lock the db as it unlocks extensions that could be done via separate apps/CLI scripts. Even when you need to manually copy db from one location to another without any automatic sync, it will help as you don't need to restart PRM.
What do you think ?
The only problem with this is PRemoteM.exe itself not being syncable (because it is locked and can't be unlocked) but that is IMO minor point and PRM can itself refuse to work with the newer db (if remote PRM gets updated before local PRM, the local could ask you to update the local PRM copy before continuing).
@VShawn commented on GitHub (Jan 18, 2022):
It was the feature of the very beginning version of PRM. The db was unlocked at that time, the synchronization of the database occasionally conflicts. Some users moved the database during use, and then the crashed.
In any case, this is indeed the easiest solution to achieve data synchronization, but we have to add monitoring of db file, to update the memory cache as soon as the db is updated. A warning pops up once the database is removed.
@majkinetor commented on GitHub (Jan 18, 2022):
This is not particularly needed - PRM can recreate it from its in memory state I guess. I wouldn't even popup user (you could make this configurable - [x] automatically recreate db from memory if it doesn't exist)
Conflict could happen indeed, but it could happen with any solution really.
Inaccessible db will also happen on both sync process side and PRM side if they run in parallel (depending who started first). PRM can simply ignore this and retry again later - if db stays locked longer then lets say 1m, it should complain to the user. If it happens on the synthing side, synthing will just ignore this event (it will be shown in logs) and try again later.
I would do this:
The idea is to not molest user but require attention when its absolutely necessary.
@majkinetor commented on GitHub (Jan 18, 2022):
About conflict:
in-memory lastChangeDate != disk lastChangeDateyou have conflict)There could be more advanced conflict resolution later, i.e. if changed stuff is from different connections, you could merge disk changes with memory changes easily. For that to work you just need to check if last changed date is the same or not on particular connection. If there are changes on both sides on different connections, you can load the disk db in memory, copy the changed connections to in memory database, then save the in memory database to disk. This is definitely good to have as 99% of db changes by multiple processes would most definitely be on different connections. If conflict is detected PRM could ask which connection to keep and even show diff.
This is all a bit more advanced, but IMO easy to grasp and not that hard to implement. However, this can be left for future - this type of sync is basically multi-user edit in parallel of the same database which is not something that is relevant for single person (its only relevant if 1 of the machines is offline, and for that reason we could have basic conflict resolution described in 1).
@VShawn commented on GitHub (Jan 19, 2022):
sounds reasonable.
We can first provide an option to decide DB is occupied or not, and then consider advanced things.
@majkinetor commented on GitHub (Apr 19, 2022):
So is this done? Can I test it again?
@VShawn commented on GitHub (Apr 21, 2022):
I have not add an
option to decide DB is occupied or not,But yes you can test it, the version after 6.1 will not occupied the database.
FYI, on my PC, my sync app always make my db to
read-onlybecause of the conflict.@VShawn commented on GitHub (Apr 25, 2022):
sync make my db read-only again! :(
@majkinetor commented on GitHub (Sep 13, 2022):
I found time to check how sync works now, and while I can open and change db while PRM is running, the changes are not visible until I restart PRM.
Did I miss something or you didn't implement this part ? Without it, the whole point of not locking the db is lost.
Like I said before, you could just show tray message or something (change the titlebar, border etc.) that database is locked but continue working with memory cache, trying to save data periodically. Sync should not lock db forever, but it will lock it temporary.
@VShawn commented on GitHub (Sep 14, 2022):
Yeap. I didn't implement this part.
A db file modify time checker for data update checking was build-in PRM, but its work wasn't good enought, so I delete.
Now I add a timer to check the update timestamp in db for check update and reload data in #282, WIP still.
@majkinetor commented on GitHub (Sep 14, 2022):
OK, let me know when its done. Thx.
@VShawn commented on GitHub (Sep 15, 2022):
sure
I got the update check logic into mess BTW
Now I need draw a flowchart before restart write new code.
@VShawn commented on GitHub (Oct 23, 2022):
since PRM do not occupied the database after 6.X and #282 make MySQL available, I'll close this ticket.
@majkinetor commented on GitHub (Oct 24, 2022):
That is not exactly the same thing:
@VShawn commented on GitHub (Oct 25, 2022):
Yes, it can now, 1RM will check database update and reload data every 10s by default since #282.
@VShawn commented on GitHub (Oct 25, 2022):
And PRM will stay in current version, no farther feature will be added.
@majkinetor commented on GitHub (Oct 25, 2022):
O, didn't know that. Let me test that and get back to you.
What do you mean? Forever or regarding this feature or... :)
@VShawn commented on GitHub (Oct 25, 2022):
The next update of the app will be
1Remote:)@majkinetor commented on GitHub (Oct 31, 2022):
I am on 0.7.1.7 beta 3 and it doesn't reaload after 10s. Am I missing anything ?
@VShawn commented on GitHub (Nov 1, 2022):
OH nonono, 0.7.X is PRM, PRM does not support new feature.
1RM 1.0.0 alpha should works.
@majkinetor commented on GitHub (Nov 1, 2022):
It doesn't work either. I opened dbeaver, changed the connection, closed dbeaver to release a lock, and waited, nothing happened. Restarted the app and the changes were visible.
BTW, 1RM didn't load my PRM database file.
@VShawn commented on GitHub (Nov 1, 2022):
1RM will reload the database only if database was modified by another 1RM.
there is a key to identify the latest modify time of connection data.
1RM will check this key and compare with its
last read time. Iflast read time<latest data time, 1RM will read the changes.Now 1RM will port from PRM automatically when
1Remote.json1Remote.dbnot existed and there is aPRemoteM.dbin the folder.BTW, you can get the latest build from github action artifacts now
https://github.com/1Remote/PRemoteM/actions/runs/3345610634
@majkinetor commented on GitHub (Nov 1, 2022):
Awesome.
Ah, OK. Not sure why you had to put that requirement. Adding automated column or trigger to insert last change date no matter what process did it sounds like better option? Not a problem tho, I can set that field too. I see only
Config.UpdateTimestamp. Where is thislatest data time?BTW, I don't think we are at 1.0 version yet. I would make 1RM 0.9 variant as I think we need to redesign protocols a bit probably in incompatible way and do relational db tests.
@VShawn commented on GitHub (Nov 1, 2022):
UpdateTimestamp is the
latest data timeI was once try to check modify time of the .db file but it got into a mess, latter I design the UpdateTimestamp.
of course, we can make it 0.9
Since my progress is slow for months, it would be a long time before I made a stable release of 1RM. And I would make 1RM publish with nightly build before jobs done.
@majkinetor commented on GitHub (Nov 1, 2022):
This column is under Config. Why didn't you introduce on each connection? This way you always need to relad everything instead of individual changes.
@VShawn commented on GitHub (Nov 1, 2022):
because PRM doesn't have a db interface to reload a individual connection. A global flag is the easiest approach to goal the sync feature based on PRM.
introduce on each connectionis my step2 plan, i will have to redesign db interface before get start withintroduce latest data time on each connection@majkinetor commented on GitHub (Nov 13, 2022):
OK, now I am on latest. Config column now has only RSA stuff. Where is the
UpdateTimestamp?I still can't change stuff externally without a restart without it.
@VShawn commented on GitHub (Nov 14, 2022):
you mean 0.7.1.7? I have to say this feature and other new feature will be add to next version, 0.7.X is LTS version.