Add the ability to use other online file storage/file-sharing solutions (besides Dropbox), such as:
I think any and all of these will work today without RM doing anything to make it happen.
However, doing so is not recommended because of a potential for corruption of your RM database. These solutions are only safe if the software has an option to pause sync while you are using RM, and if you as the user exercise this option without fail.
The official recommendation is to use an online file storage/file-sharing solution only for your RM backup files and to keep your RM database outside such a sharing solution. It seems to me that this recommendation is fine for single computer situations. But for using RM on two more more computers, the recommended solution requires that you always backup your RM database when you shut down RM and that you restore your RM database when you start up RM. Itās easy to remember always to backup, but itās not so easy to remember always to restore your RM database rather than opening it. From much personal experience using RM on more than one computer, I find it much easier to remember to pause and resume the sharing solution than it is to remember to backup and restore (most especially the restore). So I have always done the pause and resume thing.
If you are talking about the RM9 feature invoked via File => Export => Dropbox, then that feature is primarily in support of the RM mobile app on an iPhone or iPad. The mobile app only supports an RM7 database, and the File => Export => Dropbox feature creates an RM7 database from your RM9 database that will work with the mobile app. The RM7 database that is created in this manner is not complete enough to allow you to use the feature to revert from RM9 back to RM7. It only does just barely enough to make the mobile app still work on an iPhone or iPad.
1 Like
I have read about this a lot on the RM forums but I wonder if the risk of database corruption is as real as you say. I have used RM in this way for some time as I was working on a Mac mostly (still do some times). Syncing to iCloud is baked in to a Mac and has been for years. It is possible to put your RM database in a folder at a higher level that does not sync, but this is not where RM puts it when it installs. And the only way to āpauseā sync to iCloud is to sign out and that is a real mess!
It is worth noting that Windows behaves this way too now if you are signed in to your PC via a MS account rather than a local account, as most people will do now I suspect since it is the default way to set up a new Windows PC. You can of course still pause OneDrive, but as you say remembering to do so and RM will have also put the database in a location that is synced in Windows by default.
So far I have had no issues using RM on computers with sync. Watching sync behaviour, syncing only takes place when RM makes a save of your work periodically - it is not constant. I do have my backups but I have, fortunately never needed them.
What behavior would be symptomatic of a corrupted RM9 file after using a syncing product with an active RM file such as Dropbox and OneDrive? Iāve used RM for years with my active local RM file that syncs normally with Dropbox and OneDrive (Not simultaneously since I work with several RM files). Iāve not noticed any data file corruption or detected lost or corrupted data. But I do regularly use Tools to check integrity and rebuild indexes almost daily. Much thanks for your help.
You raise many good points in your message.
Many users have the same question. And it is a good question. Thereās another example of the same question from Ole Seminole just below your message. I donāt know how to quantify the level of risk. I just know that users have reported corrupted RM databases when their RM database was in a folder that was managed by a syncing product. I have never had such corruption, but before I did the pause sync thing I definitely had I/O errors in my RM database caused by Dropbox sync operations. I would have to shut down RM and restart, but I never found any corruption.
Here is a link to an article about using iCloud with SQlite databases. It recommends against doing so. The article is not specific to RM, and remember there are many SQLite applications other than RM.
I agree that itās worth noting, and the problem on Windows and OneDrive can be even worse. Thatās because OneDrive defaults to keeping your files only in the cloud without keeping a permanent copy also on your local hard disk. I go to great pains to be sure my Windows account is a local account instead of a MS account. I also go to great pains to be sure all the files in my OneDrive folder are stored locally at all times in addition to being stored in the cloud.
I would have to disagree with this comment. My experience with Dropbox in particular is that unless itās paused, it syncs after every single change to your RM database, so matter how trivial or trite that change. And sometimes just switching from one person to another will make changes in your RM database even though you didnāt go into the Edit Person screen or anything like that. The Dropbox sync is not a periodic sync, like every ten minutes or anything like that. The Dropbox sync for an SQLite database is continuous.
Because Dropbox is so aggressive about continuous syncing, in addition to possible corruption using it with RM can also result in enormously heavy Internet traffic that surely is not desirable.
The only two syncing tools I have used are Dropbox and OneDrive. If I had to choose one or the other to store my RM database without pausing, I would say that OneDrive is by far the safer or the two. It is nowhere near as aggressive as is Dropbox about continuous syncing and sometimes it even seems like itās aware that RM is doing something to the database and that it shouldnāt sync. But sometimes OneDrive syncs anyway, and as a result I always pause it when using RM.
The bottom line is that I donāt know how to quantify the risk level. I just know the following. I know that there are persistent reports of RM users with corrupted database caused by using RM without pausing the syncing tool. I know that the RM company recommends against using RM with a syncing tool. I know that there are technical articles that are not specific to RM concerning both Dropbox and OneDrive that recommend against using SQLite databases with a syncing tool. I know that there are technical articles that are not specific to RM concerning both iCloud and Time Machine that recommend against using SQLite databases with a syncing tool.
2 Likes
How to stop using OneDrive for my RM9?
Exclude .rmtree files (instead of .png shown in the tutorial)
Exclude Specific File Extensions from Backing up to OneDrive in Windows 11 Tutorial
In my experience I have OndDrive setup to minimize issue by maintaining a local copy OneDrive with then pause syncing of files āin useā ā I do not have any issues with my databases (or media) ā However this setup it discouraged by RM and not advised.
Basically, you would need to make sure the folder is not being backed up by OneDrive (or other cloud service. Alternately you could pause the cloud back up but that still has potential risks
In 2020 I was surprised to see that OneDrive paused sync automatically on an RM7 file while it was āopenā. In 2023, I was dismayed to find that it did not for a RM9 file. I wonder if there was obfuscation from having possibly changed my Windows login from local to online in between. I had thought that the later RM version had lost some flag feature thst OneDrive depended on to detect that the sqlite database file was āopenā in an app - thatās different from the write-lock applied with every change to the database.
https://rootsmagic-users.groups.io/g/main/message/3567
1 Like
Interesting.
My OneDrive backups 99% of 2nd drive (my laptop is older but has two HD bays). I have 2 SSDās one for OS/programs and the 2nd is for data files which is basically all for OneDrive (in a folder OneDrive (drive also identifies as O ). I know if my case OneDrive āpausesā the SQLITE RM Database file when open in RM (or I believe anything else). I am on W10 (PRO) ver 22H2 (19045.4291)
I have tried the following experiment using RM9 and OneDrive.
- Leave OneDrive in a syncing state - which is to say that it is not paused.
- Start up RM9 and open a test database which is a full copy of my production database.
- Open up OneDrive to determine its status. It reports a syncing conflict where it is not able to sync my test RM9 database because it is in use. This is great because it suggests that it is safe to use OneDrive and RM9 together without pausing
- Go about my business on my PC for a few minutes, doing non-RM9 stuff like reading emails, etc.
- It is now a few minutes later. Open up OneDrive again to determine its status. It no longer reports a syncing conflict with my RM8 database. This is terrible because it suggests that it is unsafe to use OneDrive and RM9 together without pausing.
This, in addition to other similar reports about using RM9 with an unpaused OneDrive being unsafe such as the one from Tom, is why I now pause OneDrive when Iām using RM9.
SQLite is embedded into many software apps and is by far the most used relational database in the world. It has many billions of installations, probably close to a trillion. If you have a smart refrigerator, the computer in your refrigerator is probably running an app that uses SQLite. If you have a smart car (and all modern cars are smart), then some of the computers in your car are probably running an app that uses SQLite. If you have a smart watch, then the computer in your watch is probably running an app that uses SQLite. If you have a smart phone, then the computer in your phone is probably running an app that uses SQLite. Etc.
So SQLite is a very mainstream product. However, itās designed for use by a single user at a time. It certainly is not designed so that two different users on two different computer can be using the same RM9 database at the same time via some sort of file sharing tool. SQLite does have some ability which I donāt fully understand to be running in multiple threads on the same computer and for the multiple threads to coordinate with each other via software locking mechanisms. But itās unclear to me exactly how such multiple threading is supposed to work between something like RM and Dropbox or between RM and OneDrive. And in fact, it seems not to work very well.
The only other app I use on my computer besides RM that I know for sure that uses SQLite is Microsoft Visual Studio which I use to develop C++ programs. Visual Studio pretty much has a new version every year and the year is a part of the name of the product. Visual Studio started using SQLite with Visual Studio 2015. Prior to Visual Studio 2015, my Visual Studio files were all in my Dropbox folder and I didnāt pause Dropbox when I used Visual Studio. Because there was no SQLite in Visual Studio prior to Visual Studio 2015, I had no problems with Dropbox and Visual Studio being used together. But starting with Visual Studio 2015, Visual Studio would crash because of Dropbox syncing and some of my Visual Studio files would become corrupted.
My short term solution was to pause Dropbox when I was using Visual Studio just like I did when I was using RM. But I soon discovered that OneDrive seemed to work just fine with Visual Studio without pausing and I moved all my Visual Studio files from Dropbox to OneDrive. My theory was that Microsoft had probably discovered the problem with syncing SQLite files and had made OneDrive smart enough not to sync an SQlite database that was in use. Therefore I tried moving my RM databases from Dropbox to OneDrive and using RM without pausing OneDrive. It worked great for several years, and then it didnāt. So I now choose to pause OneDrive when I am using RM. I donāt know what changed - whether it was something in OneDrive or something in RM or both. And by the way, Iām currently on Visual Studio 2022 and it still works just fine without pausing OneDrive. OneDrive shows that Visual Studioās SQLite databases are in use and doesnāt sync them until I shut down Visual Studio. Go figure.
2 Likes
Puzzling ā I guess the pausing of the RM data does not happen in OneDrive ā made a simple Change in RM and then it Updates to OneDrive Cloud (my uploaded speed is somewhat slow (only 20 mb up)
However, I still do not have issues. (although one might want to minimize the versions to not take up space) My database is currently around 180mb
I read somewhere that sqlite3 is used extensively by or is embedded in the MacOS, perhaps even used by the file system. So there may well be a different behaviour for OneDrive on MacOS from that for OneDrive for Windows and perhaps more certainly so for iCloud Drive on the two OSās.
This summary may help us understand whatās going on:
I believe that SQLite does only full-file level locking.
To clarify, I was referring to the āMyRootsMagicā online personal tree. Whenever people attempt to view any media in the Media Gallery, an error states āthis image failed to loadā. It would be nice to have the ability to set up our family tree so that each media will load the document/image based on the local-file location and/or image URL we enter.
The normal behavior for RootsMagic currently is that when youāre working with the local file thatās saved on your computer (i.e., when youāre working offline), it pulls the media location off your computer.
The feature to add is for when someoneās viewing the family tree thru MyRootsMagic online. Maybe Great Uncle John Doe has a 1950 census and a profile photo attached. You click on the census (located on google drive), and it displays the document. You click on the profile photo and it displays the full image (located on Flickr).
This makes all media accessible to relatives who have access to the family tree online, no matter where each media is stored. (It also accounts for the multiple/various online file storage options people use.)
Why didnāt you say so in your OP? How could anyone guess the unusual context you had in mind?
Your concept presents security challenges and risks that would probably deter it from ever being realised. However, it would allow for a potential bypass of the 200MB limit on oneās MyRootsMagic.com accountā¦