That’s certainly distressing. I will investigate a little more myself, but I’m sure I will now need to start pausing OneDrive just like I used to have to pause Dropbox.
I’m putting my computer to bed. I’m continuing my experiment of leaving OneDrive active every other night and paused every other night, and leaving both my RM7 and RM9 databases open overnight. I want to see if I get an Access Violation when I resume using RM in the morning.
This is the night to leave OneDrive active. Therefore, I checked the status and it reports that OneDrive is not syncing my RM databases because they are active. That’s the normal behavior I would have expected until I saw Tom’s message.
No SQLite manager is presently running. I’m on Windows 10, all updates installed, latest version of both RM7 and RM9.
I had a hard time getting a screen shot because the OneDrive pop-up window disappears when I initiate my screen grab software. So I took a (terrible quality) picture of my screen with my phone.
1 Like
This morning, there was no Access Violation in RM9 when I clicked on the Home tab. But OneDrive no longer reported that my RM9 database was still in use even though RM9 was still running and my RM9 database was still open. Instead, OneDrive reported that my RM9 database was backed up. This is in accord with Tom’s recent observations.
OneDrive did report that my RM7 database was still in use and was in need of backup. Obviously, something has changed, either in OneDrive or in RM9. RM7 and OneDrive together are still an open question for me since my OneDrive is still waiting on RM7 to shutdown before doing the backup
Jerry and Tom: your detailed testing makes it clear that having your database in a cloud service adds a serious risk of sync damage and that changes by an apparently safe cloud service happen.
I duplicated my largest database (only 400 people) and put the renamed FTM and RM9 test files in my iCloud Drive folder only. After they uploaded I opened each in their program on my mac, added a new person and moved to the root person. While the programs were open the files did not upload and showed as waiting to sync. Once each program was closed both files immediately uploaded.
Apple’s iCloud is designed to share files across devices under the same Apple ID or reduce file storage on a mac and access as need from the internet so it may behave differently than general storage and backup services.
That is consistent with the way OneDrive used to work for me and still does for Jerry but only for RM7, not RM9. I will test FTM and Heredis which also use SQLite but I suspect it is a behavioural change outside the applications. I used to have iCloud for Windows but probably no longer.
Thinking about it, iCloud is probably designed to prevent synching of an active file until it is saved or closed because its focus is on sharing files for one user among devices and keeping a master copy in the cloud.
Apple programs/services ported to Windows OS probably do not work very well or in the same manner.
I’m not a Mac user, but it seems to me that iCloud potentially has the same risk that other cloud syncing services have in syncing with SQLite databases. Namely, iCloud might sync SQLite databases while they are still open and being updated. There would need to be some evidence one way or the other before declaring iCloud safe or unsafe. iCloud is not necessarily safe for SQLite databases simply because it’s an Apple product and because it’s safe for other file types.
For one example, the evidence that Dropbox is unsafe with SQLite databases is two-fold. One piece of evidence is that you can have an RM database in use, and you can then see Dropbox doing an immediate sync while RM is still running if you make a change to your RM database. The other piece of evidence is that RM users have reported corruption to their RM databases as the result of this Dropbox activity. I have never understood the exact mechanism by which Dropbox reading an SQLite database can cause corruption. It seems to involve the reading by Dropbox preventing updates by RM. But even if there were no corruption, this mode of Dropbox operation is going to generate an enormous amount of disk and network activity that will have an adverse impact on your computer’s performance.
For another example, the evidence that OneDrive used to be safe with SQLite databases was two-fold. One piece of evidence was that OneDrive would report that it was unable to sync an SQLite database because it was in use. The other piece of evidence was that there have been no reports of corruption associated with storing an SQLite database in the OneDrive folder. However (and for reasons unknown), there is now evidence that OneDrive will sometimes sync an SQLite database even though it is in use. That means that OneDrive should no longer be considered safe with SQLite databases. So OneDrive needs to be paused while using RM if the RM database is stored in the OneDrive folder.
Therefore, it seems to me that Mac users need to do some actual tests with iCloud and SQLite databases. For example, if you open an RM database which is stored in the iCloud folder (or however iCloud does it), can you see that iCloud is not syncing the RM database because it is in use?
I was a Dropbox user for years who used it safely with RM by pausing it whenever I was using RM. A non-genealogical product that I use heavily from time to time is called Microsoft Visual Studio. Since 2015, Visual Studio has been using SQLite databases. I didn’t realize that to start with, and sometimes my Visual Studio would crash because all my Visual Studio files were stored in my Dropbox folder. Many months later, I discovered that Dropbox was causing the Visual Studio crashes. I also discovered that Visual Studio works well with files being in OneDrive. Apparently, MIcrosoft realized that they had a problem with OneDrive and Visual Studio and they made OneDrive safe for Visual Studio by not syncing SQLite databases while they were open. So I moved my RM databases and my Visual Studio files from Dropbox to OneDrive and I have been a happy camper ever since. Until now. I am now pausing OneDrive when I use RM. It’s still an open question whether I will need to paus OneDrive when I use Visual Studio in the future.
So I urge Mac users to be cautious and test whether it’s safe to store RM databases in such a way that iCloud will sync them. I don’t know if iCloud has a Pause option. And if it does, I don’t know if RM databases are safe being synced in iCloud when iCloud is not paused. So please research the question and please test before trusting iCloud and RM to work together. It would be great if it works, but research it and test it first.
Remember that the official recommendation from RM is never to store an RM database in a cloud synced folder. Instead, store RM databases outside of cloud synced folders and store RM backup files in cloud synced folders. I think that’s a good recommendation. I have ignored the recommendation by pausing Dropbox (and now OneDrive) while I was using RM. But it’s a good recommendation.
2 Likes
Jerry: in my comment I basically did the testing you mentioned. FTM and RM9 unique files were stored only in my iCloud drive and not on my mac. When opened on my mac neither file updated after changes until the file was closed or the program quit. iCloud is not a sharing or synching service so it may be different from other mainstream cloud services. The only special feature of Apple’s iCloud is the design/use focus on one user…multiple devices rather than many users.
Personally I only keep databases on my mac for control and to ensure they are backed up automatically.
iCloud Drive for Windows behaves with RM7 and RM9 database files the same as OneDrive does on my Win 10 computer, i.e., both are actively synced, meaning frequent uploads to the cloud while work is going on.
Neither iCloud Drive nor OneDrive has the option of keeping the file only in the cloud except when opened by RM or SQLite but both have the “Free up space” control which removes the file data from the local drive but leaves a shortcut or virtual file. Both RM7 and RM9 had no problem opening the virtual file - there was simply the delay of the download added to the opening procedure.
What was more intriguing, however, is that initiating “Free up space” on the local copy while it was still ‘open’ in RM or SQLiteSpy suspended that operation until the file was closed or detached by the application. However, you could still work on the local file in the application and those operations continued to sync. So those cloud sync systems can detect that the sqlite database file is registered/open in an application.
iCloud seems very different on Windows. On mac your files are either in cloud or on mac but not both. If you switch off “Desktop and Documents in iCloud” you end up with files only in the cloud and new empty desktop and documents folders on the mac. Then you can copy from cloud to mac to get them back locally. I don’t think I have a "free up space"option on mac.
A very competent mac software guru recommends using the cloud for your files but I want them only on my computer under my control and automatically backed up in a way that I understand.
Another major design purpose for apple’s icloud is daily automatic iphone and ipad backup to the cloud. Users can turn this off (dumb move) but most are clueless about settings and will be protected despite themselves.
I have looked at “robocopy /?” and the options seem rather overwhelming.
When I am just copying one file from OneDrive to my hard-drive - or the reverse when I finish a session of working on it, are options needed to ensure the file I place on my harddrive is the same as the one I last edited and then uploaded to OneDrive (i.e. things like timestamps, attributes, directories, etc are maintained)?
I have changed my Documents folder to D:\OneDrive\Documents, so I am guessing the command to bring database Wyn2021.rmtree from the cloud onto my hard-drive would be:-
ROBOCOPY OneDrive – Personal\Documents\Family History\Wyndham\ D:\OneDrive\Documents\Family History\Wyndham\ Wyn2021.rmtree.
Does that automatically overwrite the file that is already at that location on the hard-drive?
Also, I have noticed that while I am editing a file the OneDrive Personal icon shows “Processing Changes”, it only switches to reporting “your files are synced” and “up to date” after I close the file – I have noticed that with text files, I haven’t checked database behaviour. Does that mean the edits are not being uploaded to the cloud until the file is closed, or is it regularly uploading changes but continuing to report “Processing Changes”?
Of course as often as not the file I am copying from the cloud will have last been viewed or edited on a different device (I believe every time an RM database is opened but not edited, then closed, it obtains new time stamps – hence new upload to OneDrive).
Apologies for a string of questions – I haven’t used many command lines for ages, and if I start taking this precaution when working on databases I want to be certain I am not making things worse.
I understand what robocopy does, but I’m not sure why you would need to use it with OneDrive. My understanding and use of OneDrive is such that the concept of copying from OneDrive to my hard-drive or vice versa is not really the way it works.
The OneDrive software creates a OneDrive folder on my hard-drive. The OneDrive folder is just one of many folders on my hard-drive. I can place files in the OneDrive folder just as I can place files in any other folder. The difference is that the OneDrive software syncs the contents of the OneDrive folder on my hard-disk with a second copy of my data on a Microsoft server somewhere in the world. As a result, I automatically have a second copy at all times of any files a place into my OneDrive folder.
The syncing is accomplished by the Onedrive software itself doing the copying. So I don’t understand what role that robocopy would play. The only thing I can of is that you are storing your data primarily in a folder on your hard disk which is not your OneDrive folder. Then you are using robocopy to copy your data to and from where the data is on your hard disk when you are really using it and the OneDrive folder. That seems like sort of a manual sync rather than using OneDrive’s automatic sync. Is that your concept?
Hi Cedric,
This is the command line I use when copying from OD to my hard drive:
robocopy “C:\Users\Larry Arnett\OneDrive\Genealogy” “C:\Data\GENEALOGY” ARNETT.rmtree /XO /log:“c:\Data\GENEALOGY\RoboCopy_to_HardDrive.txt”
The /XO ensures that the file on OD is newer than the one on my harddrive, if it isn’t the copy doesn’t happen
I also create a log file so I can see what has happened if I am curious.
I have this command stored in a .bat file so that it can be run by the system whenever I unlock my computer.
I have another .bat file that copies from harddrive to OD with the same optons:
robocopy C:\Data\GENEALOGY “C:\Users\Larry Arnett\OneDrive\Genealogy” ARNETT.rmtree /XO /log:“c:\Data\GENEALOGY\RoboCopy_to_OneDrive.txt”
I then use the Windows Task Scheduler to create 2 tasks: 1 that triggers when the workstation is locked and one that triggers when it is unlocked.
The "unlock’ task runs the “to Harddrive” .bat file and the “lock” task runs the “to OneDrive” .bat file.
I set the same files and tasks up on my laptop so both machines stay insync.
I have used this for a couple of years without any issues.
@thejerrybryan, On 24 June (might have been 23rd if you are in a western hemisphere time zone) I responded in this discussion (seems to be numbered 7/33 as I write this, no doubt once I post this it will be 7/34). @Spyder’s reply to my post (17/33) raised an issue I previously didn’t know about - but someone on another genealogy forum has confirmed the issue, and recommended I use @Spyder’s method. My recent questions were about how I do that, he has given me all I should need (just below your answer).
And about your other questions, Family History folder that contains a couple of RM databases is within my Document’s folder, which is within my OneDrive – Personal folder on my D:/ drive. I use it to both synchronise my desktop and laptop without my doing anything, also to keep a backup in case something happens to the file on the computer I last worked on.
Thanks for your response
Spyder, thanks, those command lines should be all I need.
I haven’t used Windows Task Scheduler before, so I will look it up.
Cheers