Uploading to Ancestry from RM 8 - Avoiding huge numbers of images

Restoring from a backup with media, puts the backups of the media into a folder under the restored database file which has its media paths repointed there. I think I have that right - it used to.

So the results of my experiment:

I went through the steps I outlined earlier, I created a separate folder called Rootsmagic - Testing and move/renamed the existing database into that folder (I have realised that the move/rename does exactly what it says it and gets rid of the original database, however as I have everything backed up multiple times I can replace the original DB in the original folder by just copying back, I use software called Beyond Compare which I have used for years on my image folders)

I unlinked the tree from Ancestry.

I had created an empty sub folder called “Blank” under the folder where all of the "Ancestry Images had been placed, so in settings files I changed the Media Folder to this sub folder. I checked the Media tab and all of the images had a red cross and a warning that the image could not be found.

I then went through the upload and creation of the new Ancestry Tree I left the upload media tick box ticked. The whole upload took just over 8 mins compared to the 1hr 15mins that my original took!

The loading of the media files was about 1 minute. Afterwards I realised that some of the images that appeared in RM had been actually uploaded directly to Ancestry (I should perhaps explain that I recently took over responsibility for this project from another family member, so have been trying to understand not just RM but also the structure and locations of the tree) so they did not go up. So I will have to move all of the photos etc that were uploaded and downloaded to a different folder that I can reference, or perhaps just find them in the Ancestry download media folder.

In fact thinking about it and looking at the filename in the Media part of the Person Sheet the location is there, so why did it not load. The only difference I can see between the images of me that I loaded yesterday that did load and some of my Grandfather taken in the 1920s that were found online is that the “include in scrapbook” tickbox is not ticked. So I will have to do some more experimenting.

When I clicked on the source files in ancestry they appeared almost instantly. I don’t know whether Ancestry locates the sources and loads the images in the background or whether it happens when you click.

So all in all I think I am making progress, will try another upload after looking at the include in scrapbook issue, there is no mention of it needing to be ticked to upload in the RM8 Wiki.

Such fun!

My experience is that Sources and media attached only to the ‘person’ do not upload to ancestry via treeshare. Sources and media that have been attached to “Facts” do upload via treeshare. Good find on the ‘include in scrapbook’ setting. I thought this only impacted RM reports but it makes sense that it would carry over to exports.

I hope that that setting has no effect on exports. It never used to. If it is implemented, then there must be an option in the export and Upload dialogs to choose All or Scrapbook Only. That’s the only way it would make sense.

Good point. I was thinking that it would make sense if ‘scrapbook’ aligned with Ancestry’s ‘gallery’ but your comment is more well thought out.

So life caught up with me for awhile so have not done much on Ancestry and RM8.

I have done a bit more testing and have found the following:

  • include in scrapbook. I have found this to be variable, so sometimes items marked with include in scrapbook do upload sometimes not. So finding it hard to understand whether or not this has any effect. I suspect that it does not.

  • Kevinm stated “Sources and media attached only to the ‘person’ do not upload to ancestry via treeshare”. I think that the reason for this is that they also do not cause the person to appear on the changed list. I found that if I change a Source and Media attached to the person and then go to Treeshare and go to that person it does show that there have been changes and them you can upload them! I am still waiting for a response to my question to Rootsmagic as to what triggers the Changed List. Perhaps they do not know?

  • Uploading Experiments: I think I have come up with a solution which for me works 100% of the time. In an earlier post I talked about creating an empty sub folder called “Blank” under the folder where all of the "Ancestry Images had been placed, then in settings files then changing the Media Folder to this sub folder, but not move any images. I checked the Media tab and all of the images had a red cross and a warning that the image could not be found. As stated earlier when I tried this it reduced the load time from 1hr 15 to around 8 mins. However, the next time I tried it the load time was back to over 1hr. I then discovered that RM8 had changed the location of the individual files (all 5388 of them) to the “blank” sub folder! So then I created a completely separate folder called “Media Temp Location”, and moved all of the 5388 files to that folder. I then uploaded to Ancestry and it was back to around 8 mins (the timings are: Media finished at 1:04, Tree finished at 7:34 and Collect changes finished at 8:26). After the upload is finished I copy the 5388 files back (which on my M.2 SSD takes about 30 secs) and when I click on the Media tab all of the files are found (no Red crosses). So as this worked 100% of the probably 10 times I have tried it. So based on this and the unreliability of the change list I am updating my data in RM 8 and doing lots of backups and then spending the 8 mins uploading to a new Ancestry tree and delete some of the previous test trees.

The issue that I am also having to how to deal with Ancestry hints. I have realised that I cannot trust the Treeshare to highlight any hints that I accept and incorporate into the Ancestry Tree, so that I can copy them to RM8. How do people deal with incorporating Ancestry hints into RM8 without using Treeshare. I did try doing some manually but that is a pain or at least the Census find that I tried yesterday was convoluted to put in mildly. Any suggestion gratefully received.

With regard to Media locations I have, for now, adopted kevinm’s method of having a separate dedicated media folder. One of the reasons for this is I found that when I tried to add .jpg files that were stored in directories that had very long paths RM8 would not add them. This was irrespective of whether I browsed to the location or dragged and dropped them. However, when I copied the files to the dedicated media folder I could add them by either method. So I wonder if RM8 has adopted the “Long File Names” discipline that Microsoft introduced in I think 2017. The registry in my PC has been set to use them and other applications work. I will ask Rootsmagic once they respond to my current questions.

I just do it manually, pain or no pain. That way I can name the downloaded images the way I want them named. I can store the downloaded images in the folders where I want them stored. And I can create sources and citations using my own source templates that look like I want them to look.

I’m probably just a glutton for punishment, but downloading the images and setting up the sources are not the most time consuming part of my workflow. I also do a complete transcription of any document I download. The transcription becomes a note both for a fact and for a research note for a citation. So by comparison, just downloading the image manually and entering the source manually is a relatively small part of the job.

I don’t use Treeshare. Every item of data in my database is entered manually. I don’t download data from anywhere but do sometimes turn on hints for Ancestry or FindMyPast. Usually I find that I have already found the information in the “hints”. I do of course download media which I save into my media folders/subfolders renamed as per my standard naming format.

I have had a reply from Rootsmagic regarding this issue:

Michael,

Sorry to take so long to get back to you. Sadly, the answer is with the Ancestry API that the upload of media is all or nothing. You would physically have to remove the media from the Media Galley so it doesn’t upload when creating a new Ancestry tree or adding a new person or source that has one. We have a request to be able to privatize media so it cannot be uploaded, and hopefully that will be possible one day. We don’t have a lot of control over the Ancestry API since Ancestry is the one that builds and maintains it.

So it looks like the only way to avoid loading 1000s of images is to use a workaround like the one I described above not ideal but it does work.

That response seems illogical. RM passes data from its database to the API so surely RM has control over what it passes to the API. If it passes no media path and name to the API, the API has no media file to upload just as if it is given a broken path.

TomH makes a good point. However I suspect that as was stated to me it is an all or nothing. So if you do not tick the “Upload Media” during the upload process than this is no media path is passed and that also excludes the "personal photos that have been added as well as the files downloaded from Ancestry?

What would perhaps be better is to have more than one Media location in RM 8, one for the files you want to upload and the other for the “Ancestry created ones” and during the upload you get to chose whether you want to upload from location, 2 locations or to upload no Media.

Like Jerry,

“I just do it manually…[and] name the downloaded images the way I want them named. I can store the downloaded images in the folders where I want them stored. And I can create sources and citations using my own source templates that look like I want them to look.”

And Like Terry,

“I don’t use Treeshare. Every item of data in my database is entered manually…I do of course download media which I save into my media folders/subfolders renamed as per my standard naming format.”

I assume at least some RM users maintain family trees on additional web sites beyond Ancestry, like MyHeritage or FindMyPast, and that the difficulties with Ancestry described in this thread also occur on those other sites.

If I were just using RM for myself I wouldn’t bother trying to deal with online trees at all. But to share my work with other family members and encourage them to contribute information, I’ve posted my tree in multiple locations. So far I’ve done so on JewishGen, Ancestry, MyHeritage/Geni, and GEDMatch.

For me the least stressful way of dealing with the complications of multiple formats has been to upload a GEDCOM to each site, and then, when I’ve accumulated a meaningful amount of new data in RM, I delete the tree from each site and upload a new GEDCOM.

I’m open to suggestions for doing things more efficiently, as long as I don’t have to use SQL (a skill I lack).

[BTW, in the same spirit, I’ve tested my DNA at FamilyTree (y-700, mt full sequence, autosomal), 23&Me, and Ancestry, and also uploaded the results to MyHeritage/Geni and GEDMatch. As I’m sure most of you know already, genetic matching has a whole other universe of complications.]

And who said “genealogy” was a hobby - perhaps creating “ghost trees” is a work-around but it only goes part way to solving my conundrum. Thank you for your thoughts.
Cheers

I have received a response from Rootsmagic regarding the issue that I hadwith the “Changed List” not actually showing changes. This was not as a result of my original question but one that I raised in the question about uploading media

I looked it over and the “Changed List” for TreeShare can actually be triggered by Ancestry simply moving things around in the background. We’ve seen trees have all the people showing as Changed when nothing was actually changed. So it’s hard to say sometime what triggers people being added to it. There are a few things on the RM side that we do not trigger the changed list. One is place name changes. If you happen to clean your place list everyone using that place will not slow up on the TreeShare changed list. We initially had that update the change list but it made the Place List extremely slow to the point it was unusable.

Sources are another one. If you modify an existing source it also won’t show on the change list. The Ancestry API and website for that matter doesn’t handle updating source citations. It’s better to delete the source and add it again if you need to change it.

Media should show as a change, but if you added it on the general media or Person row on the Edit Person screen it won’t trigger it to show on the Change List. It won’t even show the primary media as an option to upload if you added it to someone. You would need to uncheck the primary media box to make it appear. So media can be a little difficult depending on where you have it added. We can add primary photos to everyone when creating a new Ancestry tree.

So it appears that the change list is not actually very comprehensive and by the look of the statement "So it’s hard to say sometime what triggers people being added to it" it appears to be a little bit random!

I will await further developments from RM but I think my stance of not trusting the Changed List and uploading fresh trees is prudent. Perhaps a health warning needs to be added to the Changed List?

I may be remembering incorrectly, but I remember the problem slightly differently. I don’t remember the Place List itself being slow when the change list was being updated. Rather, I remember the problem being the hundreds or thousands of people needing to be approved for upload to Ancestry on a case by case basis after any kind of Place cleaning operation.

Agreed. That’s another odd statement coming from Support and definitely inaccurate. Who signed the support response?

Not sure I understand the issue here. Finding a census or other document on ancestry I just click Review, scroll down to View the original and click the tools icon. Then click Print AND choose Include transcript and source citation. You end up with a downloaded pdf file with the original, transcript and source citation in one painless swift bundle. Then I Ignore that hint in ancestry.

The issue has to do with which direction data is flowing between RM and ancestry. If you only have data flow from Ancestry to RM as in your example, that is one experience. If you only have data flow from RM to Ancestry to, that is a different experience. If you have data flow in both directions between RM8 and Ancestry, that is a still different experience. And if you disconnect and RM8 database and an Ancestry tree and then make a new Ancestry tree from RM8, that is yet again a different experience.

This thread is about data flowing from RM8 to Ancestry. The RM8 tree was created from Ancestry, but now is flowing the other direction from RM8 back to a different Ancestry tree. The question is how to avoid all the images that flowed from Ancestry to RM8 from flowing back to a different tree in Ancestry. The Ancestry API and the TreeShare interface to the Ancestry API only know that the files are the same files so long as the same RM8 database remains connected to the same Ancestry tree. As soon as you disconnect an RM8 database from Ancestry and then upload to a new Ancestry tree, the whole process loses the connection of the images being the same between RM8 and Ancestry.

In FTM 2019 you control which way data flows up to ancestry or down to your tree. I guess RM8 lacks that ability to avoid auto contamination of your database from ancestry. I work ancestry hints and manually download any valid finds only syncing up to ancestry later.

RM’s TreeShare feature allows you to copy data in either or both directions. It’s just that it doesn’t flow automatically in either direction. When you use RM and Ancestry together, you start out by copying an entire RM database to a new and empty Ancestry tree, or by copying an entire Ancestry tree to a new and empty RM database. Thereafter, nothing flows. You must manually approve each individual piece of data that is copied in either direction.