Proposal for treeshare improvement

[AlanWatson]

This suggestion only deals with the part of treeshare, after an intial upload/download, which allows the user to exchange further information between RM and Ancestry.

As somone whose main tree (of 55,000 individuals) is in Ancestry, Treeshare is potentially a major benefit in RM, but some bugs and missing features and the need for an excessive number of clicks makes it only borderlne usable for me. I think it could be improved significantly without too much difficulty. I suggest fixing some bugs, making some improvements and cutting out at least 90% of the clicks, more if possible.

Fix bugs that corrupt data
I am aware of two of these.

1. When using the main screen to add relatives of a person who is already matched, adding two children with the same first name creates corrupt data in RM, which is not cleaned up by the database tools. This situation happened quite a bit when one child died in infancy and a second was given the same name.

2. After adding a married/divorced couple to RM, the process of transferring them to Ancestry creates corrupt marriage/divorce data there. The two people in the couple are inevitably transferred one at a time, each time with their marriage. The result is that the first person ends up with two marriages to the same spouse only one of which appears on the spouse. It is appalling that Ancestry’s interface allows this to happen, but even so RM should avoid triggering the problem. To do so, it should not add a marriage to Ancestry unless the spouse already exists there.

Abolish update screens
When you choose to update (rather than add, link or delete) a person, a name or an event, you are presented with a screen which asks you to select the specific items to update. In RM 7, you had the option update all, but even this took several clicks, In RM 9 you have to select each item separately, which is intollerable. I think that the whole thing is over-engineered and should just be dropped. I have never chosen to update one item and not others and I can’t see why I ever would. RM should just update everything.

Alternatively, the main screen could contain two update options, ‘update detail’ which would open the current screen and ‘update all’ which would have the same effect as ticking all the boxes in the current screen without ever opening it.

Improve transfer of marriages to RM
Transferring marriages from Ancestry to RM is a hassle. RM currently requires you to add the spouse and the marriage in two stages, which requires you to toggle between showing everyone and only showing unchanged people. It should allow you to add a marriage for a spouse who already exists in the database or is being added; the database procedure would then obviously add the new spouse first and the marriage afterwards.

Improve adding of not changed marker
When Treeshare first came out, the user had to click the option to mark a person as not changed after accepting all the changes. Then RM updated the marker automatically whenever a person was added or changes to the person were made. The problem with this is that people with new events (like marriage or divorce) or new relatives to add are marked as not changed even though more processing on them is needed. I have to be extremely careful if I am adding people who married each other; if I add both the people concerned from their parents, then I may never notice that a marriage is missing. People should only automatically be set to ‘not changed’ when they have no missing events or relatives.

Default the choices from the main screen
RM already goes some way to deciding whether person details, names, events and relatives of the person in focus match, appear to need updating or appear to be new (or candidates for deletion). It should complement this by selecting the appropriate option for the user.

To help RM, I would give the user a button at the top of the screen to toggle between ‘Update RM’ and ‘Update Ancestry’. RM would then use this selection to populate the choices for each item. For example, if there is one name on both the source and the destination but they differ, I would update the name. If there are two names on the source and one in the destination, I would add the second name. When a new marriage event is present in the souce and the person has only one spouse there, I would default this person as the spouse for the marriage. When an event is missing from the source and present in the destination, I would select to delete it.

The end result would look rather like the screen does now after going through the individual items, except that it should be possible to see and modify the option that RM has chosen, or indeed to see that RM had not been able to choose an option, eg with one marriage to add and two spouses. When finished, the user would accept changes as now.

Get rid of phantom updates
There is one rather obscure case in which RM wrongly marks its own data as changed adding an uncessesary person to be managed in Treeshare. This happens when using the ‘Share data’ function to match someone between RM and FamilySearch. If person A is already matched between RM and FS, I share data for person A and from that screen match some of person A’s relatives, then person A shows up as changed the next time I open treeshare. The people who have been matched are not affected.

I believe that something simlilar happens as a result of certain person changes in Ancestry which are not mirrored to RM. Certainly if I add a web link in Ancestry, the person appears changed in Treeshare, although the web link never appears in RM. There are also some well known problems where RM shows a different number of name and person sources between RM and Ancestry, apparently because of differences in the data model. I have no idea whether it would be possible to fix these.

Improve functionality
No doubt there are many ways in which functionality could be improved. I am particularly keen on two.

1. Bring web links into RM. I have many thousands of web links in Ancestry, which I started using long before RM. RM is the main backup of my Ancestry data, and it is a significant weakness that the web links are not present there. I have no idea whether they are available to you in the API, but if they are, you should bring them in as web tags

2. Give users an option on replicating ‘cosmetic’ changes from RM to Ancestry. Some time ago, RM stopped actions like merging place names from triggering an update to Ancestry. This makes sense for people who only use Treeshare and Ancestry to generate hints in RM, but it makes these data administration tools in RM useless for people whose master tree is in Ancestry. Since our requirements are completely inconsistent, it would make sense to give us a system (or file) option about whether or not to mirror these changes to Ancestry.

I hope that these suggestions receive a positive response from RM. Needless to say, I would be very interested in hearing others’ comments and suggestions and would be happy to expand on/clarify anything which is not clear.

Alan

[rzamor1]

1. When using the main screen to add relatives of a person who is already matched, adding two children with the same first name creates corrupt data in RM, which is not cleaned up by the database tools. This situation happened quite a bit when one child died in infancy and a second was given the same name.

I’ve not seen this. If you have an example with your tree that you can show us this happening then open a support ticket.

2. After adding a married/divorced couple to RM, the process of transferring them to Ancestry creates corrupt marriage/divorce data there. The two people in the couple are inevitably transferred one at a time, each time with their marriage. The result is that the first person ends up with two marriages to the same spouse only one of which appears on the spouse. It is appalling that Ancestry’s interface allows this to happen, but even so RM should avoid triggering the problem. To do so, it should not add a marriage to Ancestry unless the spouse already exists there.

This is an Ancestry issue and how they handle family facts. Only add the marriage fact when both individuals are on the receiving side. One spouse will always be out of sync.

A number of these enhancements requests RM does not have control over. Ancestry is the one that determines what is available in the API they give us. Some of them are listed in the FAQ. The link is on the bottom left corner of the TreeShare screen “View help on using TreeShare and WebHints for Ancestry”. It is found in the RM9 Help here -

Ancestry TreeShare

[thejerrybryan]

I agree with all your concerns, but I suspect that many of them can never be addressed due to limitations of the Ancestry API. Therefore, I long ago concluded that the only practical ways to use TreeShare was to update only Ancestry and download to RM or else to update only RM and upload to Ancestry. In this mode of operation, you can periodically delete your RM database and download anew from Ancestry, or you can periodically delete your Ancestry tree and upload anew from RM.

[russhart]

I’m like you in that I use Ancestry as my main point for research and then use RM as a backup, and use all the great tools to check for errors etc.

Jerry & Renee are correct in that there is not a lot RM can do with Treeshare (other than to continue a dialogue with Ancestry) as Ancestry control what features their API will work with.

Re your point on “Abolish Update Screens”, I don’t understand why you have raised this as RM9 does include a single check box which when selected will add all the fact updates for that person in one hit. This feature was included in RM7 but dropped in RM8.

Re your point “Improve Functionality”, there is a post from Renne regarding the issue over updating Place names, where a process will trigger that change in Treeshare. You can find it here

Place Name Merge and Tree Share - RootsMagic - RootsMagic Community

Overall I find Treeshare very helpful and easy to use, once you understand its limitations.

[AlanWatson]

This is from my main tree in RM7

All the initial corrupt entries in the index were created by Treeshare and have not been cleared by repeated use of the database tools (after first deleting all the visible data in the corrupt records.)
This is from a small tree that I am playing around with in Ancestry and downloaded to RM9 via Treeshare. After doing so, I added a make believe wife and two daughters

After adding the new records, the second daughter is only partly there

She appears on the edit person screen as a blank space

And nothing changes when I run all the database tools.
If I go back to treeshare and add the second daughter again, then the family view shows this

This is slightly different from RM7. There I could see the phantom record in the side panel, sort of edit it to delete some of the data and unlink it from its parents. I don’t seem to be able to do any of that in RM9.
It is easy to try it for yourself.

Alan

[AlanWatson]

Responding to RussHart’s comment on ‘abolish update screens,’ I see that RM9 has reintroduced the ‘update all’ option dropped in RM8. I apologise for missing that. Even so, my point still stands. When I go to an item that has been updated and click on the action sign, I see this.

I would like to trigger the update of all the attributes associated with the person’s name by selecting ‘update existing name in RootsMagic’. Ancestry is my master file; why would I ever not want to update everything? If RM were my master file, why would I not want to update everything in Ancestry?
Instead, I get to the ‘update screen’, where I have to click to update all and ‘OK’, two more clicks.
My main objective is to make treeshare much more usable by reducing the number of clicks very significantly (target reduction >90%), so why would I not want to cut these unnecessary clicks out?
As I wrote in the original post, an alternative solution that would accommodate people who want the ability to update items individually, would be for the original screen to give me three options, one which would add the item in Rootsmagic as new, one that would open the current update screen and one that would mark all aspects of the item to be updated as if I had opened the screen, chosen to update all and clicked OK.
I can’t see any disadvantages to doing this, and I don’t think that it would be difficult to do.

Alan

[rzamor1]

When you see those really high record numbers it means there is corruption in the database. Try doing a drag n drop. I wouldn’t continue to use that database until you have done that.

[AlanWatson]

Hi Renee,

Thanks for your feedback. As I understand it, I can’t drag & drop into a new database and retain the link to my Ancestry tree. Is this correct? If so, I will have to download the whole thing from Ancestry again and manually reconstruct my groups, not problem list, colour coding, etc and re-link everyone to FS. Is that right?

Also, are you able to duplicate the problem from Treeshare that creates the corrupt data in the first place? I hope that you will add it to the bug list.

Alan

[rzamor1]

Drag n Drop and even GEDCOM will both retain the link to the Ancestry tree and FamilySearch matches. Groups are not preserved but color coding is. You could color code each group. Use the different color sets if color coding overlaps the groups.

More than likely the issue you are having was due to corruption in the file in the first place. It will only continue as you use that database as is. Make sure you do not have the database inside of any folder syncing with the cloud like OneDrive, Google Drive or Dropbox.

[AlanWatson]

Many thanks for the information on the drag and drop. I am trying it now - it has been going for 14 hours so far.

However, as you saw above, I replicated the issue on a completely new RM9 database, one which was created by downloading a new small tree from Ancestry and then amending it there. That file had nothing whatever to do with the previous one, a different file created by RM9 rather than RM7, so I don’t see how the issue could be related to previous corruption in the file.

To investigate what is going on, I looked inside the RM9 database for my small test file. The file that I originally downloaded from Ancestry with Treeshare had 55 people. I then added three extra people, ‘wife’ and 2x ‘daughter’ to Cadwallader Douglas, who was person no 12 in the database.

As you will have seen from the screenprints, ‘wife’ and the first ‘daughter’ seemed to add properly from the Treeshare screen whereas the second ‘daughter’ originally created a partial record. When I went back and added the second ‘daughter’ for the second time, it appeared to create a proper record, although the partial record for the original second ‘daughter’ was still there.

I can see exactly the records in the database. ‘Wife’ and 2x ‘daughter’ appear as the last three entries in the ‘NameTable’ and in the ‘PersonTable’, numbers 56, 57 and 58. The new family formed by adding the new spouse and new children appears in the ‘FamilyTable’ as FamilyID 18 with FatherID 12 and MotherID 56. But when I look in the ‘ChildTable’ for FamilyID=18 I see three children, not two. These are obviously the three children who appear on the screen when I look at the family. These three children have ChildID 57, 700 and 58. Screenprint below

After finding this, I ran all the database tools as recommended. They found no problems. The dummy entry for the ‘corrupt’ child still appears on the screen for Cadwallader Douglas’s family

and child 700 still appears in the ChildTable.

Plainly, RM9’s treeshare process has corrupted the database in exactly the situation that I described - adding two children with the same name from the Treeshare screen for one of their parents.

Please can you confirm that you are adding this to the list of bugs to be fixed.

Alan

[AlanWatson]

The drag-and-drop worked well, despite taking a long time.

Alan

[TomH]

Another surprise is the value in the ChildTable.ChildOrder column: 500. Iirc, that’s been 0 until chidren are manually arranged and then each child of the same parents is assigned a serial number starting at 1.

@thejerrybryan has reported such rogue numbers in PlaceTable.MasterID for PlaceType=0 records. It’s normally 0. And since the community beta of RM8, there have been reports of rogue RIN’s with sometimes extremely large integers.

It’s as though post-RM7 versions are making transient use of some fields and leaving behind detritus. Or there are memory leaks corrupting one’s database.

[kbens0n]

A surprise is that no other report of such a “grayed out” child block has been reported here before. Bugs need a minimal reproducible case.

[AlanWatson]

It has happened to me many times in RM7. Unfortunately, it took me some time to realise what the common factor was - adding two children with the same name in a single ‘accept changes’ operation, ie from the screen comparing one of their parents.

To replicate it in RM9, I downloaded a small, clean database, after having written here predicting exactly what would happen, and it did. That is not just a reproducable case, it is a case reproduced clearly for you all to see. I find it hard to believe that there is anything uncertain about this, but just to be sure, I am doing it again as I type. I have created a new tree in Ancestry with only two people in it and have just downloaded it to via Treeshare to a new RM9 file, as you will see

Then in Ancestry I created two children with identical names (but different dates, just to show that the name is the crucial element here). Going back into Treeshare, I went to the father’s screen to add the two children. This is Treeshare before pressing ‘accept changes’

and after.

See the space for the corrupt child. Coming out of Treeshare and into Family view you see just the same as before, with a blank space for the dummy child.

I then went back into Treeshare and added the second child a second time, giving the same pattern as before - one family with two real children and a blank, corrupt, record.

Just in case anyone is still not 100% convinced, I went into the database to have a look at the content of the tables. Not surprisingly, the NameTable and PersonTable both have four lines in them. The FamilyTable has one entry with FatherID 1 and MotherID 2. The entries in ChildTable are copied below

As before, RM9 wrongly tells me that the database integrity is fine

and the database tools do not fix anything.

For a further demonstration, I went back into the same file, added a spouse for one of the new sons and two more sons for this family, both with identical names. Adding them from Treeshare as before yields exactly the same results

.
At this stage, I have still not re-loaded the second child. As you would expect, the ChildTable now has five entries for two families, two of them corrupt

You might wonder whether I could fix the problem by linking the Ancestry record for the second child to the corrupt record in RM9. The answer is no. Nothing comes up for it to link to

and I am left to add it as a new child, which (as you have seen) creates a new record in the ChildTable, leaving the corrupt record in place.

So I have now experienced the problem many times over, reported it and, after reporting it, reproduced it three times out of three attempts, using two separate clean database files. Is that enough to convince everyone that it is a bug? If not, you can try it yourselves. The steps are extremely easy to reproduce now that you know exactly what they are.

Alan

[rzamor1]

If you have an Ancestry tree that is creating this then open a support ticket. Share the Ancestry tree with Editor permission with the agent. What we won’t work with is a database that has been opened in SQLite outside of the program. So we have to work with a clean one from the Ancestry download.

https://support.rootsmagic.com/hc/en-us/requests/new

Can you tell me where you database is being stored? This is not typical behavior with using TreeShare. If it was so reproducible then every user would have this problem and that is not the case. What we need to look at is where you are storing the database you are working with that appears to have corruption. Is it stored in a folder syncing with the cloud like OneDrive, GoogleDrive or Dropbox, or anything similar to that? Are you working on a NAS network?

[AlanWatson]

My RM tree is being stored on the main SSD of a Dell PC that I bought a few months ago running Windows 11. I started having the problem on my previous PC using its main SSD. Nether of these is syncing with anything else. The issue is related to RM7 and RM9 software, not my PC.

I will create another clean Ancestry tree and share permissions as you request.

[AlanWatson]

As @rzamor1 wants me to share an RM file the tables of which I have never looked at outside the RM user interface, I am creating a new one. As before, I am writing up what I do as I do it. This time, I plan to do the whole thing in RM, 1) creating a file there including a family with children having the same name, 2) using Treeshare to replicate the same file in Ancestry, 3) deleting the children in RM and 4) using Treeshare to recreate the children and demonstrate the completely repeatable bug in the Treeshare software. I will make a copy of the RM file before I have shared it with Ancestry and after the whole process both of which I will put on Dropbox where any/all of you will be able to see the files. Please do try this at home!

Meet the family. Abby Barnard and Charles Dodds were born on the same day and died on the same day. They had six children in quick succession after marrying, two of whom unfortunately died when one day old. These were Edward, Edward, Frances, Frances, George and Harriet.

I have now exited RM and copied the file (test for treeshare.rmtree which is not being synchronised) to my Dropbox and have created a Dropbox link (which you should be able to use to download the file) here.

Opening the file again in RM, I have uploaded it to Ancestry using Treeshare, screen below

Exiting Treeshare, I have deleted all the children from the RM file. New family view

Going back into Treeshare, I am preparing to recreate the children, from the screen matching the mother. Before doing anything

After choosing to add all the children back

And after accepting changes

Here I must confess to some surprise. This may be the first time that I have done this with two duplicate names; I had expected both of them to create corrupt records, and the fact that only one of them has is intriguing. I am puzzled as to what kind of coding or SQL error could have done that. Anyway, here is the family view again with its usual blank space for the corrupt record.

I have not gone back into Treeshare to load the missing child again, but I could easily do so. After loads of experience, I can tell you that the blank (corrupt) record would still be there.

I have now copied the file again (this time called ‘test for treeshare - copy.rmtree’) and put it on dropbox where you should all be able to see it here.

I will now set about the support ticket and will access this tree directly in Ancestry for the first time to share it with RM support.

Alan

[AlanWatson]

Moving back to the general point of improving Treeshare, it is clear that I should have gained a bit more experience of RM9 before posting my comments. Most of the proposal still applies, but I have some RM9-specific comments.

1. I like the RM9 main screen for updating/adding/deleting people and items. It looks good, the < signs are clear and the screen responds promptly to your clicks. Good.

2. In his post @russhart pointed out that I was wrong about the ‘update all’ tickbox. I apologise again for getting this wrong. RM9 has now solved a problem introduced in RM8. Good.

3. My original post also referred to some entries seemingly requiring update although nothing appeared to have changed. This is the same point made here by @rwh. I am now in a parallel run between RM9 and RM7. I can see many items which appear to need updating in RM7 but not in RM9. (Even if you update them in RM7, they still show that they need an update.) I have done quite a bit of testing and RM9 seems to have fixed at least most of the problem. Very good.

4. @JOSO asked in this thread for the screen to indicate whether or not an item had been amended. RM9 seems to me to have fixed this too; the < signs now disappear from the RM9 screen after you have clicked on them so that you are never likely to action the same item twice. Very good.

5. RM9 has a system option ‘clear on accept’ to remove items from the changed list after clicking ‘accept changes’. It can’t be bad to give the user a choice, although it is not that good either; the problem is that we all need to remove some people from the changed list after accepting changes (because there is nothing else to be done) but not others (because there is). I still think that the system should work this out. However, a possible small and useful improvement would be to put ‘clear on accept’ as a toggle at the top of the main screen. If I then actioned 10 people in a row with nothing else to do and another 10 who also had marriages etc to add, then I could manually switch between the two modes. I don’t know whether the system is written in a way that would make this possible, but if so it would be very good.

6. In this thread @gallde refers to a problem where the focus moves to the top of the screen after dealing with an item. I think that this may be a new problem in RM8/9 and that it may not have been understood. The problem arises when there are so many lines on the screen that scroll bars appear at the side. In RM7 the screen returns where it was after you process each line. In RM9, the screen moves not to the top, but to the middle. I had just processed an item right at the top of the screen before it moved here,
   

   
   
   This slows the user down and is annoying. Not so good.

7. There is a similar new problem in how the the index behaves when you switch from ‘only show changed people’ to ‘view everyone’. In RM7, the focus stays on the same person, but in RM9 it moves back up to the top of this list. Every time I deal with someone with a marriage, I have to make this switch, so I need note the person’s name and type it into the screen to find them again. Not so good. (Moving the ‘clear on accept’ option to the main Treeshare screen would largely remove this problem.)

So in summary, RM9 has fixed one fairly significant issue introduced in RM8 and a few other longer-standing problems raised by users. It has also introduced two new problems of middling importance.

I hope that my comments and suggestions here are seen as constructive. That is certainly how they are intended.

Alan

[rzamor1]

Confirming issue has been reported to development.

[thejerrybryan]

@AlanWatson

I can see many items which appear to need updating in RM7 but not in RM9. (Even if you update them in RM7, they still show that they need an update.)

I’m not sure which of your items have this issue in RM7. One problem that seems to have no solution is that notes in RM7 that contain blank lines can never be made the same between RM7 and Ancestry because Ancestry doesn’t support blank lines in notes. Well, what Ancestry really seems not to support is any kind of hard returns that you might use to help format a note. Blank lines are usually produced by two consecutive hard returns. The blank line issue is not the only data that can never be made the same between RM7 and Ancestry, but it’s one that stands out. I have not yet checked to see how RM9 handles this particular issue.

As a result, even immediately after an initial upload from RM7 to Ancestry it is possible for persons in your RM7 database not to match exactly with persons in Ancestry. I don’t use TreeShare in the other direction, so I’m not sure if the problem is equally acute in the other direction. Nobody will be on the changed list immediately after an initial upload from RM7 to Ancestry, but that doesn’t mean that all the data matches between RM and Ancestry.

It’s a subtle but important distinction that TreeShare basically does not provide you with a list of persons who have differences between RM and Ancestry. Rather, it provides you a list of persons who have a different date for the last change. But a person can have the same date for the last change in both RM (or at least in RM7) and Ancestry and still be different. Or a person can have different dates for the last change between RM and Ancestry and still be the same. So the changed list is date based, not data based. You can’t see the actual data comparison except on a person by person basis after highlighting each person in turn.