Merge all duplicate citations - RM10 issue

RootsMagic
1

After using Rootsmagic for many years, I’m now moving on to RM10, and still very much a newbie. The main issue so far is the endnotes in a narrative report – up to 7 times as long as in RM7. After discovering and using ‘Merge all duplicate citations’ in RM10 the endnotes were shorter, but still about 2-3 times greater than in RM7. Reviewing earlier comments in this forum on similar topics did not help.

My sources are very basic – virtually no media, repositories, research notes or comments and no imports from ancestry.com. The issue seemed to be with sources where there was no citation name - citations for these sources were not merged by RM10’s ‘Merge all duplicate citations’, whereas all others were. Delving into SQLite, the ‘Citation Name’ field in RM10 appeared to be the ‘Source Details’ field in RM7. Many of my ‘Source Details’ fields had an empty string, and it seems that ‘Merge all duplicate citations’ does not work on empty strings.

The solution I found is complex:
Import the RM7 database into RM10
Run RM10’s ‘Merge all duplicate citations’
In SQLite replace the empty strings in the ‘Citation Name’ field with text e.g. [No Name]
Run ‘Merge all duplicate citations’ again
In SQLite replace all the [No Name] entries in the ‘Citation Name’ field with an empty string

After this process, RM10 produced endnotes that appeared to be the same as in RM7, at least in my testing to date.
So far I have seen only one instance of ‘ibid’ in an endnote. This related to 2 citations which had identical wording, but only one had media attached – this is possibly the only instance of media attached to a source in my database.

I would be interested to know if there is a simpler way to fix this endnotes problem.

2

While I don’t understand the logic of your fix, the Citation Name column did not exist in RM7 and is separate from the Source Details fields. Its there in support of the “reusable citation” feature.

If memory serves, when you upgrade a database from RM7 to RM10, the Citation Name is filled in with the concatenation of the values of the Citation’s sentence template variables. That yours were empty suggests that either my recollection is wrong or your citation variables were empty. Either way, if you had media or webtags at the individual Citation level in RM7, you may now find some Citations with multiple media and webtags and will have lost many values of Citation Text and Note.

EDIT: I forgot about the update that @thejerrybryan describes and agree with him that evidence is needed for anyone to understand what is going on.

3

A common cause of the blank citations names is import from Ancestry, but you said you didn’t have any import from Ancestry. So the chief or probably only cause in your case is the sources that had no source details data.

I’m thinking that this is sort of a Catch 22 situation. For the longest time, RM8/9/10 did in fact merge citations that had blank citation names. That sounds perfect for your situation. However, the behavior of Merge Duplicate Citations was changed recently so that it wouldn’t merge those with blank citation details. The behavior was changed in part due to requests from users who had the reverse problem from yours. Namely, they desperately needed citations with blank citation names not to be merged.

Their problem was that they had citations that originally were imported from Ancestry. Such citations were really different, but the difference was not in the source details fields but rather was in either the media files or in the Web tag fields. The Catch 22 is that if the recent change were undone, it might fix your problem but it would put the problem back in play for users with sources imported from Ancestry.

I think my analysis is correct for your situation, but it’s difficult to be sure without seeing a few examples of what your citations looked like in RM7. But I think my analysis is pretty close. I wonder if the recent change made by RM was really the wrong solution. It approached the problem by looking at the Citation Name field. But I wonder if instead it needed to approach the problem by looking at media files and Web tags when there was no other source details data.

As far as your exact solution, I don’t quite follow all the details of how you solved the problem or why it worked. Again, I would need to see what some of the your citations with the problem looked like in RM7. But almost certainly I would have approached the problem by using SQLite to fill in something into the citation name field before merging the duplicate citations. I would have tried to make citations that were really different to have different citation names and citations that were really the same to have the same citation names. I don’t know how to do that without seeing your citation data.

4

The original poster shared some of the citations with me privately which helps a great deal in identifying what’s going on.

All of the original poster’s citations in RM7 that won’t merge after import into RM10 are using the Free Form source template and they all leave the Page Number field blank. There is nothing at all wrong with either procedure, and I have a lot of citations in my own database that are set up the same way. As the saying goes, “I resemble this problem”.

These sources are what are called split sources, or sometimes they are called extremely split sources. That’s because 100% of the source data is in the Master Source and 0% of the source data is in the Source Details. And when using the Free Form source template, there is basically only one Master source field, namely the Footnote fields, and there is basically only one Source Details field, namely the Page Number. Well, there are also Master Source fields for the Short Footnote and for the Bibliography, but those two additional fields have no impact on this discussion. What matters is that leaving the Footnote field completely blank is what creates extremely split sources when you are using the Free Form Source template.

Pretty much all the sources in my database are extremely split sources. Most of my sources are now based on source templates of my own design. But some of them are still based on the free form template because I have not yet gotten around to converting them from free form templates to using my own templates. With sources based on the free form template or based on the templates of my own design, if I were doing my conversion from RM7 to RM10 today instead of when I originally did it, I would not be able to do the conversion because of this problem with extremely split sources.

It’s probably worth reviewing that I became an extreme source splitter in RM7 only reluctantly and only because RM7 did not support reusable citations. Suppose I had created a citation in RM7 and then memorized it and pasted it many different places. Then suppose I found a typo or some other error in the citation. Because RM didn’t support reusable citations, I would have to chase down every single place I had pasted the citation that had the error and make the same correction every single place. What a pain that was.

It’s also worth reviewing that in addition to adding support for reusable citations, RM8 made a subtle but radical change in the way it managed the printing of citations in reports. In RM7, when you used the report option to combine duplicate endnotes, it combined them based on their text being the same. In RM8/9/10, when you use the report option to combine duplicate endnotes, it combines them based on the citations being the same citation that has been reused. As a practical matter, what this means is as follows. If you are an extreme source splitter who is converting from RM7 to RM8/9/10 and if you wish to combine duplicate endnotes in reports, then you are forced to run the Merge All Duplicate Citations tool. Otherwise, your endnotes will not combine. This tool worked for extremely split citations in RM8 and RM9, but it no longer works in RM10.

It’s worth reviewing why the change was made in RM10. It was made because RM8 and RM9 combined citations downloaded from Ancestry via TreeShare that actually were different but that did not differ in their text. Rather, they differed only in their media file or their Web tags. Therefore, such citations were merged even though they were really different. And that’s why the change was made to RM10.

In my personal opinion, the design of the change in RM10 is completely wrong. It still does not test for differences in media files or Web tags when merging duplicate citations, and instead it simply refuses to merge duplicate citations when the citation name is blank. That leaves all extreme source splitters in a lurch.

I therefore respectfully request that the recent change to RM10 be re-examined. The Merge All Duplicate Citations tool actually should merge citations with a blank citation name. And it actually should not merge citations that differ in a different media file name or different Web tag.

If I were converting to RM10 today, I suspect I actually would succeed. But I would have to do it the same way the Original Poster did. I would use SQLite to add some sort of dummy Citation Name to all my citations with blank citation names. Then I would run the Merge All Duplicate Citations tool, all would be well. But this solution is not available to most RM users, and it is not recommended by RM in any case.

3 Likes