Complete Success in Getting RM8 Endnote Lists to Match RM7 Endnote Lists in Reports

RootsMagic
1

I have posted about this before, but I thought a new thread would work better than adding to the old threads.

From the beginning of running narrative reports in RM8, I found that an endnote list for a report created by RM8 was much longer than the same endnote list for the same report created by RM7. If you compared the two endnote lists carefully, the symptom appeared to be that RM8 was not honoring the Reuse Endnote Numbers option in a report. This has to do with assigning the same endnote number and thereby reusing the endnote number when two different citations are the same. What was happening was that the same endnote was getting a bunch of different endnote numbers and therefore was being included in the endnote list a bunch of different times. That is not a good thing.

Well, RM8 is honoring the Reuse Endnote Numbers correctly. But RM8ā€™s definition of when two citations are the same and RM7ā€™s definition of when two citations are the same are not equivalent. RM8ā€™s definition of when two citations are the same is when they are the same entry in the CitationTable. RMā€™s definition of when two citations are the same is when they produce the same endnote sentence.

This distinction is tied in with the new facility in RM8 to be able to reuse citations. If you do a Paste/Copy, RM8 makes a new entry in the CitationTable. The new entry has the same data as the entry you Memorized, but it is a new entry. If you do a Paste/Reuse, RM8 does not make a new entry in the CitationTable. Instead, your Paste/Reuse is just a re-use of the existing entry in the CitationTable. The huge advantage of the RM8 way of doing it over the RM7 way of doing it is that because with Paste/Reuse there really is only one citation, you can make corrections just to that one citation and the corrections will take effect in every place that citation is reused.

So far, so good. But who knew at the beginning of using RM8 that this excellent new feature would have such a huge effect on endnote lists in reports. I certainly didnā€™t. Because RM8 is basing the Reuse Endnote Numbers feature on the entries in the CitationTable rather than on the endnote sentence itself, there are two important considerations for users.

One consideration is you that must run the Sources => Three Dots => Merge All Duplicate Citations tool immediately after import from RM7. I havenā€™t tested yet, but I suspect that also means that you must run the same tool after importing data from GEDCOM or ancestry or FamilySearch or any such place.

The other consideration is that with one exception, you must always use Paste/Reuse rather than Paste/Copy when memorizing and pasting citations. The one exception would be if you intend to paste a citation and then edit it to make it into a different citation. This makes me wonder if the names of the Paste/Copy and Paste/Reuse operations should be changed. Maybe Paste/Reuse should just be Paste, and maybe Paste/Copy should be something like Create New Citation from Memorized Citation. I know the second name is much too long, but itā€™s hard to get the concept of whatā€™s going on into just a couple of words. Whatever itā€™s called, I also think that it would be important that Paste/Copy add the string ā€œ(Copy)ā€ without the quotes to the citation name of the newly pasted citation. It does not do so at the present time.

This brings us back to me, and to my own problems with long endnote lists in RM8. I thought I was doing all the right things to get my citations in RM8 merged and reused properly, but I was still getting extra entries in my endnote lists, lots of them. They were all caused by things I needed to clean up in my RM7 database before I imported my RM7 database into RM8.

  1. I had what appeared to be identical citations in RM7 that wouldnā€™t merge in RM8 because of miniscule differences in the Research Note field. The difference could be as minor as an extra blank somewhere that you couldnā€™t even see on the screen.
  2. I had what appeared to be identical citations in RM7 that wouldnā€™t merge in RM8 because of miniscule differences in the Detail Comment field. The difference could be as minor as an extra blank somewhere that you couldnā€™t even see on the screen.
  3. I had what appeared to be identical citations in RM7 that wouldnā€™t merge in RM8 because in RM7 I had changed some user defined templates to move data from the Citation to the Source. This actually copied the data from from the Citation in RM7ā€™s database tables to the Source in RM7ā€™s database tables without deleting it from the Citation in RM7ā€™s database tables. The left over data was not visible in the RM7 user interface, but it was still there. Sources I made after changing the Source template would not have the left over data. But the invisible leftover data was being imported into RM8, making what appeared to be identical citations in RM7 be different in RM8.

I suspect that many users will have the extra long endnote lists in RM8 if they forget to Merge All Duplicate Citations after importing data into RM8.

I suspect that many users will have the extra long endnote lists in RM8 if they use Paste/Copy without realizing why itā€™s important to use Paste/Reuse most of the time. By the way, If you have been using Paste/Copy instead of Paste/Reuse, not to worry. You can fix your citations by using the Merge All Duplicate Citations tool at any time.

I suspect that more users than you might think will have the extra long endnote lists in RM8 because of Research Notes field not matching between otherwise identical citations in RM7. As a result, the Merge All Duplicate Citations tool will not fix the problem. I fixed all such problems on the RM7 side of the house by using SQLite. I think most users would be better served by fixing such problems on the RM8 side of the house after importing from RM7 by using RM8ā€™s manual Merge Citations tool on a case by case basis. This merge will ignore the trivial differences that prevent the Merge All Duplicate Citations tool from working.

I suspect that more users than you might think will have the extra long endnote lists in RM8 because of Detail Comments field not matching between otherwise identical citations in RM7. As a result, the Merge All Duplicate Citations tool will not fix the problem. I fixed all such problems on the RM7 side of the house by using SQLite. I think most users would be better served by fixing such problems on the RM8 side of the house after importing from RM7 by using the manual Merge Citations tool on a case by case basis. This merge will ignore the trivial differences that prevent the Merge All Duplicate Citations tool from working.

I suspect that virtually no users will be bitten by my problem of citations not merging because their source templates have been changed to move data from source to citation or vice versa. If you donā€™t define your own source templates, donā€™t worry about this potential hazard. And even if you do define your own source templates, donā€™t worry about this potential hazard unless you move data from source to citation or vice versa.

I suspect that there are few other differences that I have not yet identified that may cause extra long endnote lists in RM8. For example, I donā€™t use repositories, but I suspect that citations using repositories wonā€™t merge automatically if they are using otherwise identical repositories that havenā€™t been merged yet. And I havenā€™t tested what happens if otherwise identical citations have different media attached in RM7. All my media was 100% consistent between otherwise identical citations in RM7.

I know this message sounds long and complicated and potentially a little scary. But for most users most of the time, you only have to remember two things. First, merge your duplicate citations after you import your data from RM7. If you imported your data from RM7 a long time ago, itā€™s not too late. Run the merge now. Second, use Paste/Reuse rather than Paste/Copy unless there is a compelling reason to use Paste/Copy. And if you have been using Paste/Copy, not to worry. Just go ahead and merge your duplicate citations now and user Paste/Reuse in the future.

4 Likes
2

Gosh youā€™ve done a lot of meticulous work in this area Jerry, thanks for sharing it all with us so we too can learn. Thank you!

Iā€™m also wondering if I can ask about this statementā€¦ā€¦ā€¦ā€¦ā€¦.

Well, thereā€™s one in every crowd Jerry and thatā€™s me!

Unfortunately, I have designed my own templates and moved fields from the source to the citation blocks after using them (many times). I may have also moved fields from the citation block to the source block in the past, but canā€™t be sure (I guess this is a lesson that I should document major changes such as this that I make to the database!)

Iā€™m still using RM7 as my main database and creating a new copy in RM8 virtually every day. This morning Iā€™ve created a new RM8 database, and using one citation where I know that the underlying template was changed, have memorised that citation and paste-copied to another fact. I did not change this fact at all so it should have been identical. I then memorised the original citation again and paste-reused the citation. I then ran the ā€œMerge all duplicate citationsā€ function and the citation merged as expected, with the result that there are now 3 instances where the citation has been used.

Sample size population of one and a very narrow review in just one part of the system as Iā€™m a bit rushed for time at the moment, but do you know of any issues that will crop up as a result of moving fields in custom templates from source to citation, or vice-versa?

Much appreciate any advice that you might have - thank you!

1 Like
3

Iā€™m going to post some technical details of exactly how I cleaned up my RM7 database so my citations could be merged automatically in RM8, but Iā€™m going to post them in a more technical forum. Suffice it to say, some SQLite was involved.

But the citation merges can be done without cleaning up first in RM7 and without using any SQLite in RM7. You simply do manual merges in RM8 after you do all the automatic merges in RM8. It can be a little tedious, although perhaps not too much so. And you have to do it after you are completely committed to RM8 and have done your last import from RM7 to RM8. Otherwise, your citation merging in RM8 will have to be done over again.

It never occurred to me that anything I was doing with citations in RM7 was ever going to cause me a problem in the future. Thatā€™s because RM7 identified duplicate endnotes by looking at the sentence themselves rather than looking at the database structure. I have just submitted a request for the same feature in RM8, but Iā€™m preparing as if such a feature will never appear. Indeed, Iā€™m basically done preparing now. After I make my final import from RM7 into RM8, all I have to do is merge all duplicate citations in RM8 and my citations in RM8 will be ready to go.

Believe it or not, an RM8 user who never changes any templates and who only uses RMā€™s built-in templates could encounter the same problem of Reuse Endnote Numbers not working for what appear to be duplicate citations. There are a number of things a user could do in either RM7 or RM8 that could create trivial differences in otherwise duplicate citations. So you donā€™t have to be doing things as extreme as making your own templates and then moving fields between Source and Citation. I think itā€™s pretty easy to get into this situation by total accident.

4 Likes
4

I just tried your recommendation to ā€œMerge All Duplicate Citationsā€ on one of the files, prepared a narrative report of a portion of it using RM8, and compared the RM8 generated report to the RM7 generated report. In both reports, ā€œReuse endnote numbersā€ was invoked. IT WORKED! RM7 and RM8 each produced reports with exactly 237 citations, with the same citation numbers used in the exact same places. RM8 generated a 62 page report (before it was well over 100 pages). That compares more favorably to the 50 page report generated by RM7. THANK YOU FOR YOUR EFFORTS!

There are still other issues that need to be solved that I shared with the other users and with you elsewhere.

1 Like
5

Jerry - thatā€™s good work, thanks. As you may have noticed in another chain in the Community I have recently learned more about ā€œproperā€ citation construction from Evidence Explained. In order to try to implement some of that education I have THOUSANDS of sources and citations in multiple databases which need to be cleaned up.

Obviously, merging duplicates before doing the heavy lifting is the right thing to do, taking advantage of this new RM8 capability. But BOY am I nervous about setting RM8 loose to ripple all through my database and cleanup all those things! Again obviously, I need to do it on a copy just in case I need to revert to the original, and so I can always go back and check in case something looks odd in the future.

But the hardest part will be detecting if the merge tools have messed anything up. My 2 main genealogy databases contain #1 - 60k+ people, 102k+ events, 1k+ sources, and 169k+ citations, and #2 - 20k+ people, 77k+ events, 50k+ sources and 56k+ citations. After merges it should be much smaller of course but that is still a formidable list to verify.

Is there any documentation I should read anywhere on Merge Sources, Merge all Duplicate Sources and Merge all Duplicate Citations? They do not seem to be in the Wiki (yet). I need a set of doā€™s and dontā€™s, a list of situations to look out for, an order in which to apply the tools. Iā€™m quite comfortable with SQLite if there are scripts that can help me KNOW that RM8 got it right.

6

Avoid ā€œMerge all duplicate citationsā€ unless youā€™ve ensured that there are textual differences in the citation name or other citation detail fields for those having different media or webtags. See

7

Tom - I was thinking Merge dup citations was the first thing I needed to do because I imported my database from RM7 in which I used lots of Paste citation, and because since using RM8 I have always Paste Copy citations, not any Reuse Citation. Agree?

8

I neither agree nor disagree as I do not know what your citations look like. However, this is a new feature that has not been completely thought through in its development and we have certainly seen the pitfalls it has with a number of Ancestry Sources downloaded through TreeShare. Maybe your practice provides the necessary differentiation in the text fields between citations that have different media or webtags. Itā€™s a powerful and dangerous function for which the only undo is restoring from a prior backup so donā€™t go developing your database after merging citations without first doing a very thorough check that nothing untoward has happened to it.

9

I may be in a simple enough configuration that the potential problems you see are nothing I need worry about:
1 - In 25 years I have only ever used FreeForm, never any of the RM templates
2 - I do not download people/sources from Ancestry at all, and from FamilySearch only minimally

With 225k citations spread across 2 databases, all done in the same ā€œstyleā€, any plans to do a ā€œvery thorough checkā€ are surely wishful thinking! But merging duplicate citations surely reduces the effort and is the right place to start.