So it would seem highly advisable to substitute RM10 Export-Import for Drag’n’Drop until there is an update that fixes this bug.
Not as dangerous as an Improvised Explosive Device, but RM10’s Drag’n’Drop IED is Insidious Explosive Data. With the release of RM10, Drag’n’Drop changed from a background GEDCOM export-import procedure to one that we understand to be more direct between the two databases. One might expect there to be unanticipated bugs in the new process but it’s only in the last few days that there is the beginning of some understanding that there is one that may be leaving a trail of erroneous data that is nearly invisible to the user with some exceptions that may be immediately apparent if you happen to be looking at an affected person. That would be the case if an IED drops on a person already in the target database. As more people are added to the database, their record number may trip over an ‘unexploded’ IED causing unexpected parentage to be added and marriage or other family-type events to ‘explode’ into selected views.
So far my testing has focussed on Drag’n’Drop of “Descendants and their spouses” with pedigree collapse in the line but, latterly, also “Descendants only” and was startled that it created so many more IEDs. I do not know if they are created by other Drag’n’Drop criteria.
Drag’n’Drop
IEDs
People
Families
Descendants only
7340
8595
3229
Descendants and their spouses
129
14083
5825
Those stats are for a transfer into an empty database so none of the IEDs have exploded on an existing person.
The good news is that RM10’s Tools > Database Tools > Remove Phantom Records clears out the minefield of unexploded IEDs. Unfortunately, it cannot detect injured parties and there may be no way to find them short of person-by-person inspection.
I have also tested RM10 GEDCOM Export-Import and that has not resulted in IEDs. And I imported the GEDCOM into RM7 and its drag’n’drop produced no IEDs (remember, it is a background GEDCOM export-import). So it would seem highly advisable to substitute RM10 Export-Import for Drag’n’Drop until there is an update that fixes this bug.
thanks Tom for Sharing – I was about to use this method and the timing was good so I did not.
These types of issues are concerning however, and may point to larger unknown hidden issues. I do understand that fixing one issue can introducing new issues (direct or indirect) but seems like lately been headed in the wrong direction.
Here’s an example of the effect of an IED having added ghost parents to a person among those transferred as a spouse of a descendant.
ChildID
CPID
FatherID
FPID
MotherID
MPID
FamilyID
44027
Y
1837
N
4625
N
14369
Child-44027 has a record in the database’s PersonTable (CPID=Y) and a name in the NameTable. A ghost family of ghost parents (FPID=N=MPID) has been created by drag’n’drop (an IED). This screen capture shows some subtle effects:
FamilyView shows no parents, prompting to +Add Father and +Add Mother.
Sidebar Summary>Parents tab states he has 1 set of parents, listed below with no name and a Marriage date.
The Marriage event for the ghost couple has apparently been added to the database along with links to the Place, Citations, Media if these were in the originating database. But we have no way of finding them through RM because they are associated with records that are not tied to a Person.
The Edit Person window has a Parents row that it would not if there were no parents, complete with the dropdown choice of switching to the ghost (father) or (mother). And the Relationships are shown as Birth and can be edited!
Some reports show the ghost couple’s Marriage event, obviously with no names, e.g. Ancestor Narrative, Pedigree Chart.
Good news: this was an example that the Clean Phantom Records tool resolved.
@rzamor1 , Now finding examples of lineage corruption caused by drag’n’drop that is not ghost-like and is not cleaned up by Clean Phantom Records. Referring to the examples I posted in RM10 Issue with copying.
Not cleaned.
Not cleaned.
Cleaned in the above case where drag’n’drop was into an empty database,
but
Not cleaned where DnD was into an existing database and there was an existing person with the same RIN as her mother, making her mother a male born after she was.
Cleaned in that empty database target
but
Not in the intended target where she got new, impossible parents from the existing database who are her age-peers, and a whole new lineage.
Cleaned in the empty database target. What the RM cleaner did was to replace the erroneous RIN for the non-person spouse with 0 and that results in the name “Unknown spouse” appearing where it normally would instead of blank. The couple events stay on the known spouse’s profile.
Not cleaned in the non-empty database target which already had a person with RIN=25218 so he got a new completely wrong ancestral lineage.
I don’t know how one can find the uncleaned corruption. Problem alerts may find some but not all and is masked by many that already exist. If the bug originated with the first release, there are potentially many large databases that have been affected.
I have not been able to spend much time at the computer for about a week and a half, so I’m not quite as update to date as I would like to be on this problem. I have some questions.
I tried the following experiment. I created a new and empty RM database. I dragged and dropped myself and all my descendants with spouses into the new database, There are now 19 people in the new database with no obvious corruption.Most everybody got a new RIN number. I repeated the drag and drop and told it that the Jerry Bryan in both databases was the same person. It did the drag and drop again with no obvious corruption. Everybody in the new database except for me is duplicated, so there are 37 people in the new database - 2 * 18 + 1.
I didn’t expect to see any corruption from such a simple test. So here is my question. Does there exist a database (or a collection of databases) which do not have any obvious corruption, and a specific set of drag and drops such that at the end there is corruption?
I think the answer is yes. But I confess I have gotten lost in all the details. It’s important to be sure that there was no corruption before the drag and drops and that there was corruption after the drag and drops. Otherwise, the drag and drops could be the victims of the corruption rather than the causes of the corruption.
I’ve no definitive, conclusive answer other than that my testing has been with @CherylC 's database as a source and another of hers as the target along with empty ones as targets with no merge. I’ve also tested another large one to an empty and it also exhibited what I’ve been reporting. Both sources pass integrity tests and export to GEDCOM so there is no indication of data corruption in lineage or db structure. I’m wondering if pedigree collapse is a contributing factor. There is in the small 3-gen and 5-gen descendants from the starting person that she reported on in November and I would not be surprised if that was not also the case in the other database because it is large and I started at an end-of-line ancestor.
When you’ve the opportunity, could you test a DND with a known marriage of cousins below the start person and inspect visually and with the SQLite query I posted?
So I did a drag and drop of Henry Peters Sr. and all his descendants plus spouses to a new and empty database. On its face, everything looks ok. I have not run your script yet because I’m being dense and can’t find it. Could you do me the favor of posting it again, right here in this thread?
Henry Peters Sr. would be in generation 6, just off the right of the screen. Two of his sons are in generation 5, Thomas and Henry Jr., and and I’m descended from both of them. And as you can see, I’m descended from Thomas twice.
John H. Peters and Hulda Asberine Cross were second cousins who married. Alva Edward Peters and Sallie Jane Cole were second cousins who married. If we need first cousins who married, I can get them as well because John England and Jane (Jennie) Peters were first cousins who married. But to test with them, I would need to do the drag and drop on the England family rather than on the Peters family.
If it gives any results, I would call them latent IEDs. Examining people whose RINs reported there are in the database and their parents should reveal the subtle cues I’ve described which will be removed by the Clear Phantom Records. They are latent in the sense that the non-present RINs reported that are higher than the highest RIN in the database may correspond to some person added in the future. That’s when they ‘explode’ causing erroneous spouses and lineage.
What the query cannot (or may not) report are IEDs that ‘exploded’ in the drop process. This gets hard to explain but I posit that the process is including family records with links to people in the source database that it should not (viz the results of the query) and some of those ‘foreign’ RINs could correspond to different people in the target. Therefore the explosion is immediate rather than latent.
Yes. Here’s a more definitive example involving these two databases:
OneFamilySource.rmtree: nuclear family, two children and their parents. OnePersonTarget.rmtree: one person with the same RIN as the mother in OneFamilySource. (the image shows two people in separate ‘trees’ because I repeated the process with a second person just to be sure)
the mother is now Someone Else-1. She inherited the kids’ father as her spouse, his Marriage event to Real Mother-1 and its Sources and Images along with the children and they have a whole different lineage.
There is no pedigree collapse - this is programming error. I’m quite certain this did not happen in RM7 but, I guess, that should be confirmed.
EDIT: I should add that this is an example where the IED blew up on contact in the drop and my SuspectChildParent.sql script detects no latent IEDs.
Thanks again for your hard work on this!
Not begs the question in my mind at least – is this the only scenario where this issue can be introduced. Have to wonder if any else had run Phantom tool and lost people and did not recognize.[ EDIT] I misunderstood the phantom issue
You may be misinterpreting my (poor?) explanation of what I think is going on. Clean Phantoms would have no effect on the corrupted nuclear family because it could not be run until after the IED ‘blew up’ and is no longer latent; that tool can only fix latent ones. The Clean Phantoms is not the problem; it does good things cleaning up a lot of garbage that RM leaves behind from normal procedures.
What was the IED that exploded? It was the FamilyTable record for the couple that was improperly processed.
Database
FatherID
MotherID
OneFamilySource
2
1
OnePersonTarget (after drag’n’drop)
3
1
What it should have been:
- without spouse
3
0
- with spouse
3
the new RIN for Real Mother who would be a new person in the database
Had MotherID not corresponded to someone else in the database, this would be a latent IED (phantom spouse/parent) that could explode later when a corresponding RIN person was created. That would not happen in this example because, even if RIN 1 had been deleted long ago, it would never be re-used. Newly added people always receive RINs greater than the maximum RIN in the database.
I’ve done more tests of drag’n’drop of various sets using RM10.0.3 and conclude that, apart from “Everyone in the same tree” (and probably "Everyone in the same database), any set involving Descendant calculations generates IEDs detected by my query and can therefore contain couple and lineage corruption or the potential for it as people are added. The list of drag’n’drop transfers to be avoided until the bug is fixed includes:
Descendants
Descendants and their spouses
Ancestors and their Descendants
Select from list
Descendants
Descendants and their spouse
Descendants and collateral lines (EXCEPTION - no IED’s but people count was just 1 less than “Everyone in same tree” for both 10-gen and 3-gen - that is unexpected; another bug?)
Ancestors and their Descendants
Ancestors and all Collateral lines (same EXCEPTION as for Descendants and collateral lines)
And it does not cause the corruption and latent potential corruption in a couple of my simpler test cases. It will take some time to be assured that drag’n’drop is safe to use under all conditions.
Because family relationships are at the core of any family tree, the trail from the problem addressed by this innocuous-sounding fix could be quite large, having affected many family trees that have had drag’n’drop operations done on them with RM10 since it was released 19 Jun 2024. What can be done to recover from the damage caused?
I sent the following (different order and italics mark added text) to the independent forum this morning but I’m not sure it is complete or even accurate and would welcome constructive suggestions:
If you have used drag’n’drop with descendants under RM10 prior to 10.0.4, there is a probability that the lineage of some people has been corrupted. Some persons may have different spouses and family-type events or lost their spouse. Their children will have half their lineage changed or lost both of their parents.
Recovery is only possible through inspection and editing or by restoring from the last backup before such a drag’n’drop operation was carried out and then repeating the steps since with 10.0.4.
Do not use Drag’n’drop involving the transfer of a person’s descendants in any version of RM10 prior to 10.0.4.
Run Clear Phantoms on your database immediately before any addition, import or transfer of people of any kind. This should need doing only once if the database is being used by RM10.0.4 but should have been done after every drag’n’drop involving descendants in prior RM10 versions.
I would like to see some such guidance coming from RM Support but user-to-user help is a start.
As for inspection and discovery, I can imagine that the RM Problem Search might help find some cases but not all:
Born before parents
Born after Father|Mother’s death
Father|Mother’s age
A search for children whose surname does not match their father’s surname would be useful. Maybe RM could turn around that quickly as an enhancement (@rzamor1 ?).
Another enhancement that would help would be in Compare Files if it could filter on matching UniqueIDs between, e.g., a suspect database and another that was the source of a drag’n’drop. The UniqueID is carried over for a person. The display would need to be family oriented for a given matched person. I’m trying to conceive something rudimentary with SQLite.
It’s hard to know how much residual damage there really is in the user community. It depends on how much use there has been in the user community with drag and drop since the error was introduced.
It’s about as easy to overreact to this problem as it it is to under react to it. I’m a sample size of 1, as I so many times say. But I very seldom use drag and drop. I work very heavily in my one and only database. If I make a test database, it usually starts out empty and I type a few dummy people into it or else the test database is a full copy of my complete database, And even if I drag and drop a few people into my new and empty test database, I will be deleting the database when I am done testing.
I think a thing that might be most useful would be for a new tool to be added to RM very quickly that identifies the kinds of structural errors that your error detector found. It could say simply no errors or found or errors found. And it could recommend either restoring from a previous backup or else running the Clean Phantoms tool if errors are found.
Also, the Clean Phantoms tool should be enhanced to clean up any and all referential integrity problems by deleting them. There is really no way to fix them automatically. About the only way to move forward from referential integrity problems other than deleting them is to restore from a backup from before the error occurred. By referential integrity problems, I mean things like somebody having a mother or father or child or spouse whose RIN is no longer in the database.
Agreed. Especially without tools for that purpose.
Clean Phantoms already does and ‘corrects’ them by resetting the phantom FatherID|MotherID to 0. But that does nothing to find or correct an erroneously replaced or removed parent caused by the defective drag’n’drop.
Yes. My assumption is that it began with 10.0.0 but there is a parallel in Set Relationships & Kinship List omit descendants of person with unknown spouse! 10.0.1.0 that led to a fix in 10.0.2 in September along with * Fixed: Drag and dropping a parent to a single parent on the family view didn’t link the parent. There’s no mention of any change to drag’n’drop in 10.0.3 so the problem must have been introduced no later than September and no earlier than July last year, 5-7 mos ago.
I’ve downloaded and installed v10.0.4.0 and performed the same drag & drop with the same files that helped identify this issue in the first place. Tom’s query didn’t find any errors, nor did I find any via checking for duplicates, which is how I spotted the problem initially. I’m hoping that other users make as little use of drag and drop as I generally do, but I’m afraid that there may be a significant amount of database corruption out there and the only users who are likely to know about it are the ones who read this forum.
