When I receive files from others I do cleanup/standardization in order to make publishing of narratives more practical and to optimize electronic matching.
Some of the files I receive have items in Person Notes that should be facts or sources. Some of them are straight cut and pastes from various public records searches. Often these are garbled visually due to some special characters, and this results in exceptions that are usually recoverable.
What I end up doing if I can recover from the exception I cut the entire notes and paste into a plain text TextEdit (Mac) file, the copy the respective text I wish to keep and paste back into the requisite facts or notes. In some instances I need to remove this notes via SQL to keep RM from crashing. Often these are Obits which I’ve put into their own fact so I can include/exclude as needed.
What’s more annoying is when things display normally so I (mistakenly) assume I can edit/cut/paste in the Person Notes field. I’ll have something like (parts intentionally obscured)
Past Address
2002-2010
33*** **** Rd,
Burlington, NC 27215
that I want to change to:
Past Address
2002-2010 33*** **** Rd, Burlington, NC 27215
so I can cut/paste into a residence fact in one pass.
Unfortunately placing the cursor in front of 33** and deleting the EOL will often cause a division by zero and RM will crash. My suspicion is RM mishandles line endings when moving between platforms. Line endings on Mac/Linux are a LF (line feed) when on Windows they CRLF (carriage return/line feed). If someone sends you something that originated on windows in the notes field, and you edit it on the Mac in the notes field, the control used for that field gets confused and aborts due to differing character counts for line endings.
Long and short is special characters aren’t handled correctly in the Notes field, and this can lead to RM crashes or incorrect displays both when displaying or publishing a narrative.
I wonder if pressing Shift-Command-T within TextEdit to convert the current file (being pasted into and copied from) to plain text format …will result in properly stripping Windows-style formatting such as CRLF down to just LF for the subsequent copy/pasting of text? That, along with using RootsMagic’s Note feature for Paste as plain text (Cmd+Shift+V) might lessen such occurrences.
That’s what I’m doing to clean up the text normally, unfortunately in cases like the picture I have to do the cut and paste via SQL. In most cases the only way I can delete the line endings is to cut and paste the entire notes text into TextEdit and edit in there. As you can see from this Hex listing from SQLiteStudio there are lots of CRLF in the notes:
I work on a PC instead of a Mac, but I suspect the issues are similar. I pretty much never paste from the Web or from anywhere else directly into notes in RM. Instead, I paste into Notepad, then copy out of Notepad and paste into RM. And when I paste into RM I use Paste as Plain Text. That generally cleans up most of the problems. I have never seen the “divide by zero” problem.
Of course, you are copying out of notes in RM that already have the garbling so you can paste elsewhere in RM. You said you are using TextEdit on the Mac. I’m not a Mac user, so I assume TextEdit is similar to Notepad. Does pasting in and out of TextEdit not clean up things for you sufficiently?
I think the CR/LF vs LF issue is a separate issue than other special character and rich text vs. plain text formatting. I can’t even begin to understand how RM is supposed to handle this issue on a Mac. As I understand it, Mac uses the UNIX file system so that native Mac text files use the LF convention. I have worked on UNIX machines even though I have never worked on a Mac, so I do understand the tension between CR/LF vs. LF. But the same RM database is supposed to work on both a PC and a Mac and RM on a PC uses the CR/LF convention. Since I don’t use RM on a Mac, I think somebody from RM is going to need to speak to that issue. How does RM on a Mac manage to use the CR/LF convention when a Mac itself uses the LF convention? I don’t know.
TextEdit does clean things up for ME sufficiently. The issue at hand is I receive files from others either in RM, FTM, or GEDCOM formats where people may not have been as judicious as I am about pasting things into notepad or textedit first, or simply send me a file that originated on Windows.
My normal process is to put these into a separate RM file, standardize names and places, cleanup as many notes as practical then drag and drop them into my main database and merge the dups to hook everything up.
The issue as hand is the cleanup process is complicated by RM’s incomplete handling of special characters and line endings.
Are you saying “paste via SQL” (within SQLiteStudio) gets you past the problem (in and of itself) OR are you “massaging” the hex code (to edit the 0d 0a sequences) and then pasting the result back (within SQLiteStudio) to restore proper note formatting?
What I’m doing in those cases is I’m finding the record via SQL, cutting the entire Notes field out and making it null, updating, then pasting the text into TextEdit for cleanup. This is the only way I’ve found to deal the situation when RM won’t allow me to open the Notes field for editing. Some of the text may go back into the Notes field, or into Residence, Occupation, Education, or Obituary facts.
I’m still not sure I understand the issue. Are you needing the 0a to become 0d0a? Or are you needing the 0d0a to become 0a? Or is it something different?
Again, if RM does’t use the 0d0a in all cases - PC and Mac - then I don’t understand how RM can run the same database on both a PC and a Mac. And some users have reported in this forum that they do precisely that.
I’m unsure I understand what RM functionality you believe is not working as designed, then? The “visual presentation” of such afflicted text streams is understandably convoluted. It also sounds like there’s no errors in the initial note creation phase of Paste and Save. So, presumably it’s the subsequent desire to edit/reformat the text previously saved.
That sounds like an easy Tech Support ticket. Here’s a sample database with a set of “problem” notes, that generate errors or crash RM when revisited for editing.
I don’t care if it uses CR, CRLF, or LF. What I want is for RM not to crash when I’m editing the notes field when special characters are present in the text. Not a big ask.
RM needs to filter characters it can’t deal with on import and paste and manage text within the fields in a consistent manner.
It’s a good ask. You need to submit a trouble ticket with a database that has the problem and a procedure you can go through to recreate the problem.
RM isn’t handling special characters correctly, simple as that. It shouldn’t crash or display text improperly no matter where it comes from. I’m speculating that it has to do with line endings because that’s where the most severe and unrecoverable crashes are coming from.
I’m working around the problem because with over 50 years working on computers I can, but I don’t think I should have to nor should anyone else.
Before I submit a ticket I need to copy the database to my Windows machine and see if the same issue is there. It’s always best to be as specific as possible since RM seems to close the ticket if they don’t find the issue right away.
Agreed and a good step to confirm that it’s a cross platform issue. As I recall the Note Editor is licensed from a third party. That’s not intended as an excuse, more a potential complication for all.
Sorry BUT I don’t let them close the ticket— if they say they can’t replicate the problem I’ve told them to try again – when they didn’t get back with me, I sent them a message…
RM should not accept any (special or otherwise) characters that might cause issues with editing or crashing.
RM should clean up any characters that might cause issues on import etc.
The challenge might be that there might be more scenarios that one might thing to cover them (mostly all) – I recall some educational platforms I worked with had similar challenges
I finally got my DB moved to my WIndows machine. Unfortunately this issue corresponded with my reorganization of several thousand pictures I also moved to OneDrive so my DB was behind all of them. Ouch.
The bad/good news is the text is displayed bad on windows too, but it doesn’t error out.
Now I need to organize things and get the ticket in.
I reported an issue awhile back (end of summer just after Rm11 came out)
I can not say for certain what the root cause was but I can say for 99% certainty that that the Media Tool caused the problems with both duplicated media and unlinked media. The tech person told me that it should be impossible – yet I repeated the same results 5 times using slight different scenarios. I suspected to be fair to RM – that there was gremlin hiding the contributed to this occurring.
The brief synopsis was that I reorganized my media into subfolders where all 17K media was previously just in the one main RM media folder. The database was NOT in the cloud. After I completed moving the media - I ran the media tool – that cause two issues – one was ~600 media was duplicate with one of them not having any links. (The MediaFile was SameName but had two diff MediaIDs – the orig mediaID plus a new duplicated ID – one set has the correct media links and the other set had NO media links. Yes I did have backup – but they were not much help other than confirm I could repeat the results with the media tool. The files were already moved on the computer. It was possible the database conversion was to blame, something with 11.0.1 or a gremlin, I would have to confirm but as I recall it ver 11.0.1 before 11.0.2 came out. I dont think a gremlin would cause issues with 100’s of media files across all the subfolders - only that the tables did not have that issue before I ran the tool as I confirmed from restoring from backup. (The links would be broken however just not duplicate and media with no links)
I was able to clean most of the two problems using Sqlite that would have take me a lot longer manually by hand. Not sure if you ran the MediaFixTool – but if it was after you may want to mention that it could be related.
