Organizing Media and Data

In RM, yes, the path to a piece of media is the full directory, starting at the highest parent.

Example: The media path in RM for the 1870 Census page for Constantina, etc (as seen in the screen shots) is …
C:\Google_Drive_Mirror\Genealogy\sources\Census - US Federal\1870 Census\US Census 1870 Georgia Chattooga County Dist 870 Trion Factory Post Office Pg 60.jpg

Note that Google Drive sync creates a local PC folder. In my particular install, that folder is named ‘Google_Drive_Mirror’. Other cloud storage providers may operate differently.

Just the ol’ grey matter wetware.

Basic sequence is:

  1. Download some image item. Example: the census page when attaching it to my tree on Ancestry.
  2. Go to the download directory and, before my old brain forgets to do it, …
    a rename the file to align with my convention
    b apply the metadata
  3. Move the file into the appropriate source folder/sub-folder

Once done there isn’t too much revising of metadata that needs doing. It’s too infrequent and ad hoc to have any programmatic tools needed.

I was thinking about those who might want to apply your metadata concept retroactively to their media, regardless of the folder and naming convention. I did something years ago between RM and metadata to help manage media with Picasa:
https://sqlitetoolsforrootsmagic.com/media-metadata-read-write-compare-with-picasa/

AH! Yes, that might be useful to others.

Suggestion … we shouldn’t hijack this thread, since it’s a tangent to Robert’s question. Perhaps you’d like to make a new thread where the discussion can continue?

A consideration: If you organize media by lines, will media for women reside under their maiden surname or married surname? If married, will media for pre-marriage reside in one area and another for married? What if they marry multiple times?

This is why I try to avoid organizing by human names as much as I can. It gets messy.

Probably should but I do not wish to bite off more than I can chew. I did that work more than a decade ago and my grey cells have gotten a lot greyer!
@rzamor1 can you move this fork to a new topic starting with Kimberley’s:
https://community.rootsmagic.com/t/renaming-media-files-in-rm9/6992/8

I don’t mean organizing media by lines, but working the data by lines.

@rzamor1 Thanks for spawning a new thread so we could continue the discussion :slight_smile:

@TomH I think your concept has a lot of merit. It would be great if RM had a function for it embedded. But even if not, an SQL tool would be useful for those who wanted it.

You could pick up from RM the names that the media is linked to, then somehow push those into the file properties area through OS calls. If fact, while you’re at it, you could even push the source information into the ‘Comments’ field, etc.

That, too, would make it much easier to look at the unorganized RM media directory of a TreeSync and figure out what is what.

My liaison between RM and image file metadata has always been via sqlite generating batch files to run ExifTools on the image files to deliver metadata or retrieve them in a text file for import to sqlite. It’s a crude, cumbersome procedure. Is there an application such as Excel, PowerShell… that can ODBC to the sqlite database and read-write metadata on image files?

Unfortunately, programming is not in my wheelhouse. So, I asked ChatGPT :yum:. Here’s what it said:


Can’t vouch for ChatGPT’s accuracy, but on the face of it using PowerShell sounds like an option.

Ha! …minds think alike but I posed the question differently and got an answer at a higher or more general level:

Is there an application that can connect to a sqlite database and read-write metadata on image files?

Yes, there are several applications that can connect to a SQLite database and read-write metadata on image files. Here are a few examples:

  1. ExifTool - This is a powerful command-line tool that can read and write metadata in a variety of file formats, including JPEG, TIFF, PNG, and many others. It can also read and write metadata stored in a SQLite database.
  2. PyExifTool - This is a Python wrapper for ExifTool that allows you to easily read and write metadata using Python. It also supports SQLite databases.
  3. Digikam - This is a free and open-source photo management application that supports a variety of image formats, including JPEG, PNG, and TIFF. It also has built-in support for SQLite databases and can read and write metadata.
  4. Lightroom - Adobe Lightroom is a popular photo editing and management application that supports a wide range of image formats, including JPEG, TIFF, and PNG. It also has built-in support for SQLite databases and can read and write metadata.

These are just a few examples of applications that can connect to a SQLite database and read-write metadata on image files. There are many other applications and tools available, depending on your specific needs and preferences.

With Digikam, would it be feasible to exchange data between its database and a sqlite family tree database that Digikam would write to the image files and retrieve for comparison.

Yes, it is feasible to exchange data between Digikam’s database and another SQLite database, such as a family tree database. Digikam has built-in support for SQLite databases and can read and write metadata to image files.

To exchange data between the two databases, you could use SQL commands to query and update data in both databases. For example, you could write a script that reads data from the family tree database, uses that data to update the metadata on the corresponding image files in Digikam’s database, and then writes any updated metadata back to the family tree database.

You could also use Digikam’s API to automate the process of updating metadata. Digikam provides a powerful API that allows you to access and manipulate metadata programmatically. With the API, you could write a script or application that reads data from the family tree database, uses that data to update the metadata on the corresponding image files in Digikam’s database, and then retrieves any updated metadata for further processing or comparison.

Overall, while there may be some challenges involved in exchanging data between two different databases, it is certainly feasible to do so using Digikam’s built-in support for SQLite databases and its API.

[----------------------------------------------------

That sounds promising and Digikam looks to be a worthy successor to Picasa, provided its sqlite database is not encrypted…
https://www.digikam.org/

P.S. I’m not in the programmer’s wheelhouse either!

Fair point. A DBA / data integration person isn’t exactly a programmer. But they’re about 1000 steps closer to that technical skills realm than me, lol.

If you decide to tinker with craft up a solution to your concept, I’d certainly be interested.

I tinkered with digiKam and had some success.


That’s the result of a query to attach the digiKam database to a RM database, generate a list of people to whom each media item has been ‘tagged’ in RM, translate that to new records in the dK Tags table and links from there to the dK Images table. Then when dK is reopened, have it write the tags from its database to the image files.

Some of what I learned is that the Subject field you use is a Windows-specific field named XPSubject that is poorly supported in digiKam and elsewhere. Also that the digiKam database really only supports one Caption (or Comment) field per media file and tags (or keywords), so I had but one choice where the data should go. And it all gets very confusing among EXIF, IPTC and XMP metadata specs. IPTC tags are limited to 64 bytes, which could be a limit of less than 64 characters if any 2-byte Unicode ones are used. XMP tags are unlimited and it seems that dK does its best to place the data from its Tags records into a field of the appropriate standard.

I’m unsure whether the data should be concatenated to one tag per media file or left as, say, a tag per person so that a file gets multiple tags. I don’t think Windows or digiKam care - they do the concatenation of multiple tags (there are a couple of examples above where I had entered tags manually in dK).

OTOH, just now having uploaded a file to Google Photos and to GDrive, I find that the tags are neither displayed nor searchable but the Caption is. So rather than transmitting the RM Caption to the dK Caption as I thought I might, maybe it is better to send the concatenated string of data that I had put into Tags. It would actually be easier!

I have a Mac and use a ‘type’ folder system like Kimberly.

The citation for each item goes in the “get info” area since those can get crazy long. So far, I have been using item name to include basic what and who. I like the PC function of being able to add subject names like that, but unless you add them after the citation (which sometimes it may not all fit) the only other option is adding them as a tag. But I tried tags before on Mac and just didn’t like it. The ‘get info’ box char limit needs increasing.

@TomH Lol, I just knew you’d get a wild hair and want to tinker. It’s a fun little challenge.

Personally, I chose to use the ‘Subject’ field for names because that’s the subject of the image. I left ‘Tags’ for future use. For example, if I wanted to tag a picture with a place name or category (ie census, death cert, etc).

Your findings about the Subject field for Windows, and the MacOS information from @epona548 got me curious. So I fired up my Linux Mint virtual machine and inspected how Linux might do things.

I copied across a Windows file that had some ‘Subject’ information. I discovered that the default ‘Files’ app doesn’t reveal any metadata at all. In fact there’s not even a way to add any. But one of the image viewing apps, Pix, does show it in read-only.

Seems like there’s no universal tagging solution across operating systems.
¯\(ツ)

(I know RM doesn’t run on Linux. Just adding that insight because I found it interesting.)

Long term the management of Family Records will not be up to me so no fancy programming, no total reliance on Software by itself.

RM will be on a USB memory stick but all media and records will be within folders and simply catalogued in a Index file with this file in various formats to make it as future proof as possible, a printed version will be in the Box with family albums and memorabilia and treasures

Ah, I see you’re a neat & tidy organizer too. I use folder # prefixes for sorting in some of my other data areas too, but haven’t for the genealogy stuff.

As for future-proofing, I don’t worry too much about the electronic media. Being in IT, I know that current operating systems, file formats, hardware, etc all get more difficult to use / maintain / support over time. By the time 2 generations have past, nobody will likely be able to access those media files anyway. So the good ol printed word is the best bet for long term storage.

Side note: I don’t rely on the metadata for long-term efforts. It’s just a technique for helping me locate and inspect without having to open up files.

One of the more fascinating long-term storage projects I’ve read about is The Rosetta Project, part of The Long Now Foundation’s efforts. They’re using microprinting to create a multi-lingual reference. Their first prototype stores over 14,000 pages on a 3-inch nickle disk. The project’s intent is to make a language cross reference, similar to the original Rosetta Stone, that will last for thousands of years. https://longnow.org/ ; Concept - The Rosetta Project

I’ve given thought to how that could be adapted to a long-term storage strategy for a printed genealogy. Imagine having tree diagrams, reports, media like census page, etc all microprinted on a single small plate. No computer obsolescence to worry about. You could even make many duplicates and send them all to your family in frames to display like a picture.

I see your arguments and to an extent I agree, the box I have has printed documents which are more life stories of each branch of the family, and specific individuals.

I do not have concern regarding electronic media, Word has been around since the mid Eighties, PDF’s shortly after, Txt since the days of DOS and files that I created 35 years ago are still able to be opened.

The various MPEG committees that have been around since the fiasco days of Betamax and VHS and their activities have set standards with the key element being “backward compatibility” and JPG images has been around reliably for over 30 years.

I am more concerned about the Software we use for our family tree still being available in 40 or 50 years time hence JPG files of the family tree, Gedcom’s etc.

Keeping options open is imo the way to go to try to cover all bases and not to rely just on software like RM,

It’s essentially the same technical challenge that the National Archives is facing too. Chasing Technology | National Archives

This comment is just for topical discussion.

The folder structure is the file name for the computer. A folder structure can be applied as a filename replacing the slash or back-slash with a hyphen or other legal punctuation, eliminating the need for folders. Or the folder structure could be attached as tags with everything in the root. File names are permitted to be very long.

If the folder structure is:
Gen-docs/document-type/year/location-state/location-town/surname,
the same could be sorted by searching with a filename:
Gen-docs,document-type,year,location-state,location-town,surname.txt.

The folder construct makes it easier to read in File Explorer (Win) like it was a presorted long filename.

I won’t use giant filenames, and I won’t use sub-folders out to the smallest detail. I like about 50 well-named files in a folder to keep down the number of folder hoops I have to jump through. I may have hundreds of well-named images in a folder: “Lname, fname-ISODate-Location-Event-Notes.Type”. They can be computer sorted or visually read quickly.