Organizing Media and Data

[KimberlyGreen]

@RobertL

Or is there an alternative to my naming convention that would require less work?

@RobertL

easier to locate.

If you’re interested, here’s how I organize my media. Since one of your goals is to make people easier to locate, my method might give you some ideas.

First, I store media by macroscopic categories. Within those categories, there may be sub-categories. For example, in the Census folders, there are further subs for year. See screen cap:

For the media themselves, I name them by what the source is. Using Census as an example again, the typical filename goes from largest to smallest component.

Now, here’s where the people come in. You’ll notice they aren’t part of the filename. Instead, they are added as properties of the file … metadata. Specifically, they are added as names in the ‘Subject’ field, separated by a semicolon. I can add as many as I need to, even if there are multiple households on a page that are relevant.

I’ve added the Subject column in Windows Explorer so that I can see them at a glance.

But I can also use the Explorer search feature to find them, even if the file is in a sub-directory at a lower level. Side note: In the Search field, the search parameter is formed by entering in the field name and text value, separated by a colon.

For women I also add their maiden name in (), so that a search for their family surname would return them as well.

You’ll see in the screen cap below that I did a search for ‘McCain’ across my entire collection of Federal Census images.

Caveats & considerations:

• Moving a file between two Windows computers will retain the property values. Example: If your hard drive crashes, but you have a backup, you can restore the files and still have the property values.
• I don’t know if the property values are retained if you move a file between a Windows and Mac computer. I have no experience with Apple devices.
• If you use a cloud storage service, you probably won’t be able to see the property values in their web-based interface. Example: I use Google Drive sync and cannot view the property values through their web interface. But the values are still there and visible via Windows Explorer.
• This method works for media files such as images and videos, as well as MS Word .doc files. It doesn’t work for filetypes .txt or .PDF.

[TomH]

Fascinating. Do you use any programmatic tool to name the files and add and revise the metadata?

[RobertL]

TomH: I think I’ll pass on renumbering the media. I can just start with the new people.

KimberlyGreene: I assume you have the path set the high level directory and you place the media in the appropriate sub-directory, correct”. I use a Mac so I’ll have to study your method some more, proving it to a mac system.

Thanks to all who responded. I have a lot to clean up before I tackle new items; and I have to get treeshare to work (tech support is working on that).

[RobertL]

What I may end up doing is , after I get my mass cleanup done, I may start reorganizing by my Lines and go from there.

[KimberlyGreen]

@RobertL

I assume you have the path set the high level directory

In RM, yes, the path to a piece of media is the full directory, starting at the highest parent.

Example: The media path in RM for the 1870 Census page for Constantina, etc (as seen in the screen shots) is …
C:\Google_Drive_Mirror\Genealogy\sources\Census - US Federal\1870 Census\US Census 1870 Georgia Chattooga County Dist 870 Trion Factory Post Office Pg 60.jpg

Note that Google Drive sync creates a local PC folder. In my particular install, that folder is named ‘Google_Drive_Mirror’. Other cloud storage providers may operate differently.

[KimberlyGreen]

@TomH

Do you use any programmatic tool to name the files and add and revise the metadata?

Just the ol’ grey matter wetware.

Basic sequence is:

1. Download some image item. Example: the census page when attaching it to my tree on Ancestry.
2. Go to the download directory and, before my old brain forgets to do it, …
   a rename the file to align with my convention
   b apply the metadata
3. Move the file into the appropriate source folder/sub-folder

Once done there isn’t too much revising of metadata that needs doing. It’s too infrequent and ad hoc to have any programmatic tools needed.

[TomH]

@KimberlyGreen

It’s too infrequent and ad hoc to have any programmatic tools needed.

I was thinking about those who might want to apply your metadata concept retroactively to their media, regardless of the folder and naming convention. I did something years ago between RM and metadata to help manage media with Picasa:
https://sqlitetoolsforrootsmagic.com/media-metadata-read-write-compare-with-picasa/

[KimberlyGreen]

@TomH

I was thinking about those who might want to apply your metadata concept retroactively to their media

AH! Yes, that might be useful to others.

Suggestion … we shouldn’t hijack this thread, since it’s a tangent to Robert’s question. Perhaps you’d like to make a new thread where the discussion can continue?

[KimberlyGreen]

@RobertL

may start reorganizing by my Lines

A consideration: If you organize media by lines, will media for women reside under their maiden surname or married surname? If married, will media for pre-marriage reside in one area and another for married? What if they marry multiple times?

This is why I try to avoid organizing by human names as much as I can. It gets messy.

[TomH]

@KimberlyGreen

make a new thread where the discussion can continue?

Probably should but I do not wish to bite off more than I can chew. I did that work more than a decade ago and my grey cells have gotten a lot greyer!
@rzamor1 can you move this fork to a new topic starting with Kimberley’s:
https://community.rootsmagic.com/t/renaming-media-files-in-rm9/6992/8

[RobertL]

I don’t mean organizing media by lines, but working the data by lines.

[KimberlyGreen]

@TomH

I was thinking about those who might want to apply your metadata concept retroactively to their media, regardless of the folder and naming convention.

@rzamor1 Thanks for spawning a new thread so we could continue the discussion

@TomH I think your concept has a lot of merit. It would be great if RM had a function for it embedded. But even if not, an SQL tool would be useful for those who wanted it.

You could pick up from RM the names that the media is linked to, then somehow push those into the file properties area through OS calls. If fact, while you’re at it, you could even push the source information into the ‘Comments’ field, etc.

That, too, would make it much easier to look at the unorganized RM media directory of a TreeSync and figure out what is what.

[TomH]

@KimberlyGreen

could pick up from RM the names that the media is linked to, then somehow push those into the file properties area through OS calls.

My liaison between RM and image file metadata has always been via sqlite generating batch files to run ExifTools on the image files to deliver metadata or retrieve them in a text file for import to sqlite. It’s a crude, cumbersome procedure. Is there an application such as Excel, PowerShell… that can ODBC to the sqlite database and read-write metadata on image files?

[KimberlyGreen]

Unfortunately, programming is not in my wheelhouse. So, I asked ChatGPT . Here’s what it said:

Can’t vouch for ChatGPT’s accuracy, but on the face of it using PowerShell sounds like an option.

[TomH]

Ha! …minds think alike but I posed the question differently and got an answer at a higher or more general level:

Is there an application that can connect to a sqlite database and read-write metadata on image files?

Yes, there are several applications that can connect to a SQLite database and read-write metadata on image files. Here are a few examples:

1. ExifTool - This is a powerful command-line tool that can read and write metadata in a variety of file formats, including JPEG, TIFF, PNG, and many others. It can also read and write metadata stored in a SQLite database.
2. PyExifTool - This is a Python wrapper for ExifTool that allows you to easily read and write metadata using Python. It also supports SQLite databases.
3. Digikam - This is a free and open-source photo management application that supports a variety of image formats, including JPEG, PNG, and TIFF. It also has built-in support for SQLite databases and can read and write metadata.
4. Lightroom - Adobe Lightroom is a popular photo editing and management application that supports a wide range of image formats, including JPEG, TIFF, and PNG. It also has built-in support for SQLite databases and can read and write metadata.

These are just a few examples of applications that can connect to a SQLite database and read-write metadata on image files. There are many other applications and tools available, depending on your specific needs and preferences.

With Digikam, would it be feasible to exchange data between its database and a sqlite family tree database that Digikam would write to the image files and retrieve for comparison.

Yes, it is feasible to exchange data between Digikam’s database and another SQLite database, such as a family tree database. Digikam has built-in support for SQLite databases and can read and write metadata to image files.

To exchange data between the two databases, you could use SQL commands to query and update data in both databases. For example, you could write a script that reads data from the family tree database, uses that data to update the metadata on the corresponding image files in Digikam’s database, and then writes any updated metadata back to the family tree database.

You could also use Digikam’s API to automate the process of updating metadata. Digikam provides a powerful API that allows you to access and manipulate metadata programmatically. With the API, you could write a script or application that reads data from the family tree database, uses that data to update the metadata on the corresponding image files in Digikam’s database, and then retrieves any updated metadata for further processing or comparison.

Overall, while there may be some challenges involved in exchanging data between two different databases, it is certainly feasible to do so using Digikam’s built-in support for SQLite databases and its API.

[----------------------------------------------------

That sounds promising and Digikam looks to be a worthy successor to Picasa, provided its sqlite database is not encrypted…
https://www.digikam.org/

P.S. I’m not in the programmer’s wheelhouse either!

[KimberlyGreen]

@TomH

P.S. I’m not in the programmer’s wheelhouse either!

Fair point. A DBA / data integration person isn’t exactly a programmer. But they’re about 1000 steps closer to that technical skills realm than me, lol.

If you decide to tinker with craft up a solution to your concept, I’d certainly be interested.

[TomH]

I tinkered with digiKam and had some success.

That’s the result of a query to attach the digiKam database to a RM database, generate a list of people to whom each media item has been ‘tagged’ in RM, translate that to new records in the dK Tags table and links from there to the dK Images table. Then when dK is reopened, have it write the tags from its database to the image files.

Some of what I learned is that the Subject field you use is a Windows-specific field named XPSubject that is poorly supported in digiKam and elsewhere. Also that the digiKam database really only supports one Caption (or Comment) field per media file and tags (or keywords), so I had but one choice where the data should go. And it all gets very confusing among EXIF, IPTC and XMP metadata specs. IPTC tags are limited to 64 bytes, which could be a limit of less than 64 characters if any 2-byte Unicode ones are used. XMP tags are unlimited and it seems that dK does its best to place the data from its Tags records into a field of the appropriate standard.

I’m unsure whether the data should be concatenated to one tag per media file or left as, say, a tag per person so that a file gets multiple tags. I don’t think Windows or digiKam care - they do the concatenation of multiple tags (there are a couple of examples above where I had entered tags manually in dK).

OTOH, just now having uploaded a file to Google Photos and to GDrive, I find that the tags are neither displayed nor searchable but the Caption is. So rather than transmitting the RM Caption to the dK Caption as I thought I might, maybe it is better to send the concatenated string of data that I had put into Tags. It would actually be easier!

[epona548]

I have a Mac and use a ‘type’ folder system like Kimberly.

The citation for each item goes in the “get info” area since those can get crazy long. So far, I have been using item name to include basic what and who. I like the PC function of being able to add subject names like that, but unless you add them after the citation (which sometimes it may not all fit) the only other option is adding them as a tag. But I tried tags before on Mac and just didn’t like it. The ‘get info’ box char limit needs increasing.

[KimberlyGreen]

@TomH Lol, I just knew you’d get a wild hair and want to tinker. It’s a fun little challenge.

Personally, I chose to use the ‘Subject’ field for names because that’s the subject of the image. I left ‘Tags’ for future use. For example, if I wanted to tag a picture with a place name or category (ie census, death cert, etc).

Your findings about the Subject field for Windows, and the MacOS information from @epona548 got me curious. So I fired up my Linux Mint virtual machine and inspected how Linux might do things.

I copied across a Windows file that had some ‘Subject’ information. I discovered that the default ‘Files’ app doesn’t reveal any metadata at all. In fact there’s not even a way to add any. But one of the image viewing apps, Pix, does show it in read-only.

Seems like there’s no universal tagging solution across operating systems.
¯\(ツ)/¯

(I know RM doesn’t run on Linux. Just adding that insight because I found it interesting.)

[Biggles]

Long term the management of Family Records will not be up to me so no fancy programming, no total reliance on Software by itself.

RM will be on a USB memory stick but all media and records will be within folders and simply catalogued in a Index file with this file in various formats to make it as future proof as possible, a printed version will be in the Box with family albums and memorabilia and treasures