Extracting NFO/poster/bg from mkv attachments

[DingoPD 5]

I started adding emby generated nfo's and chapter thumbnails into MKV's as attachments, as a form of backup. If (when) the database blows up, it just takes a simple script to extract the files and make the restoration process much less CPU and time intensive. My goal is to eventually add posters and backgrounds also ... when i figure out how to find them.

My question is: Do i even have to extract the files back out of the MKV's or can Emby actually read them straight out of the MKV when looking for metadata?

Thanks!

PS: Here's a pastebin of example MKV structure (in JSON).

Edit: Oops, I meant a hardware failure, not trying to imply there's a problem with Emby

Edited April 16, 2021 by DingoPD

[Carlo 4561]

Hi,

You can setup libraries to save graphics and NFO files directly with the media so if read in again later it would have your custom info from the NFO and would already have your graphics on hand.

You can also make use of the Configuration Backup plugin that backs up all key config info including user watched states.  Set this up to save to a network location (if possible) and this info is saved and will be usable if your main computer were to die on you.

You can always manually backup the Emby "Data" folder as well which includes the DB one a month/week as well for added piece of mind.

The first two things make reloading and setting up your system much easier while the 3rd item protects you from a corrupted database.

[Luke 42083]

You can't embed the nfo into the mkv, or the chapter thumbnails, if that's what you're asking.

[DingoPD 5]

5 minutes ago, Luke said:

You can't embed the nfo into the mkv, or the chapter thumbnails, if that's what you're asking.

I was asking if there is anything that Emby would automatically look for in the matroska attachments when identifying the file, for example poster/background images or the NFO. I don't really expect Emby to look for it every time someone browses the media :). That'd be a bit resource intensive.

If that was the case, i wouldn't have to extract the files by script ...

[Luke 42083]

Poster yes, background no, although if there's some kind of standardized convention we could look into that.

[DingoPD 5]

8 hours ago, cayars said:

Hi,

You can setup libraries to save graphics and NFO files directly with the media so if read in again later it would have your custom info from the NFO and would already have your graphics on hand.

You can also make use of the Configuration Backup plugin that backs up all key config info including user watched states.  Set this up to save to a network location (if possible) and this info is saved and will be usable if your main computer were to die on you.

You can always manually backup the Emby "Data" folder as well which includes the DB one a month/week as well for added piece of mind.

The first two things make reloading and setting up your system much easier while the 3rd item protects you from a corrupted database.

Yeah if i had to set up emby again i'd include the images in the same location as the media file. I don't really have the capability to store backups on another machine or off-site, so i use the backup plugin and rsync to have it on 2 different physical drives. That's the best i can do.

What happened to me last time was a raid controller meltdown that nuked every file that happen to be written into over the 10-12 hours before i got home. It was a mess, but between the backups and the media storage that didn't really get hit (write once, read many times), i was back up in couple of afternoons.

[DingoPD 5]

42 minutes ago, Luke said:

Poster yes, background no, although if there's some kind of standardized convention we could look into that.

I understand that it would be niche feature. Unless there is a full support from the media server, (automated embedding/extracting), very small portion of the users would ever bother with it. So I'm not trying to ask for something like that.

There's lot of containers that support embedded standardized metadata - XMP/CMML/RIFF. Is a big mess at the moment with everyone just making up the tags as they go. XBMC/Kodi had a discussion about reading embedded metadata years ago. I've no idea if they ever implemented it.

Only 4 video containers, that i could find, support attachments.

I try to stick with matroska as much as possible, and for my purposes i think that embedding all the metadata and images required, within the media file itself is probably the least resource intensive way to have it all at hand without having to pull the metadata from some remote API.

I'm still looking into a way to append an attachment without remuxing.

[Guest]

Yeah... in MetaX.. you can write in Chapters from file import.. several different types including ChaptersDB but no images except covers.. and yes MKV or MP4 or the two types.. WMV sort of supports tags and only a few tags are supported with certain types of AVI.. however image embedding is not acceptable at all for AVI.

[DingoPD 5]

2 minutes ago, Hxemby001 said:

Yeah... in MetaX.. you can write in Chapters from file import.. several different types including ChaptersDB but no images except covers.. and yes MKV or MP4 or the two types.. WMV sort of supports tags and only a few tags are supported with certain types of AVI.. however image embedding is not acceptable at all for AVI.

Matroska, MPEG-4, BDAV-MPEG-2 and QuickTime are the only video containers, i think, that support attachments of any type out of the box. WebM supports MPEG-DASH that could be probably used for embedding metadata.

You are right about AVI, there's RIFF support in AVI, but no support for subtitle streams, so that container might as well curl up in a dark corner and die swiftly.

Out of the five above, only Matroska has a free license, (WebM's MPEG-DASH is patent encumbered). That's the biggest reason i picked Matroska. Patent encumbered and proprietary license containers tend to die after few decades ...

I've never heard of MetaX, had to look it up :). Looks nice.

I mostly stick to ssh console and scripting. my media server is headless, only UI you get is console. My gigabit ethernet is the bottleneck. I could probably set up remote GUI, but that's a lot of overhead just for modifying files. Since i started this thread, i read up on mkvpropedit and it can add attachments without remuxing, so that is what i will most likely do. I still have to figure out if fragmentation is going to be a problem on XFS if i do that. If fragmentation becomes a problem, i can probably just xfs_fsr it once i'm done.

I'm trying to do something with pieces from multiple sources, that are not always guaranteed to be present when i am. So i will automate it as much as i can.

[Guest]

Yes I did definitely forget about MOV.. MetaX is definitely useful... good tool getting my archive together.. Getting adult metadata has ben harder.. BUT I still use it to plug everything in by hand.. 

I actually plugged data into my trailers to get them to display and show metadata when they are playing.. very similar to the actual movie file but use the 'movie name' Trailer for the Chapter and filename, embedded filename with a short description.. I get STAR rating, film Rating, year and a proper name out of it.. Do need to work on my intro video.. but haven't really found the right one yet..

About the bottleneck.. eSATA or USB 3 ( maybe even Gen 2 ) is what I have been looking at lately... to data repositories.. MediaSonic has caught my eye lately..

[DingoPD 5]

21 minutes ago, Hxemby001 said:

Yes I did definitely forget about MOV.. MetaX is definitely useful... good tool getting my archive together.. Getting adult metadata has ben harder.. BUT I still use it to plug everything in by hand.. 

I actually plugged data into my trailers to get them to display and show metadata when they are playing.. very similar to the actual movie file but use the 'movie name' Trailer for the Chapter and filename, embedded filename with a short description.. I get STAR rating, film Rating, year and a proper name out of it.. Do need to work on my intro video.. but haven't really found the right one yet..

About the bottleneck.. eSATA or USB 3 ( maybe even Gen 2 ) is what I have been looking at lately... to data repositories.. MediaSonic has caught my eye lately..

That is a nice setup

The bottleneck problem will solve itself once 10gig ethernet infrastructure doesn't cost an arm and a left nut.

I could always just pick up couple of cheap-ish 10gig cards and do ad hoc connection between my normal computer and the media storage, they on top of each other in the rack anyway, a 2ft cable would do it. I just can't really justify the cost.

For storage i got old T-5500 motherboard (2xE5620, 48Gb RAM) in a 4U case with 12 sata3 hot swappable trays, i only use 8 of them right now, so i got space for expansion when i need it. It's all JBOD with important data rsync'ed weekly to multiple physical drives. Media is stored in standardized directory structure, and then symlinks are auto generated with scripting to form directory structure that Emby likes. I just named the mounts media0-media5 (last two are root and SSD for frequent file operations). It's all quite easily expandable. And fairly robust, in a sense that when one drive manages to die, i only lose a fraction of the data, not all of it, like it could happen with raid.

I tried the external storage route years ago and it ended up looking like something out of a matrix movie. And back then, the bandwidth just wasn't there. It was slow, complicated, and it had way too many points of potential failure. Modern hardware is much better in every aspect of what i had back then.