Jump to content
topbanana

Improve reporting features

Recommended Posts

topbanana

Your libraries of movies, TV, etc. almost certainly have lots and lots and lots of misidentified items, errors...  5%, perhaps more is possible...
And 5% of a large collection, is a lot of items.  Wrongly identified.
 

Emby relies on the various databases on the internet, and between them, they inevitably make the odd mistake...   That's fair enough.  Many factors are involved, bad filenames, rare titles, foreign character, etc, etc... Some are just stupidly wrong. They're going to happen...  

BUT.

We still don't have any tools to find them.
You have no way of knowing how many errors are in your libraries.
We have to rely on stumbling upon the error, or if you have an intimate knowledge of what titles you do have, you'll spot a few more of the errors perhaps.

As attempted a couple of times, some of us have tried combining the grossly, needlessly inadequate 'Reports' feature (now a plugin), with data pulled from the database files with external tools.  This has worked OK, but is a royal PITA, and not slick, not quick to do.   But it does show you how many errors emby holds.  Which might horrify/surprise you.

If you want some fun, reinstall emby from scratch, rebuilding your libraries... You'll very quickly see some errors, with many more being there without you knowing.

ALL WE WANT is the ability to, at least, list the path, filename, against the title and year from metadata downloaded...   
This simple feature, ability, would make finding these numerous errors possible, if a little bit slow to process (takes a lot of concentration to check 5,000 filenames, titles, years).

Later, yeah, it would be sweet to have it highlight non-exact matches, that would speed up the process immensely. But we need the minimum now.

 

These tools need to be in the core of emby...  It's not good enough that this media database application doesn't have the most basic data analysis tool!


Path, filename, title, year.

Please.

Edited by topbanana
  • Like 1

Share this post


Link to post
Share on other sites
PenkethBoy

LOL - bit Ranty!

 

You can do this easy enough with Db Browser for Sqlite (free) - yes it would be nice if this were built in but its not so....

 

Shut down you Emby Server and open the library.db file

 

Then use the SQL tab

 

e.g

5bb7628d8b48c_Capture2.jpg

 

You can then export this to a csv and do your analysis (hint look for the icon in the bottom right of the screen)

 

This is for the beta server only and i think i got all the relevant types but easy enough for you to add them

 

Only took 5 mins to write and test

  • Like 2

Share this post


Link to post
Share on other sites
TylerV76

I don't have any errors or misidentified items in my library, zero. When I add new content, I see it in the "new" section and if it doesn't match I correct the metadata. This is EXTREMELY rare and typically only on an obscure file.

 

As long as the file is properly tagged you shouldn't be having any issues.  

Share this post


Link to post
Share on other sites
topbanana

LOL - bit Ranty!

It's been going on for years...  We've asked for the most basic of tools (just adding the path, filename columns to the reports, absolute basic db task). But still we have nothing.

 

 

You can do this easy enough with Db Browser for Sqlite (free) - yes it would be nice if this were built in but its not so....

Yes, we've detailed this method in another thread.

It works a treat, i found many errors this way...  but is very time consuming and not the kind of thing every, or many emby users wants to do.

 

It would be much better to get the software to do this for us...  That's the point of software like this, wrangling huge amounts of data and automating tasks... Emby is excellent as a media server, but it doesn't yet have any feature to help us find the errors that unavoidably, inevitably happen.

All we want to do is, say, go to the report, compare the path and filename with the title and year that emby has gotten from the internet databases.

 

 

Only took 5 mins to write and test

So much of this software can be done externally in only 5 minutes...

But, we're better off paying a programmer to spend that 5 minutes writing the code and adding it to the software itself...  That way all the users can make use of the feature over and over and over.  Adding value to the software.

 

 

 

You can then export this to a csv and do your analysis (hint look for the icon in the bottom right of the screen)

Interested: Did you look through your library this way?

How many errors did you spot?

 

 

 

I don't have any errors or misidentified items in my library, zero. When I add new content, I see it in the "new" section and if it doesn't match I correct the metadata. This is EXTREMELY rare and typically only on an obscure file.

You seem very confident.

And this is entirely possible, sure. But it looks like you've invested a lot of effort up front to ensure this.  Great work.

 

But.  You have no quick or easy way of finding out if you are correct.

 

If you were to reinstall emby and rebuild your libraries from scratch, how many errors might be made?

If you were given 100+ movie files, with no nfos, just the correctly named files, how many errors would emby make iding them all?

 

 

As long as the file is properly tagged you shouldn't be having any issues.  

I don't use nfo files.  As don't many others.

Instead, i just have 'perfectly' named files.  I mostly just get filebot to name them 'title (year) (720p).mkv'...  And i correct the mistakes made manually, pretty much the same way you do.    All are correct, as far as i know (i'm realistically only 99% sure though), compared with tmdb, omdb, imbd, etc...  

But even with 'perfectly' named files, emby makes mistakes.  Some are slightly wrong, some are totally wrong.  Many have been linked to the infamous premiere date bug that's been going on for years too... 

 

A stupid example.  Given these two movies:

Live Free or Die (2006).avi

Live Free or Die Hard (2007).avi

Similar names, different years, both not obscure...  emby identifies them BOTH as 'Live Free or Die Hard (2007)'...  So an 'easy' error to spot if you happen to browse past the movies in emby, as the posters are obviously identical... and either you have two files for the same film, or emby has made a mistake.

 

This it just one example.

Emby mostly makes mistakes such that a completely different title is identified, so you have no way of knowing the error exists unless you browse past the movie poster and recognise it as not a film you thought you had...  But that takes intimate knowledge of your libraries.

 

If i'd kept a log of all the mistakes i'd stumbled across, or found using the Db Browser for Sqlite tool, you'd find it horrifying or hilarious.

 

You, I, we just need a feature in emby to easily find these errors.

Share this post


Link to post
Share on other sites
Happy2Play

Or have a look at EmbyStats Server, it is a work in progress as links need updated with url "/index.html#!/" change.

 

5bb8539c1e02f_suspicious.jpg

Share this post


Link to post
Share on other sites
topbanana

Or have a look at EmbyStats Server, it is a work in progress as links need updated with url "/index.html#!/" change.

Again, it's an external tool.

Does it find the blatantly wrong mistakes, that are fairly common?  i.e. wrong movie title and year altogether.

One that i couldn't get to run, and i don't have docker, but anyway...

 

Emby makes mistakes, will always make mistakes.  They're unavoidable.

So emby should include at least the minimum feature/tool to help us find them.

All emby users should have this at hand.

Remember, there's probably a very high percentage of us pendantic, ocd spectrum emby users out there!   We love to sort and maintain our collections! And we can't live knowing theirs blatant misidentified movies in there!  ;-)

 

Adding a database column or two to a report is the absolute basic of basic tasks for this software.

 

 

Path, filename, title, year.

 

Pretty Please

Share this post


Link to post
Share on other sites
topbanana

Stumbled across another of the easy to spot errors.

Emby was given these two movies:

New York (2009) (720p).mkv

Synecdoche, New York (2008) (720p).mkv

And it identified them both as 'Synecdoche, New York (2008)'.  Saw the two identical movie posters next to each other.

Again, fairly easy to spot when passing them, by chance.  But i had no way of knowing or finding the error otherwise.

Share this post


Link to post
Share on other sites
Happy2Play

Stumbled across another of the easy to spot errors.

Emby was given these two movies:

New York (2009) (720p).mkv

Synecdoche, New York (2008) (720p).mkv

And it identified them both as 'Synecdoche, New York (2008)'.  Saw the two identical movie posters next to each other.

 

Again, fairly easy to spot when passing them, by chance.  But i had no way of knowing or finding the error otherwise.

 

Good example of providers returning a wrong result.

 

 

.

Does it find the blatantly wrong mistakes, that are fairly common?  i.e. wrong movie title and year altogether.

One that i couldn't get to run, and i don't have docker, but anyway...

 

From a single movie aspect, I would say no as the only way to compare is title to path assuming path is named correctly.

 

As for your example above they would be linked to the same metadata id so yes the stat server will catch it as shown above.  I don't use docker either, I got the windows zip, but that won't work either if your not on windows.

 

Things like this will always require additional tools.

Share this post


Link to post
Share on other sites
topbanana

Using DB Browser for SQLite for a very, very quick n dirty scan (just looking for first letters that don't follow the alphabet listing):
 

Miles (2017) (720p).mkv   ...   Emby IDed as '600 Miles (2016)'
Game of Death (死亡遊戲) (1978) (720p).mkv   ...   Emby IDed as 'The True Game of Death (1978)'
Princess (2014).mkv   ...   Emby IDed as 'Barbie in Princess Power (2015)'

The Experiment (Das Experiment) (2001) (720p).mkv   ...   Emby IDed as 'Acceptable Risk (2001)'
The Son (Le fils) (2002).mkv   ...   Emby IDed as 'Son of the Bride (2001)'
 

So all these were blatantly wrongly identified.

But i had to go to a website, download a software tool, copy some code from a forum post i could track down only because i participated in it, copy the database file out of my user folder, open it up in the software, run the copied code in the tool... Only then to get the 3 columns that allowed me to carefully scan down through them, to see if anything didn't match.  Only then did i find these errors with a few minutes of scanning.

Share this post


Link to post
Share on other sites
topbanana

Good example of providers returning a wrong result.

Absolutely.

Emby is only as good as the providers it's using to id the movies.  So we can't blame emby, but the error exist within emby's libraries.  So we should have the ability to find the mistakes made by whoever.

 

 

From a single movie aspect, I would say no as the only way to compare is title to path assuming path is named correctly.

Yes.  It's up to us to present the movie to emby with the correct path/filenames.

But emby (it's providers) make mistakes even when the path/filename is correct.  So we should have the ability to find the mistakes made.

 

 

 

Things like this will always require additional tools.

The request, which goes back years, has been to, as a minimum, add the path/filename columns to the reports feature.

Adding a column to the reports is dead easy.  The most basic of requests for a piece of software built on a database.

The Reports feature is now a plugin, ok.  But this additional feature should be included.  And i think it should be in the core of emby.  Finding the errors that unavoidably happen should be a task that emby deals with.

 

We want emby to be awesome.  It presents lots of beautiful posters and info about our movie collections, allows us to browse them, and to play them on all our devices.  But we also need it to help us find the errors that always happen.

Share this post


Link to post
Share on other sites
Happy2Play

Not that I really have any idea how I did it but are you wanting this in the Reports plugin, paths for Movies and Episodes.  I will have to document all the changes I did to the download.

 

5bb886f14fc3d_reports.jpg

  • Like 2

Share this post


Link to post
Share on other sites
topbanana

Not that I really have any idea how I did it but are you wanting this in the Reports plugin, paths for Movies and Episodes.

Awesome job!

Yep, that's pretty much it!

 

For the task in hand, it's best if the title and year are together, and the path on one side or the other, so that you can compare the two. Reports doesn't allow column reordering i think.

I've been using DB Browser with the path to the left, and the name, year to the right. With the columns as such, you could probably get by without having to export anything to excel, etc, you could compare them right there in emby. Which is desirable.

 

Thanks Happy2Play!

Edited by topbanana
  • Like 1

Share this post


Link to post
Share on other sites
Happy2Play

So like this.

 

5bb89f6c22a1f_reports.jpg

 

Submitted pull request on GitHub.

 

@@Luke is there a way to make Columns options persistant, to maintain column selection after browser refresh?

  • Like 1

Share this post


Link to post
Share on other sites
topbanana

So like this.

...

 

Submitted pull request on GitHub.

Yep!

Awesome.

Share this post


Link to post
Share on other sites
Luke

So like this.

 

5bb89f6c22a1f_reports.jpg

 

Submitted pull request on GitHub.

 

@@Luke is there a way to make Columns options persistant, to maintain column selection after browser refresh?

 

A new contributor could help add this to the reports plugin, yes.

Share this post


Link to post
Share on other sites
Happy2Play

A new contributor could help add this to the reports plugin, yes.

 

I submitted a pull request, is there something else I need to do?

  • Like 1

Share this post


Link to post
Share on other sites
Luke

No, that's good, thanks.

  • Like 1

Share this post


Link to post
Share on other sites
TylerV76

Edit: NM 

Edited by TylerV76

Share this post


Link to post
Share on other sites
jlr19

Not that I really have any idea how I did it but are you wanting this in the Reports plugin, paths for Movies and Episodes.  I will have to document all the changes I did to the download.

 

5bb886f14fc3d_reports.jpg

I too have been crying out for error reporting forever, I had all but resigned due to no support or even mild interest from other users or so it seemed...

I am really interested by this added feature, how will I get it eventually?

 

And I salute @@topbanana's persistence and well presented case and arguments.

  • Like 1

Share this post


Link to post
Share on other sites
jlr19

I don't have any errors or misidentified items in my library, zero. When I add new content, I see it in the "new" section and if it doesn't match I correct the metadata. This is EXTREMELY rare and typically only on an obscure file.

 

As long as the file is properly tagged you shouldn't be having any issues.  

I am glad you have no errors/misidentified, and yes, sorting your library by descending date added is useful to catch recent stuff, but not so much on larger collections.

As far as your statement file (and path) are properly tagged warrants no issues, I disagree. I am constantly searching movies on themoviedb and tv show on tvdb.com and I am always astonished by the numbers of search results I am getting (some so far off it's almost funny) from an otherwise PERFECTLY named search. What I really don't get is sometime you find a perfect match, but only scrolling down to the 5th results for instance. How can anyone expect Emby to always grab the right data when the website it depends upon is so "loose".

Edited by jlr19

Share this post


Link to post
Share on other sites
jlr19

Case in point, this search should only yields 2 results, here they appear as 10th and 11th . . .

5bba192da51a8_Capture.png

  • Like 1

Share this post


Link to post
Share on other sites
topbanana

I am really interested by this added feature, how will I get it eventually?

Happy2Play has queued it up in github, so it should be in the beta channel soon i gather, then stable soon after that?

Just install the Reports plugin and the path column will eventually be there after an update.

 

I've been able to just scan down the list in emby, without having to resort to exporting it to excel... I just zoomed into the page in chrome to make the text massive, and shifted the window to the left to obscure the common part of the path, that's the same for all my films, such that only the film's filename is visible.  Then it's quite manageable to check the filename against the title/year for each.

I have 4,600 movies now, so that confidence of there being zero errors is non existent... and the history of finding errors, some repeatedly after rebuilds, the same errors each time, and the new ones that were seemingly correct before, but now get IDed wrong...  

 

Future...

Obviously, this is a great step.  This functionality enables us to practically find the errors.  It takes time to check each item, but it's possible...  Just do a letter or two a day, otherwise you go cross-eyed! :-D

To streamline this task tho, sometime in the future, it would be awesome to add some tools, tricks, to help with this.

The obvious trick, that would work for most files, for most people, possibly... would be to remove, or dim, the movies where the filename 'title (year).mkv', perhaps excluding any bracketed tags, perfectly matches the 'title' and 'year' emby IDed.  This i guess is fairly possible?  This should remove the majority of items, leaving only the movies with foreign names included or variations in the title regionally... and of course the errors...  So it would cut the time down to a small fraction.

Other tricks will help others with different structures, and 

I think if/when some others try this simple error check out, they'll realise how many errors are in their libraries, and they'll join us in requesting better tools to help fine them.

 

And I salute @@topbanana's persistence and well presented case and arguments.

No worries.  We knew it was an obvious feature that was needed.  And i land up rebuilding emby's libraries more often that most probbo... (OS clean installs... and forgetting to backup emby! lol)

As above.  To recognise the need for a fix, you have to know that their's a problem to begin with...  and i'd guess most emby users don't realise how many errors there are in their libraries, how many errors the movie databases return to emby when it searches them, as you point out...  Most just stumble across the odd one or two... and that's not enough to trigger them to head to the forums to rant about it...

 

We can't avoid these errors, so emby now has a tool/method to find them...  So emby is even better now!

Share this post


Link to post
Share on other sites
TylerV76

I am glad you have no errors/misidentified, and yes, sorting your library by descending date added is useful to catch recent stuff, but not so much on larger collections.

As far as your statement file (and path) are properly tagged warrants no issues, I disagree. I am constantly searching movies on themoviedb and tv show on tvdb.com and I am always astonished by the numbers of search results I am getting (some so far off it's almost funny) from an otherwise PERFECTLY named search. What I really don't get is sometime you find a perfect match, but only scrolling down to the 5th results for instance. How can anyone expect Emby to always grab the right data when the website it depends upon is so "loose".

 

 

My movie folder shows 1534 items, all correctly identified. I get that some people may have issues as none of our file lists are the same. My point simply being, not everyone has this issue. I couldnt tell you the last time I needed to correct a file, with the exception of the TV series reboot issue. 

 

This is after multiple reinstalls on different systems while trying to find what server worked best. 

 

Im not doing anything special so Im not quite sure why I don't have any problems but Im glad to see the OP got what was needed to correct their issue.

Share this post


Link to post
Share on other sites
jlr19

jlr19, on 07 Oct 2018 - 07:35 AM, said:snapback.png

I am really interested by this added feature, how will I get it eventually?

Happy2Play has queued it up in github, so it should be in the beta channel soon i gather, then stable soon after that?
Just install the Reports plugin and the path column will eventually be there after an update.

I've heard that little wonder is available in beta, but where and how do I find it?

Share this post


Link to post
Share on other sites
Luke

It depends on emby server 3.6, so you won't get the updated plugin until you have the next release of emby server.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...