Jump to content

Auto-Organize PLUS - public beta


chef

Recommended Posts

chef

 

 

Auto-Organize PLUS - public beta

thumb.gif.19e496f22d8146a4c515ebf1dc1b7c92.gif

 

Built upon the Original Auto Organize plugin, Auto-Organize PLUS is a new version which has some enhancements to help sort new media files into the Emby library.

 

Mobile View:

1727580695_mobileorganize_2.png.40df3f9ef73959303dcbfec23e118ffb.png

 

Enhanced Table Layout: with Media Info about the files in the watch folder.

1519237551_interestingbug_fixed.thumb.png.8ee2d45bfaa5d179078bf5533e1bc7bd.png

 

Multiple Watch Folder Settings:

source_folder_settings.png.c1b4d75654249ceeb30352995eea232c.png

 

Existing Library Item/Watch Folder Item Media Info Comparisons:

comparison.thumb.png.cd3bcb9cd2608feb03f914e3adaff532.png

 

File Status/States:

Files in the Watched folder can be in several states depending on user settings.

 

  • image.png.a8446fe8bbd2f875eca61f11ca107d0a.png Existing Item

Existing Items are items that exist both in the watched folder, and also already exist in the Emby library.

Users can click the existing item icon    to compare the two items, and help decide if the file should copy over into the library folder, or not.

  • image.png.8e1b4ae0b32d0ca199228dbcefb400e0.png  New Resolution 

New resolutions are items that exist in the library, but may not have the same quality as the item in the watched folder. For example: 1080p, and 2160p.

The "New Resolution"  status will happen, when "Smart Auto Detection" is turned off for a certain media type.

  • image.png.7b03de0f8fcb3397423e89ab8eb8b1ad.png Processing

The Processing status  will happen when a file is being copied, or moved into the library folder.

  • image.png.52849bcc64224aa9b841e9f3d7eaa279.png Success

Success happens when a file has completely moved into the library folder with it's new name

  • image.png.dee5ed5f433b8a529f4d9be196fd5d5c.png Failure

Failure  happens when a file name can not be fully parsed, and the media can not be understood by Auto Organize 2.0

  • image.png.40167522322978712b74f217414691df.png  User Input Required (Pending...)

If a particular TV Show spans multiple folders within the Emby library, Auto Organize 2.0 will wait for user input to decide where to place the file. This state will may  happen with "Smart Auto Detection" turned off, or on.

  • image.png.dee5ed5f433b8a529f4d9be196fd5d5c.png Not Enough Disk Space

If a file hasn't enough room to copy into the library folder, because it is full.

  • image.png.acafde2a72f39e615dbe04e1b93b7089.png  In Use

When a file is being copied into the monitored folder from another source, or is being used by another process.

When the file becomes available the status will change.

  • image.png.d1abe37f3c072c516f74d207b0c677fb.png  New Media

When a file has been added to the monitored folder, and it doesn't exist in the Emby library, but "Smart Auto Detection" has been turned off. The file will enter the "New Item" state if all applicable metadata can be obtained to name it. This will allow the user to sort it when they are ready to.

  •  New Edition

New Edition will happen if the media item in the watched folder has a different release parsed in it's name. For example: "Theatrical", and "UnCut".

 

Table Sorting:

filter.png.1d6ae926e566636948224b59b660e916.png

Sort the files by stripping out certain types of media, and only showing others.

Also, filter the current displayed items by "Name", "Date", and "Status"

filter_table.png.58076b60cd34e8f32bad2036d277a93e.png

 

Current Issues, and on-going development

  1. Plugin configuration Page Headers are currently having issues in Emby. Options are not hidden behind the Header on scroll. This is a core issue that is currently being fixed
  2. Smart Matches are currently being enhanced, and the experience with smart matches is currently not what the end result will be.

 

Installation notes:

  • Please shut down the server
  • Remove the "fileorganizer.db" file found in Emby's data folder.
  • Place the plugin DLL in the plugins folder
  • Re-start the server.
  • Clear browsing data 

 

DOWNLOAD (stable server only): Emby.AutoOrganize_v1.6.41.43.zip

 

 

Beta server: Emby.AutoOrganize_v1.6.41.23b.zip

(Not updated recently)

Edited by chef
  • Like 7
  • Thanks 4
Link to comment
Share on other sites

chef

Download has been added to the main post. :) 

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

chef

I am currently experiencing some strange behaviour while using the mobile interface, and I'm looking into it now.

  • Sad 1
Link to comment
Share on other sites

Cheesegeezer

@chef you little tiger!!!

nice work! I shall give it a test drive tomorrow and let you know buddy!

2 hours ago, chef said:

I am currently experiencing some strange behaviour while using the mobile interface, and I'm looking into it now.

Keith will be happy you’re sorting this out lol 😂🤪😜🤪

  • Like 1
Link to comment
Share on other sites

chef

I've fixed the strange update behaviour of the mobile cards, and I will put a new link in the main post. It will be version .24

  • Like 1
Link to comment
Share on other sites

gihayes

I haven't tried the version in your latest post but the previous version is working great for me!! And subtitles too!! Great man👍. I tried it with 2 seasons from 2 series 1 movie. Everything processed fine. The only thing it didn't do was clear the txt files from the watch folder. So it did not delete the processed folders. This is such a time saver!  Great work!

  • Thanks 1
Link to comment
Share on other sites

chef
1 hour ago, gihayes said:

I haven't tried the version in your latest post but the previous version is working great for me!! And subtitles too!! Great man👍. I tried it with 2 seasons from 2 series 1 movie. Everything processed fine. The only thing it didn't do was clear the txt files from the watch folder. So it did not delete the processed folders. This is such a time saver!  Great work!

Thanks for the feedback.

Looking forward to more as we continue working on this.

👍

 

Link to comment
Share on other sites

chef

Just waiting on some last minute testing results before posting .24.

 

I did come across an interesting thing while using auto organize 2.

It has to do with provider data for TV shows that have been released but are marked as "TBA" in the Provider API.

Auto organize will sort the file but name it "TBA". This seems fine, as they become accessible in Emby, and they can be viewed.

What is interesting is what happens later if you obtain the file again, and the providers have updated their API results.

The file will sort it self with the actual episode name, and not overwrite the "TBA" version in the file system. 

Perhaps records of  TBA results be kept somewhere and delt with during a task when the actual name can be resolved by the provider.

 

 

Edited by chef
Link to comment
Share on other sites

neik
4 minutes ago, chef said:

The file will sort it self with the actual episode name, and not overwrite the "TBA" version in the file system. 

Wouldn't it make sense to check the Emby library whether the episodes already exists or not, instead of doing that based on file name?

It would be more bulletproof, I guess.

Edited by neik
  • Like 1
Link to comment
Share on other sites

Cheesegeezer
1 hour ago, chef said:

Just waiting on some last minute testing results before posting .24.

 

I did come across an interesting thing while using auto organize 2.

It has to do with provider data for TV shows that have been released but are marked as "TBA" in the Provider API.

Auto organize will sort the file but name it "TBA". This seems fine, as they become accessible in Emby, and they can be viewed.

What is interesting is what happens later if you obtain the file again, and the providers have updated their API results.

The file will sort it self with the actual episode name, and not overwrite the "TBA" version in the file system

Which provider are you using? TVDB or TMDB or both? For fall back. I would interigate both to see if one has correct EpName and compare between the 2. 

1 hour ago, chef said:

 

Perhaps records of  TBA results be kept somewhere and delt with during a task when the actual name can be resolved by the provider.

 

 

And if both state TBA - store in a Model List in config and put them in the xml(for persistence) to use for Comparing Emby Id against  source Id and if they are updated in source…. update ItemName, sortName and also fileName.

what do you think?

Edited by Cheesegeezer
Link to comment
Share on other sites

chef

Yeah, it is when there are new episodes that have just aired, and the provider doesn't have the name yet. 

This happened with Book of Boba Fett.

Each episode was obtained prior to all the providers having the actual data. 

They were sorted with the TBA episode name.

"Book of Boba Fett - s01e02 - TBA.mkv"

Thing is, once the episodes are sorted, and the database is cleared, there is no record of those episodes anywhere stating they had "incorrect" file names.

Note: Emby, doest really look at the name part of the file, it is more interested in the season and episode codes (s01e02, etc).

So in Emby, the episodes are named, but the actual file is not.

It still works... Until you obtain the file again in the future. At that point the file is not "Existing" with the actual episode name, it is only existing with the TBA name.

🤔

I suppose we could do a recursive check on the entire library and look for "TBA" specifically.

Or, keep a record of the TBA results.

Either way, we will have to attempt to fix them at the beginning of each task.

It's a small issue, but something that has to be addressed eventually.

Link to comment
Share on other sites

Cheesegeezer
3 minutes ago, chef said:

Yeah, it is when there are new episodes that have just aired, and the provider doesn't have the name yet. 

This happened with Book of Boba Fett.

Each episode was obtained prior to all the providers having the actual data. 

They were sorted with the TBA episode name.

"Book of Boba Fett - s01e02 - TBA.mkv"

Thing is, once the episodes are sorted, and the database is cleared, there is no record of those episodes anywhere stating they had "incorrect" file names.

Note: Emby, doest really look at the name part of the file, it is more interested in the season and episode codes (s01e02, etc).

So in Emby, the episodes are named, but the actual file is not.

It still works... Until you obtain the file again in the future. At that point the file is not "Existing" with the actual episode name, it is only existing with the TBA name.

🤔

I suppose we could do a recursive check on the entire library and look for "TBA" specifically.

Or, keep a record of the TBA results.

Either way, we will have to attempt to fix them at the beginning of each task.

It's a small issue, but something that has to be addressed eventually.

I get ya totally 💯. But I guarantee that user will not like displaying TBA as a title for the episode. So why not call it Episode 1 if it’s TBA but set a bool for false if the name was TBA when scraped. Then you at least know to interrogate the source for correct filename at beginning of each scan

  • Like 1
Link to comment
Share on other sites

chef
2 hours ago, Cheesegeezer said:

I get ya totally 💯. But I guarantee that user will not like displaying TBA as a title for the episode. So why not call it Episode 1 if it’s TBA but set a bool for false if the name was TBA when scraped. Then you at least know to interrogate the source for correct filename at beginning of each scan

I think you are on to something there. A flag on the result item in the file organizer database.

However, the list of "TBA"  items must be separate from the "FileOrganizerResults" list in the database, because that can be cleared. 

It needs to live on, like the Smart Lists do in the database file, so it can be referenced later. (The fileorganizer.db is actually split into two data tables - one for current results, and one for smart lists)

 

Strangely enough emby will name it TBA in the UI if it also is unable to name it at the time of the library scan.

That is definitely an interesting core thing. 

Eventually, (emby must do this during a future library scan) it will correct the name of the item in the library.db.

But, the file name itself will not be altered.

So it lives in the file system as TBA, and the library.db entry points to that file path.

 

 

Link to comment
Share on other sites

chef

I have laid the ground work to handle the TBA issue, by creating a table in the DB called "Smart Path Corrections".

smart_path_corrections.png.2a00f0cc7feecbae361226a50d8aca84.png

 

Now, when an Item returns "TBA" from the providers, it's Destination Path is saved.

The next time the task is run, we'll check to see if the providers have updated their API to have the data we are looking for.

If so, we'll fix the file name.

If not, we'll try again next time.

Simple enough fix for an issue which only happens every once in a while.

 

Link to comment
Share on other sites

rbjtech
12 hours ago, chef said:

I have laid the ground work to handle the TBA issue, by creating a table in the DB called "Smart Path Corrections".

smart_path_corrections.png.2a00f0cc7feecbae361226a50d8aca84.png

 

Now, when an Item returns "TBA" from the providers, it's Destination Path is saved.

The next time the task is run, we'll check to see if the providers have updated their API to have the data we are looking for.

If so, we'll fix the file name.

If not, we'll try again next time.

Simple enough fix for an issue which only happens every once in a while.

 

Smart .. the 'arr's do the same and don't import until it has a non TBA name ... 😉

 

Edited by rbjtech
  • Thanks 1
Link to comment
Share on other sites

kingy444

@chef mate some great work - just went through all your commits and compared to my work from when we were trying to improve the core integration.

Looks like you have ticked off everything i did except for the TBA issue which has been mentioned by @rbjtech above

Thought i would be able to add something here but looks like you have implemented all my stuff already 🤣

 

Been too busy on other dev work and didnt realise how evolved you had this one mate. Well done.

 

Let me know if you want me to look at anything - but looks like you have it all in hand

  • Thanks 1
Link to comment
Share on other sites

chef
40 minutes ago, kingy444 said:

@chef mate some great work - just went through all your commits and compared to my work from when we were trying to improve the core integration.

Looks like you have ticked off everything i did except for the TBA issue which has been mentioned by @rbjtech above

Thought i would be able to add something here but looks like you have implemented all my stuff already 🤣

 

Been too busy on other dev work and didnt realise how evolved you had this one mate. Well done.

 

Let me know if you want me to look at anything - but looks like you have it all in hand

I've been able to add a new "corrections" tab.

This tab allows you go compare the current file name to what it is supposed to be.

If they differ they get added to a list for correction.

Below, stranger things was added prior to auto organize and does not have proper file naming. It appears in the list

corrections_4.thumb.png.6e9f49d8263f64678b46826d4acd34a3.png

File system:

corrections_3.png.aec9c1a3b72bef8f9da172b17cec3aba.png

I select the items with the check box for renaming, and press a button.

Now the file system is correctedcorrections_5.thumb.png.4fb372cc83694ebe7ec4629f6c2be504.png

Best way to handle languages, and TBA, and wrong file names. I think.

 

It also makes sure to name the .nfo, thumb.jpg, and any subtitles.

 

 

Edited by chef
  • Like 1
Link to comment
Share on other sites

kingy444
10 minutes ago, chef said:

Best way to handle languages, and TBA, and wrong file names. I think.

 

Yep i think that works quite well mate - add an option to auto correct too (off by default)

 

This was another thing i thought of doing but hadnt got my head around the architecture enough to implement at the time - again well done mate.

At this stage it looks like you have the code in hand - perhaps the only thing i can offer is the logo i submitted to core that was never published🤣

I see you have one, but i quite like how mine aligned with the other core plugins.

https://github.com/kingy444/Emby.AutoOrganize/blob/master/Emby.AutoOrganize/thumb.jpg

 

Any reason you didnt write a one time 'blow away the old db if you have v1 database' ?

Link to comment
Share on other sites

Guest phoenixexcinere

Hey,

I tried the beta today, but it doesn't work with each of my series. Here is an example from logfile

2022-06-12 19:38:24.270 Info App: Provider results for X Factor Das Unfassbare has 0 results
2022-06-12 19:38:24.270 Warn App: Unable to determine series from /mnt/user/Daten/downloads/X-Factor/X-Factor Serie/X-Factor - Das Unfassbare/Staffel 2/X-Factor Das Unfassbare S02E13 HDTV 720p x264 - iND.mkv
2022-06-12 19:38:24.641 Info App: Extracted information from /mnt/user/Daten/downloads/X-Factor/X-Factor Serie/X-Factor - Das Unfassbare/Staffel 1/X-Factor Das Unfassbare S01E03 HDTV 720p x264 - iND.mkv. Series name X Factor Das Unfassbare, : Season 1,  Episode 3.
2022-06-12 19:38:24.642 Info App: Parsing Library for matching Series: X Factor Das Unfassbare
2022-06-12 19:38:24.649 Info App: Parsing Smart List Info to find series name: X Factor Das Unfassbare
2022-06-12 19:38:24.650 Info HttpClient: GET https://api4.thetvdb.com/v4/search?type=series&q=X Factor Das Unfassbare
2022-06-12 19:38:24.745 Info Server: http/1.1 GET http://192.168.178.40:8096/emby/videos/36731/hls1/main/94.ts?PlaySessionId=1a23c1f8eefd4c33afd150ec703f1737. Accept=*/*, Connection=keep-alive, Host=192.168.178.40:8096, User-Agent=Mozilla/5.0 (X11; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0, Accept-Encoding=gzip, deflate, Accept-Language=de,en-US;q=0.7,en;q=0.3, Referer=http://192.168.178.40:8096/web/index.html
2022-06-12 19:38:24.761 Info Server: http/1.1 Response 200 to host1. Time: 17ms. http://192.168.178.40:8096/emby/videos/36731/hls1/main/94.ts?PlaySessionId=1a23c1f8eefd4c33afd150ec703f1737
2022-06-12 19:38:24.800 Error ProviderManager: Error searching
	*** Error Report ***
	Version: 4.7.2.0
	Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
	Operating system: Linux version 5.15.43-Unraid (root@Develop) (gcc (GCC) 11.2.0, GNU ld version 2.37-slack15) #1 SMP Wed May 25 11:54:05 PDT 2022
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: system/System.Private.CoreLib.dll
	Processor count: 4
	Data path: /config
	Application path: /system
	MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: TooManyRequests
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
	   at Tvdb.EntryPoint.GetResponse(HttpRequestOptions request, String method)
	   at Tvdb.TvdbBaseProvider`2.FindItemInternal(String searchType, String name, String language, Nullable`1 year, CancellationToken cancellationToken)
	   at Tvdb.TvdbBaseProvider`2.FindItem(TLookupInfoType searchInfo, String searchType, IDirectoryService directoryService, CancellationToken cancellationToken)
	   at Tvdb.TvdbSeriesProvider.GetSearchResults(SeriesInfo searchInfo, CancellationToken cancellationToken)
	   at Emby.Providers.Manager.ProviderManager.GetSearchResults[TLookupType](IRemoteSearchProvider`1 provider, TLookupType searchInfo, CancellationToken cancellationToken)
	   at Emby.Providers.Manager.ProviderManager.GetRemoteSearchResults[TItemType,TLookupType](RemoteSearchQuery`1 searchInfo, BaseItem referenceItem, CancellationToken cancellationToken)
	Source: Emby.Server.Implementations
	TargetSite: Void MoveNext()

 

Edited by phoenixexcinere
Link to comment
Share on other sites

chef
3 hours ago, kingy444 said:

Yep i think that works quite well mate - add an option to auto correct too (off by default)

 

This was another thing i thought of doing but hadnt got my head around the architecture enough to implement at the time - again well done mate.

At this stage it looks like you have the code in hand - perhaps the only thing i can offer is the logo i submitted to core that was never published🤣

I see you have one, but i quite like how mine aligned with the other core plugins.

https://github.com/kingy444/Emby.AutoOrganize/blob/master/Emby.AutoOrganize/thumb.jpg

 

Any reason you didnt write a one time 'blow away the old db if you have v1 database' ?

Ohhh... I hadn't thought of the db thing.

I guess you would just the file version?

Link to comment
Share on other sites

chef
1 hour ago, phoenixexcinere said:

Hey,

I tried the beta today, but it doesn't work with each of my series. Here is an example from logfile

2022-06-12 19:38:24.270 Info App: Provider results for X Factor Das Unfassbare has 0 results
2022-06-12 19:38:24.270 Warn App: Unable to determine series from /mnt/user/Daten/downloads/X-Factor/X-Factor Serie/X-Factor - Das Unfassbare/Staffel 2/X-Factor Das Unfassbare S02E13 HDTV 720p x264 - iND.mkv
2022-06-12 19:38:24.641 Info App: Extracted information from /mnt/user/Daten/downloads/X-Factor/X-Factor Serie/X-Factor - Das Unfassbare/Staffel 1/X-Factor Das Unfassbare S01E03 HDTV 720p x264 - iND.mkv. Series name X Factor Das Unfassbare, : Season 1,  Episode 3.
2022-06-12 19:38:24.642 Info App: Parsing Library for matching Series: X Factor Das Unfassbare
2022-06-12 19:38:24.649 Info App: Parsing Smart List Info to find series name: X Factor Das Unfassbare
2022-06-12 19:38:24.650 Info HttpClient: GET https://api4.thetvdb.com/v4/search?type=series&q=X Factor Das Unfassbare
2022-06-12 19:38:24.745 Info Server: http/1.1 GET http://192.168.178.40:8096/emby/videos/36731/hls1/main/94.ts?PlaySessionId=1a23c1f8eefd4c33afd150ec703f1737. Accept=*/*, Connection=keep-alive, Host=192.168.178.40:8096, User-Agent=Mozilla/5.0 (X11; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0, Accept-Encoding=gzip, deflate, Accept-Language=de,en-US;q=0.7,en;q=0.3, Referer=http://192.168.178.40:8096/web/index.html
2022-06-12 19:38:24.761 Info Server: http/1.1 Response 200 to host1. Time: 17ms. http://192.168.178.40:8096/emby/videos/36731/hls1/main/94.ts?PlaySessionId=1a23c1f8eefd4c33afd150ec703f1737
2022-06-12 19:38:24.800 Error ProviderManager: Error searching
	*** Error Report ***
	Version: 4.7.2.0
	Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
	Operating system: Linux version 5.15.43-Unraid (root@Develop) (gcc (GCC) 11.2.0, GNU ld version 2.37-slack15) #1 SMP Wed May 25 11:54:05 PDT 2022
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: system/System.Private.CoreLib.dll
	Processor count: 4
	Data path: /config
	Application path: /system
	MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: TooManyRequests
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
	   at Tvdb.EntryPoint.GetResponse(HttpRequestOptions request, String method)
	   at Tvdb.TvdbBaseProvider`2.FindItemInternal(String searchType, String name, String language, Nullable`1 year, CancellationToken cancellationToken)
	   at Tvdb.TvdbBaseProvider`2.FindItem(TLookupInfoType searchInfo, String searchType, IDirectoryService directoryService, CancellationToken cancellationToken)
	   at Tvdb.TvdbSeriesProvider.GetSearchResults(SeriesInfo searchInfo, CancellationToken cancellationToken)
	   at Emby.Providers.Manager.ProviderManager.GetSearchResults[TLookupType](IRemoteSearchProvider`1 provider, TLookupType searchInfo, CancellationToken cancellationToken)
	   at Emby.Providers.Manager.ProviderManager.GetRemoteSearchResults[TItemType,TLookupType](RemoteSearchQuery`1 searchInfo, BaseItem referenceItem, CancellationToken cancellationToken)
	Source: Emby.Server.Implementations
	TargetSite: Void MoveNext()

 

Yes, we believe that this has to do with the providers, but we are currently looking into it.

Many thanks!

Link to comment
Share on other sites

kingy444
4 hours ago, chef said:

Yes, we believe that this has to do with the providers, but we are currently looking into it.

The original plug-in was returning the same issue so i don’t think it’s your code

I am curious with the apiv4 for tvdb did emby get a commercial token ? Hadn’t seen anything  much about v4 but perhaps it’s related to that?

  • Thanks 1
Link to comment
Share on other sites

26 minutes ago, kingy444 said:

The original plug-in was returning the same issue so i don’t think it’s your code

I am curious with the apiv4 for tvdb did emby get a commercial token ? Hadn’t seen anything  much about v4 but perhaps it’s related to that?

Yes we did.

  • Like 2
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...