This video seems to bomb the Library scanner out (version 3.6.0.63)

[scb99 187]

Wherever I put this file, the whole of the rest of the library is not scanned. 

 

The file is an mkv, I renamed it to .log and cut it down to the first 80MB so that I could upload it.

 

Here is what the log says, I assume that "exited with code 1" is bad!

 

2018-11-12 10:46:00.404 Info App: ProcessRun 'StreamTranscode 2a0be8' Execute: /mnt/HD/HD_a2/Nas_Prog/emby-server/bin/ffmpeg -noaccurate_seek -f matroska -i file:"/mnt/HD/HD_a2/Public/Shared Videos/Who1/Season 03/019 - Mission to the Unknown/E01 - Mission to the Unknown (recon).mkv" -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 copy -bsf:v h264_mp4toannexb -copyts -vsync -1 -codec:a:0 copy -copypriorss:a:0 0 -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 6  -individual_header_trailer 0 -break_non_keyframes 1 -segment_format mpegts -segment_list_type m3u8 -segment_start_number 0 -segment_list "/mnt/HD/HD_a2/emby/transcoding-temp/969e396b8111bce3e4608db15a63ec9b.m3u8" -y "/mnt/HD/HD_a2/emby/transcoding-temp/969e396b8111bce3e4608db15a63ec9b%d.ts"
2018-11-12 10:46:00.426 Info App: ProcessRun 'StreamTranscode 2a0be8' Started.
2018-11-12 10:46:00.527 Info App: ProcessRun 'StreamTranscode 2a0be8' Process exited with code 1

 

Also following this error there is this:

2018-11-12 10:46:10.956 Error HttpServer: Error processing request
 *** Error Report ***
 Version: 3.6.0.60
 Command line: /mnt/HD/HD_a2/Nas_Prog/emby-server/system/EmbyServer.dll -programdata /mnt/HD/HD_a2/emby -ffmpeg /mnt/HD/HD_a2/Nas_Prog/emby-server/bin/ffmpeg -ffprobe /mnt/HD/HD_a2/Nas_Prog/emby-server/bin/ffprobe -updatepackage emby-server-emby-server-western_digital_pr4100_{version}_x86_64.bin -nolocalportconfig -noautorunwebapp
 Operating system: Unix 4.1.13.0
 64-Bit OS: True
 64-Bit Process: True
 User Interactive: True
 Processor count: 4
 Program data path: /mnt/HD/HD_a2/emby
 Application directory: /mnt/HD/HD_a2/Nas_Prog/emby-server/system
 System.Exception: System.Exception: Error starting ffmpeg
    at Emby.Server.MediaEncoding.Api.BaseStreamingService.StartFfMpeg(StreamState state, String outputPath, EncodingOptions encodingOptions, CancellationTokenSource cancellationTokenSource, Boolean acquireResources)
    at Emby.Server.MediaEncoding.Api.Hls.DynamicHlsService.GetDynamicSegment(StreamRequest request, String segmentId)
    at Emby.Server.Implementations.Services.ServiceExecGeneral.GetTaskResult(Task task)
    at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, ILogger logger, String operationName, CancellationToken cancellationToken)
    at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, String urlString, String host, String localPath, CancellationToken cancellationToken)
 Source: Emby.Server.MediaEncoding
 TargetSite: Void MoveNext()
    at Emby.Server.MediaEncoding.Api.BaseStreamingService.StartFfMpeg(StreamState state, String outputPath, EncodingOptions encodingOptions, CancellationTokenSource cancellationTokenSource, Boolean acquireResources)
    at Emby.Server.MediaEncoding.Api.Hls.DynamicHlsService.GetDynamicSegment(StreamRequest request, String segmentId)
    at Emby.Server.Implementations.Services.ServiceExecGeneral.GetTaskResult(Task task)
    at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, ILogger logger, String operationName, CancellationToken cancellationToken)
    at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, String urlString, String host, String localPath, CancellationToken cancellationToken)

 

Finally I notice although I am running .63, in the logs and on the dashboard it still says .60, should I worry?

 

E01 - Mission to the Unknown (80MB).log

[Luke 37010]

Please install the latest version, and then report back. Thanks.

[scb99 187]

Hi Luke

 

I think I do have the latest version -  3.6.0.63

 

I did mention above "Finally I notice although I am running .63, in the logs and on the dashboard it still says .60, should I worry?"

 

Just to be sure I Uninstalled and Reinstalled. On the MyCloud Apps page it has

 

pp Name:  

Emby Server

Version:  

3.6.0.63

Installed on:  

Monday, 2018 November 12, 9:02:41 PM

 

But still says .60 on the dashboard

Edited November 12, 2018 by scb99

[Luke 37010]

The log indicates 3.6.0.60. Can you try installing the latest again? thanks.

[scb99 187]

I did already, twice!

[Luke 37010]

Ok, please try with the next beta update. Thanks.

[scb99 187]

OK it was a battle but I finally have 3.6.0.64 showing in the dashboard / logs

 

So: I still have a problem with that file but the log is completely different

So to try to break this down into elements.

 

I copied the problem file into its own library, so here first is the log from that. It seems like when I do a library scan, it creates a transient xml file maybe? But this file is not created. 

 

2018-11-13 18:16:58.393 Debug MediaEncoder: Ffprobe -i file:"/mnt/HD/HD_a2/Public/Shared Videos/Backup/E01 - Mission to the Unknown (reconx).mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2018-11-13 18:16:58.393 Info MediaEncoder: ProcessRun 'ffprobe' Execute: /mnt/HD/HD_a2/Nas_Prog/emby-server/bin/ffprobe -i file:"/mnt/HD/HD_a2/Public/Shared Videos/Backup/E01 - Mission to the Unknown (reconx).mkv" -threads 0 -v info -print_format json -show_streams -show_chapters -show_format -show_data
2018-11-13 18:16:58.411 Info MediaEncoder: ProcessRun 'ffprobe' Started.
2018-11-13 18:16:58.529 Info MediaEncoder: ProcessRun 'ffprobe' Process exited with code 0
2018-11-13 18:16:58.535 Error App: Error parsing /mnt/HD/HD_a2/Public/Shared Videos/Backup/E01 - Mission to the Unknown (reconx).xml
*** Error Report ***
Version: 3.6.0.64
Command line: /mnt/HD/HD_a2/Nas_Prog/emby-server/system/EmbyServer.dll -programdata /mnt/HD/HD_a2/emby -ffmpeg /mnt/HD/HD_a2/Nas_Prog/emby-server/bin/ffmpeg -ffprobe /mnt/HD/HD_a2/Nas_Prog/emby-server/bin/ffprobe -updatepackage emby-server-emby-server-western_digital_pr4100_{version}_x86_64.bin -nolocalportconfig
Operating system: Unix 4.1.13.0
64-Bit OS: True
64-Bit Process: True
User Interactive: True
Processor count: 4
Program data path: /mnt/HD/HD_a2/emby
Application directory: /mnt/HD/HD_a2/Nas_Prog/emby-server/system
System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not find file '/mnt/HD/HD_a2/Public/Shared Videos/Backup/E01 - Mission to the Unknown (reconx).xml'.
File name: '/mnt/HD/HD_a2/Public/Shared Videos/Backup/E01 - Mission to the Unknown (reconx).xml'
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(String path)
   at Emby.Server.Implementations.IO.ManagedFileSystem.OpenRead(String path)
   at XmlMetadata.Parsers.BaseItemXmlParser`1.Fetch(MetadataResult`1 item, String metadataFile, XmlReaderSettings settings, Encoding encoding, CancellationToken cancellationToken)
   at XmlMetadata.Parsers.BaseItemXmlParser`1.Fetch(MetadataResult`1 item, String metadataFile, CancellationToken cancellationToken)
   at XmlMetadata.Providers.MovieXmlProvider.Fetch(MetadataResult`1 result, String path, CancellationToken cancellationToken)
   at XmlMetadata.Providers.BaseXmlProvider`1.GetMetadata(ItemInfo info, IDirectoryService directoryService, CancellationToken cancellationToken)
Source: System.Private.CoreLib
TargetSite: Void ThrowExceptionForIoErrno(ErrorInfo, System.String, Boolean, System.Func`2[interop+ErrorInfo,Interop+ErrorInfo])
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(String path)
   at Emby.Server.Implementations.IO.ManagedFileSystem.OpenRead(String path)
   at XmlMetadata.Parsers.BaseItemXmlParser`1.Fetch(MetadataResult`1 item, String metadataFile, XmlReaderSettings settings, Encoding encoding, CancellationToken cancellationToken)
   at XmlMetadata.Parsers.BaseItemXmlParser`1.Fetch(MetadataResult`1 item, String metadataFile, CancellationToken cancellationToken)
   at XmlMetadata.Providers.MovieXmlProvider.Fetch(MetadataResult`1 result, String path, CancellationToken cancellationToken)
   at XmlMetadata.Providers.BaseXmlProvider`1.GetMetadata(ItemInfo info, IDirectoryService directoryService, CancellationToken cancellationToken)

[Luke 37010]

You installed the xml metadata plugin, so it's looking for xml metadata files.

[scb99 187]

Ahs so that‘s a red herring

I‘ll take this xml thing out tomorrow

The problem is: Whenever I put that file in a folder, all the other contents of the folder ‚disappear’

I‘ll do some more thinking in the morning

Thanks

[Luke 37010]

Please attach the complete emby server log. Also, what are the other contents of the folder? What exactly disappears?

[scb99 187]

Hi Luke

As it turn out I have to be away a week or so, as soon as I am back I will get on this again.

Tx for the support.

[scb99 187]

Ps i’ll try to give you a bit of info while I’m underway

[Luke 37010]

OK great, thanks !

[scb99 187]

I haven't been back to where the server is, and I don't have remote access to the logs. However I have determined that the following pattern gives the Library scanner problems.

(My Libraries are all mixed content)

Library

-contains many directories, let's call them A-Z

- each directory A-Z has multiple subdirectories; ONE of these subdirectories has exactly one media file. Let's say directory C has subdirectories C1-10 and of these subdirectory C9 has only one media file as content. In this situation (at least sometimes) only this one media file in the single-media-file directory C9 shows in the whole directory C and everything else in directory C "disappears". If I move directory C9 out of the particular library and put it somewhere else, all the rest of the content in directory C suddenly shows up.

[Luke 37010]

Thanks for the report. We'll take a look.

[scb99 187]

Thanks Luke. I have two repeatable cases of this, so if you can’t sort it just looking at the code, as soon as I’m back at base (hopefully this weekend) I can submit the thing in a controlled way with logfiles

[Luke 37010]

Have you compared your folder structure to the guidelines on our wiki?

[scb99 187]

I can’t change the folder structure unfortunately, because it’s a backup of another system.

But have looked at the wiki and I don’t think it’s ‘seriously’ out of line.

Also, if it was a structural issue, I don’t then see any obvious reason why when I remove these single-file directories it ‘springs to life’.

[Luke 37010]

Try using the home videos content type.

[scb99 187]

I’ll do that, though it wouldn’t be a solution for me as, just for instance, several of the tv series have missing episodes which are represented by their surviving soundtracks.

I should be back at base tomorrow and have time to make a full report soon.

[scb99 187]

OK I have done a complete repro of the situation (please note, I respond to your above suggestion at the end of all this)

 

First, I have a library Backup with type "Mixed Content" and directory structure as follows (see dir1.png):

- Chessgame - 3 x mkv files

- Knight Errant - 2 x mkv

- Sexton Blake - 1 x avi

- Test

 -- Test1 - 1 x mkv

This library looks like this in Emby: See screen1.png.

I now move the directory "Chessgame" into the Directory test so the structure is now like this (see dir2.png)

 

- Knight Errant - 2 x mkv

- Sexton Blake - 1 x avi

- Test

-- Test1 - 1 x mkv

-- Chessgame - 3 x mkv files

This library looks like this in Emby: See screen2.png. As you can see, the "Chessgame" files have now disappeared as far as Emby is concerned. The directory "Test" is not seen, only "Test1" appears which is seen as a single file (note the lack of a folder logo in the top left corner)

I now move the directory "Knight Errant " into the Directory test so the structure is now like this (see dir3.png)

- Sexton Blake - 1 x avi

- Test

-- Test1 - 1 x mkv

-- Chessgame - 3 x mkv files

-- Knight Errant - 2 x mkv

This library looks like this in Emby: See screen3.png. As you can see, as well as the "Chessgame" files, also the "Knight Errant" files have now disappeared as far as Emby is concerned. The directory "Test" is still not seen, only "Test1" appears

Finally, I move the single-file subdirectory "Test1" up to the root (see dir4.png. so the structure is now like this:

- Sexton Blake - 1 x avi

- Test

-- Chessgame - 3 x mkv files

-- Knight Errant - 2 x mkv

-- Test1 - 1 x mkv

This library now looks like this in Emby: See screen4.png. As you can see, the "Test" folder has appeared, the folder logo on it, and if I click on it the "Chessgame" and "Knight Errant" folders and their contents now appear correctly.

 

So the above is hopefully a clear demonstration of the problem.

 

Finally, I move the content of the backup folder to be like this (see dir5.png)

 

Then I deleted the library Backup and recreated it as a type "TV Shows". When I do this, it sees all the files are seen but it "flattens out" the content to the top directory level (Screen5a.png)

 

Then I finally deleted the library Backup and recreated it as a type "Home Videos", as per your above suggestion. When I do this, it does appear that this gives I would consider to be the "expected" result (Screen5b.png, Screen5c.png) in terms of the structure.

However...

1. It appears I am unable to use IMDB etc to identify the content in this scenario

2. Where soundtracks which still exist are substituted for missing TV episodes, I cannot make this work.

3. Honestly, I am unable to think of the above-demonstrated behaviour as in any way logical, or as anything other than a bug, really.

 

Also attached is the log. I really hope you will be able to help on this. I'm not expecting "perfection" but it would be good to get it "basically working"

 

Cheers, Steve

 

 

 

 

 

  

embyserver.txt

[Luke 37010]

Mixed content is less strict and therefore will never be perfect. For best results right now I would use either home videos, or separate into tv/movie libraries.

[scb99 187]

Hmm. I would argue that this is not an example of "Mixed Content" being "less strict" but of it simply not working correctly!! I mean, however you look at it, content shouldn't simply disappear in such a simple situation (single-file directories).

 

Anyway -

 

The TV Shows type doesn't work well for these situations:

 

1. some TV programs have a mixture of video and audio-only where episodes are lost but off-air recordings (some in cd format) or other soundtracks still exist. 

 

2. Some programs, e.g. Dr Who springs to mind but there are several others, have a structure

Program

- Season

-- Series (1-20 episodes)

 

And the Home Videos type also doesn't work well for audio or allow me to index content to IMDB.

[scb99 187]

In fact the more I think on this, the more I think this is 100% a bug which should be fixed. I'm not expecting "mixed content" to be perfect and other situations where it goes wrong I can understand (for instance if I have both sound material and video material in a single sub-directory it will under certain conditions all disappear I have noticed), but that dropping a single-file directory into a collection of other material causes all the other material to disappear is quite an insidious bug and with no obvious rationale according to the guidelines either.. 

[scb99 187]

Seems to be working much better in 4.1!