Debugging LiveTV edge case

[GWTPqZp6b 41]

I have a IPTV provider whose streams work fine in VLC all the time, and most of the time in my xteve & Emby configuration. 

When watching the stream in Emby, its is transcoded in a Transcode folder that is stored on a network share which fails after ~30 seconds with the following error. I've tried CIFS and NFS, some issue both ways. I dont have this issue if I leave the Transcode folder on the default location but this isnt possible due to limited capacity. 

2022-04-16 18:19:27.328 Error Server: Error processing request
	*** Error Report ***
	Version: 4.7.0.35
	Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
	Operating system: Linux version 5.17.1-arch1-1 (linux@archlinux) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT Mon, 28 Mar 2022 20:55:33 +0000
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: system/System.Private.CoreLib.dll
	Processor count: 6
	Data path: /config
	Application path: /system
	System.IO.IOException: System.IO.IOException: Input/output error : '/mnt/Transcode/transcoding-temp/d4eec0204e3846c29123297bb6a89b35.ts'
	   at System.IO.RandomAccess.ReadAtOffset(SafeFileHandle handle, Span`1 buffer, Int64 fileOffset)
	   at Microsoft.Win32.SafeHandles.SafeFileHandle.ThreadPoolValueTaskSource.ExecuteInternal()
	--- End of stack trace from previous location ---
	   at Microsoft.Win32.SafeHandles.SafeFileHandle.ThreadPoolValueTaskSource.GetResult(Int16 token)
	   at Microsoft.Win32.SafeHandles.SafeFileHandle.ThreadPoolValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
	   at Emby.Server.Implementations.IO.StreamHelper.CopyToAsync(Stream source, PipeWriter destination, Int32 bufferSize, Int32 emptyReadLimit, CancellationToken cancellationToken)
	   at Emby.LiveTV.TunerHosts.LiveStream.CopyFile(String path, Boolean seekFile, Int32 emptyReadLimit, PipeWriter writer, IFileSystem fileSystem, IStreamHelper streamHelper, CancellationToken cancellationToken)
	   at Emby.LiveTV.TunerHosts.LiveStream.CopyToAsync(PipeWriter writer, CancellationToken cancellationToken)
	   at Emby.Server.MediaEncoding.Api.Progressive.ProgressiveFileCopier.WriteToAsync(IResponse response, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
	Source: System.Private.CoreLib
	TargetSite: Int32 ReadAtOffset(Microsoft.Win32.SafeHandles.SafeFileHandle, System.Span`1[System.Byte], Int64)

I seem to recall once upon time seeing an option to disable LiveTV transcoding but cant find it anymore. Is it still available in latest beta (7.4.0.35)?

I'm assuming its some kind of lock or throughput issue, any help to debug would be appreciated

Full embyserver.txt log & hardware_detection

 

[Luke 37065]

Is there some other folder you can use for transcoding temp files?

[GWTPqZp6b 41]

Not on the SSD that the docker containers run on sadly. And the only other storage available to the docker host is via mounts. 
I think there may be some odd linuxy/systemd race condition as if I faff about and restart the mounts/emby a few times it often goes away, but this isnt really a pro-level solve. I'm a bit perplexed to be honest, but Ive missed one too many soccer games now so happy to spend some time debugging. All my movie transcoding works fine so its something related to LiveTV streams/.ts files etc. 
 

[Luke 37065]

Live TV has double duty because it has to first stream the original source to disk, then the transcode based off of that, so I think that probably explains it.

[GWTPqZp6b 41]

you're probably right... I'll do some testing. Thanks for the input. 

I take it it isnt possible to avoid transcoding for live broadcasts?

[Luke 37065]

Even if it's not doing a full transcode it's still going to have to repackage into supported container, so that won't reduce disk access .

What would reduce it is to eliminate the stream to local disk, but then you lose the stream sharing feature and may run into issues with your provider over number of simultaneous connections.

[GWTPqZp6b 41]

Thanks for sharing your thoughts Luke.

I have xteve set to pass through to Emby, and also tried having Emby connect to xteve but neither resolved the issue. The transcode folder is on a pair of mirrored HGST 8TB SS530 disks so should have plenty of IOPS & throughput but I suspect it may be something with the underlying networking and virtualisation limiting me. 

2022/04/16 20:25:21 [xTeVe] Channel Name:           UK: xxx SPORTS xx FHD
2022/04/16 20:25:21 [xTeVe] Client User-Agent:      VLC/3.0.0-git LibVLC/3.0.0-git
2022/04/16 20:25:21 [xTeVe] Streaming URL:          http://xxxxxxxx.xxxxx:80/xxxxxxxxxxxx/xxxxxxxxxxx/xxx
2022/04/16 20:25:21 [xTeVe] Streaming Info:         URL was passed to the client.
2022/04/16 20:25:21 [xTeVe] Streaming Info:         xTeVe is no longer involved, the client connects directly to the streaming server.

 

[BillOatman 500]

I was going to ask if you had xteve set to buffer but I see you do not.
You might want to consider not using xteve at all as it is not needed for Emby like it is for Plex.  And curate your m3u channels via other methods.  The one I use is currently closed to new users until next release, but there are other ways mentioned in this forum.  

[GWTPqZp6b 41]

This is the scenario I'm trying to avoid, my system disk becoming filled with transcoding-temp files when someone leaves LiveTV streaming.

@BillOatmanCould you drop me a PM with the service you use please. I've tried a couple but still looking for my final solution. 

[Luke 37065]

What do you mean when someone leaves live TV? When nobody is requesting a channel anymore it will stop.

[GWTPqZp6b 41]

I mean if you start a LiveTV channel playing and leave it playing, the transcoding-temp folder continues to fill with temporary files and I at some point when the disk is full services fail. 

[Luke 37065]

Ok right. That is the live TV buffer that is explained earlier.