Massive trans-coding Logfiles

[thekingswolf 48]

My trans-coding temp folder is not on the C drive.  Just over the past few days or so, I've noticed my C drive filling up and it's these massive log files.  Please advise.

embyserver.txt

Edited September 15, 2020 by thekingswolf

[Carlo 4560]

I don't think I've ever seen log files that big from Emby before.

I'd really like to take a look at one of those files but wouldn't want to ask you to upload one.

Any chance you could keep one and delete the other to free up space then allow me to do a remote in via TeamViewer and take a look?  I'd like to scroll through one of them to see if it's live TV or something maybe stuck in a loop causing such massive log file.

Let me know if this is something we can do.

Carlo

[thekingswolf 48]

sorry I already deleted them as they cause problems.  once my c drive fills up, the server becomes unwatchable from the buffering.  this is the 2nd time in several days i've had to do this tho so i'm sure they will just build back up again.  

[Luke 42077]

Yes let's look at an example. Thanks.

[Happy2Play 9780]

This issue is all logging goes to the default location.  So setting a transcode-temp location only applies the media filed being converted and nothing to do with the logs that only reside in the Logs folder.  I know there have been requests to have logs path be customizable and definitely should be if logs are getting that size.  

But in reality that does have to be continuous Live TV playback log, but not seeing that log referenced in the provided server log.

[thekingswolf 48]

even if I could set a custom location for the log files, having a 300gig logfile would still be excessive in my opinion.  Once it builds back up again i can try to post the log, but again that is a huge file to attempt to post.  even opening the file on the local machine would take a long time

Edited September 16, 2020 by thekingswolf

[Happy2Play 9780]

10 minutes ago, thekingswolf said:

even if I could set a custom location for the log files, having a 300gig logfile would still be excessive in my opinion.  Once it builds back up again i can try to post the log, but again that is a huge file to attempt to post.  even opening the file on the local machine would take a long time

If you can catch a smaller version of the file it could help, but it would suggest a run away live stream.  Is live tv left running for a long period of time?

[thekingswolf 48]

not that I know of.  I have a few users that use the LiveTV section but it's only a few users.  I have also completely restarted the machine running the server and the log file continues to build.  I also note that the Date Modified time stamp is almost a full 12 hours old despite still being written to.

[Happy2Play 9780]

To my knowledge the only way for one ffmpeg file to continue to grow like that is for it to be continuous playback session, but a dev would have to comment more.

[Carlo 4560]

5 hours ago, thekingswolf said:

sorry I already deleted them as they cause problems.  once my c drive fills up, the server becomes unwatchable from the buffering.  this is the 2nd time in several days i've had to do this tho so i'm sure they will just build back up again.  

 

2 hours ago, thekingswolf said:

even if I could set a custom location for the log files, having a 300gig logfile would still be excessive in my opinion.  Once it builds back up again i can try to post the log, but again that is a huge file to attempt to post.  even opening the file on the local machine would take a long time

Hi, Except for the wee hours of the night when I catch some Zzzs I can be available to remote in and take a look as you won't likely be able to post the files.  So if you see this happening again it would be great if you could PM me so we could do a remote session to gather some info on what is in those logs.

My initial thoughts are the same as Happy2Play that something is watching lots of continuous Live TV or similar.  But typically we'll see the transcode file, not log get this big!

[jaseinny 5]

I too am suffering from massive log files to the point it totally filled up my 1TB cache drive on my Unraid server which made it completely inoperable. 

It is a ffmpeg-transcode text file. I have kept the latest 32GB log file if anybody wants to take a look. Some of the other logs from when it crashed my server were way above this 32GB file!! 

 

Edited January 5, 2021 by jaseinny

[Carlo 4560]

Hi, By any chance do you have DEBUGGING turned on in Emby?  If so turn that off.

[Q-Droid 989]

4 hours ago, jaseinny said:

I too am suffering from massive log files to the point it totally filled up my 1TB cache drive on my Unraid server which made it completely inoperable. 

It is a ffmpeg-transcode text file. I have kept the latest 32GB log file if anybody wants to take a look. Some of the other logs from when it crashed my server were way above this 32GB file!! 

 

Can you run from shell:

head -n1000 logfilename > tempname.txt

and attach tempname.txt?

[jaseinny 5]

1 hour ago, cayars said:

Hi, By any chance do you have DEBUGGING turned on in Emby?  If so turn that off.

Debugging is off. 

[jaseinny 5]

49 minutes ago, Q-Droid said:

Can you run from shell:

head -n1000 logfilename > tempname.txt

and attach tempname.txt?

I tried to run console on the Emby app in my docker container on Unraid and I can't get past "OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown"

Sorry, I don't know how to run this command from the main shell in Unraid. Any tips would be great. 

[Q-Droid 989]

Are the logs on persistent storage accessible outside of the container? I haven't worked on unraid. I'm only guessing those are external to the docker image on host storage. 

 

[jaseinny 5]

The logs are in what's called the 'App Data' folder. The dockers all point to that. So yes, the log's are outside the docker container itself. 

[jaseinny 5]

4 minutes ago, jaseinny said:

The logs are in what's called the 'App Data' folder. The dockers all point to that. So yes, the log's are outside the docker container itself. 

 

[Q-Droid 989]

Does unraid itself have a console/shell available? If so then login to unraid shell and navigate to the appdata location for emby, whether it's /mnt/user/appdata/emby/logs or /media/user/appdata/emby/logs. I don't see the emby config in the first image.

You should be able to run the head command from there.

 

[jaseinny 5]

On 1/5/2021 at 4:52 PM, Q-Droid said:

Does unraid itself have a console/shell available? If so then login to unraid shell and navigate to the appdata location for emby, whether it's /mnt/user/appdata/emby/logs or /media/user/appdata/emby/logs. I don't see the emby config in the first image.

You should be able to run the head command from there.

 

Hi Q-Droid, I ran the command

'/mnt/user/appdata/emby/logs# head -n1000 head ffmpeg-transcode-53457878-d9f8-42be-acd9-057bb23a1941_1.txt > tempname.txt
head: cannot open 'head' for reading: No such file or directory'

However I did find this newly created temp file. 

Thanks.

tempname.txt

Also, the log files are growing at around 100MB per 10 secs. 

Edited January 8, 2021 by jaseinny
Added attachments

[Q-Droid 989]

You typed head twice but that's OK, you got the file. 

 

[Q-Droid 989]

Your file should help the devs make sense of what's going on. It looks like it's moving along fine then it starts to throw decoding errors.

11:35:34.595 frame=107542 fps=371 q=28.0 size=  587264kB time=00:29:54.52 bitrate=2684.0kbits/s throttle=off speed=6.18x
11:35:35.096 frame=107726 fps=371 q=29.0 size=  588032kB time=00:29:57.59 bitrate=2682.9kbits/s throttle=off speed=6.18x
11:35:35.596 frame=107907 fps=371 q=28.0 size=  589312kB time=00:30:00.62 bitrate=2684.2kbits/s throttle=off speed=6.18x
11:35:35.995 [mpeg2_cuvid @ 0x699a40] AVHWFramesContext is already initialized with incompatible parameters
11:35:35.995 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:35.995 Error while decoding stream #0:0: Generic error in an external library
11:35:35.998 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:35.998 Error while decoding stream #0:0: Generic error in an external library
11:35:36.003 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:36.003 Error while decoding stream #0:0: Generic error in an external library
11:35:36.012 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:36.012 Error while decoding stream #0:0: Generic error in an external library
 

[jaseinny 5]

4 minutes ago, Q-Droid said:

Your file should help the devs make sense of what's going on. It looks like it's moving along fine then it starts to throw decoding errors.

11:35:34.595 frame=107542 fps=371 q=28.0 size=  587264kB time=00:29:54.52 bitrate=2684.0kbits/s throttle=off speed=6.18x
11:35:35.096 frame=107726 fps=371 q=29.0 size=  588032kB time=00:29:57.59 bitrate=2682.9kbits/s throttle=off speed=6.18x
11:35:35.596 frame=107907 fps=371 q=28.0 size=  589312kB time=00:30:00.62 bitrate=2684.2kbits/s throttle=off speed=6.18x
11:35:35.995 [mpeg2_cuvid @ 0x699a40] AVHWFramesContext is already initialized with incompatible parameters
11:35:35.995 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:35.995 Error while decoding stream #0:0: Generic error in an external library
11:35:35.998 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:35.998 Error while decoding stream #0:0: Generic error in an external library
11:35:36.003 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:36.003 Error while decoding stream #0:0: Generic error in an external library
11:35:36.012 [mpeg2_cuvid @ 0x699a40] ctx->cvdl->cuvidParseVideoData(ctx->cuparser, &cupkt) failed -> CUDA_ERROR_UNKNOWN: unknown error
11:35:36.012 Error while decoding stream #0:0: Generic error in an external library
 

Thanks for looking into this. I'm basically converting video clips from .ts to hevc if that points to anything. 

[jaseinny 5]

Any ideas?

[Happy2Play 9780]

This snippet to me shows error loop in the file being converted.  I assume this loop happens until the end of the conversion.  Are the files actually converting properly?

Does the error always happen after 30 minutes?

But @Luke or @softworkz will have to comment further.