Trancoding and Logging Problems

[Skywayman 9]

I have attached both emby and ffmpeg logs.

About 3 days ago this server has become unusable. From what I can gather it looks like no matter the user or content, emby is forcing transcoding. So when a video tries to transcode it creates a tmp log and will play for 3-5 seconds, then create another one, rinse and repeat. I have video transcoding off for each user, and tested on my personal user removing audio transcoding as well. I do also see SQLite errors and DB errors and I'm not too experienced with what any of the errors mean. This server was working fine for about a month before this started the other day. I have also ruled out my content server as I have several other emby servers running off it. To clarify I have Mass storage server with smaller server with storage server mount. Emby runs on the client. I use CSF firewall, have enabled, disabled on all and tried every possible port including UDP. Really hoping theres no kind of corruption of DB but if so I just want to know and move forward.

embyserver.txt ffmpeg-remux-4177d72f-dea3-4962-871f-138c26066c26_1.txt

[Luke 42077]

HI, you've got some I/O failures coming from your media storage device:

2021-04-17 14:52:39.785 Info Server: http/1.1 GET http://xx.xx.xx.xx:8096/emby/videos/47562/stream.mkv?DeviceId=baf27b8a6a85d70f&MediaSourceId=58d957588c6a7eeafe17baaa75be7ce1&Static=true&PlaySessionId=c59a002f68ba4c7b8b61f92ae53ace9a. Connection=Keep-Alive, Accept-Encoding=identity, Host=85.202.80.33:8096, Range=bytes=5747-, User-Agent=Emby/1.8.55g (Linux;Android 9) ExoPlayerLib/2.11.8, Icy-MetaData=1
2021-04-17 14:52:41.700 Error Server: Error processing request
	*** Error Report ***
	Version: 4.5.4.0
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
	Operating system: Linux version 4.15.0-76-generic (buildd@lcy01-amd64-029) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 20
	Framework: .NET Core 3.1.8
	OS/Process: x64/x64
	Runtime: opt/emby-server/system/System.Private.CoreLib.dll
	Processor count: 8
	Data path: /var/lib/emby
	Application path: /opt/emby-server/system
	System.IO.IOException: System.IO.IOException: Input/output error

Perhaps try rebooting your machine and see if those go away.

Edited April 18, 2021 by Happy2Play
Removed external address

[Skywayman 9]

So I have rebooted several time. And now have just loaded fresh on a fresh client server with only emby on it> worked fine for about an hour and now is pushing all those same errors. Could this be some kind of DNS issue? I see "SocketError: Permission Denied" a lot.

[Guest]

This error is actually POSSIBLY connected permissions to access the server. AND it is also occasionally here... What Are the Common Causes of Socket Error 10013? (easytechjunkie.com)

[Q-Droid 989]

You're going to have to provide more info about your setup.

From barely one second into the startup, before server is actually done starting and for the 25 minute duration of the log your server gets HTTP requests from 14 different client IPs 3800 times.

Sooo....what have you got?

Could the DB locks be an indication of other emby server processes running? Do you have multiples on the same host or trying to sync/share between hosts?

 

[Guest]

I would check hardware NAT Filtering logs... I know this seems unlikely unless your getting errors about login.. BUT I would look for points of circumvention within the network and for your system if this not normal behavior as well.. This could be a mobile device, because of the way connections are made via cellular.. and with the use of Tor... this could also show in this manner... 

If your accounts need to be secured, it would be advisable and possibly changing your host name.. usually pretty easy ( unless SSL is involved ). I would however be looking at this very closely, system wide - If you don't know where all of these requests are coming from... Many types of online attacks... using various different methods.. that you would never actually see anything happening in front of you.. Just undesired system operation. And there are ways to mitigate.

The big thing would be that your system does not like it and it is causing a problem. Not to mention what may have happened in the meantime.

[Skywayman 9]

4 hours ago, Q-Droid said:

You're going to have to provide more info about your setup.

From barely one second into the startup, before server is actually done starting and for the 25 minute duration of the log your server gets HTTP requests from 14 different client IPs 3800 times.

Sooo....what have you got?

Could the DB locks be an indication of other emby server processes running? Do you have multiples on the same host or trying to sync/share between hosts?

 

I did explain in the first post but will go more in depth.

There is a large storage server with the content. No emby running on it. Just the content and content grabbing programs. From there I have 4 smaller servers that mount the storage server. I have been setup this way for years. I dont exceed 125 users on any server, although typically I have tested and seen about 175 is the limit. The Host server files folder is 755. The client servers all running emby as user emby, and the host server is owned by the host user account. I have gone across the board and disabled CSF on all machines and rebooted. I have 4 client servers, 3 from one provider, one from another. The single on from DP is having no issues at all. The other 3 are not throwing all these same errors.

Edited April 18, 2021 by Skywayman

[Happy2Play 9780]

Do you still get "Busy: database is locked" after restarting Emby?

[Skywayman 9]

2 minutes ago, Happy2Play said:

Do you still get "Busy: database is locked" after restarting Emby?

Just completing a restart now. Will load the logs. After a fresh start they do not appear to have immediately came. Ignore the "Does not exist." Thats because I unmounted and remounted, and didn't delete the old libraries.

embyserver.txt ffmpeg-transcode-f9014ee9-b6a5-4f3e-9de2-20946de75b1a_1.txt

[Happy2Play 9780]

5 minutes ago, Skywayman said:

Just completing a restart now. Will load the logs. After a fresh start they do not appear to have immediately came. Ignore the "Does not exist." Thats because I unmounted and remounted, and didn't delete the old libraries.

embyserver.txt 107.27 kB · 0 downloads ffmpeg-transcode-f9014ee9-b6a5-4f3e-9de2-20946de75b1a_1.txt 4 kB · 0 downloads

Do you have a max bitrate set?  It may be the fallback on the Auto quality setting.

2021-04-18 13:31:39.766 Info App: Bitrate exceeds DirectPlay limit: media bitrate: 12444906, max bitrate: 1500000
2021-04-18 13:31:39.766 Info App: Bitrate exceeds DirectStream limit: media bitrate: 12444906, max bitrate: 1500000

 

[Skywayman 9]

Just now, Happy2Play said:

Do you have a max bitrate set?  It may be the fallback on the Auto quality setting.

2021-04-18 13:31:39.766 Info App: Bitrate exceeds DirectPlay limit: media bitrate: 12444906, max bitrate: 1500000
2021-04-18 13:31:39.766 Info App: Bitrate exceeds DirectStream limit: media bitrate: 12444906, max bitrate: 1500000

 

No I have no bitrate limit. This particular server was just reloaded yesterday so it's still all defaults. Users have video transcoding off (audio on).

[Skywayman 9]

One a FRESH install, just scanned in. NO users except me. I guess I've never looked too hard at ffmpeg logs but when I  just played the first 30 seconds of a movie, am I mistaken to think this is printing way too many logs? Looks like 13 tmp logs for 30 seconds of playback. that seems off to me.

 

ffmpeg-transcode-5e8a26dd-d085-4817-ae7e-07cc150dd140_1.txt

[Q-Droid 989]

Are you talking about the .ts files? It's clearer now. Those are segments created while transcoding for streaming, not "logs". It is normal.

Regarding the "client" servers, are they each a local host installation and configuration of emby and the only shared storage is the media? 

 

[Skywayman 9]

3 minutes ago, Q-Droid said:

Are you talking about the .ts files? It's clearer now. Those are segments created while transcoding for streaming, not "logs". It is normal.

Regarding the "client" servers, are they each a local host installation and configuration of emby and the only shared storage is the media? 

 

Correct. And just for s&g I just installed emby on my storage server and changed ownership across all to emby:emby. Made no difference. Really banging my head here. 

[Q-Droid 989]

Transcoding is dictated by the clients, not the server. You can create restrictions but if the client can't direct play/stream, for whatever reason, the server will either transcode or not stream.

The DB errors are another matter. Locking errors could be too much activity at the time. I/O errors could be a system resource issue.

 

[Happy2Play 9780]

And what is the client playback quality set to?   I am guessing Auto since it is falling back to 1.5Mbps.  And if you apply a set quality does it direct play?

VideoBitrate=1116000&AudioBitrate=384000

TranscodeReasons=ContainerBitrateExceedsLimit

 

Edited April 18, 2021 by Happy2Play

[Skywayman 9]

9 hours ago, Happy2Play said:

And what is the client playback quality set to?   I am guessing Auto since it is falling back to 1.5Mbps.  And if you apply a set quality does it direct play?

VideoBitrate=1116000&AudioBitrate=384000

TranscodeReasons=ContainerBitrateExceedsLimit

 

Yes set to auto. I haven't changed any default settings. I'm gonna try ordering a server from another provider and seeing if the same issue arises.

[Luke 42077]

Try setting your own quality value if you're not satisfied with what auto is choosing.

[Skywayman 9]

Ok. So I got a new server from different provider and ran the same install as usual (Emby, CSF, mount to storage). Having NO playback issues whatsoever. So comparing logs I see one particular thing that is happening one the server failing vs the one not failing. I do see maxbitrate exceeded on both, but one handles it and proceeds. The bad server stops trying. I did mess with max bitrate on the user and get the same result every time. It starts writing in transcoding-temp, hits a snag, then forces itself to quit. I raised the max on system to 150mb and user to 50mb then 150 mb and am getting the same error.  All playback ends with an error regarding transcoding failing. I cant wrap my head around why 3 different server having the issue and 3 different servers not having it.

[Luke 42077]

I think you're misunderstanding the server side bitrate limits. They are only upper bound limits. They do not guarantee you'll get direct play as opposed to transcoding.

I would check and quality settings in the app that you're playing from as previously mentioned.

[Skywayman 9]

I understand I won’t get direct play. I am trying to play the same movie hosted on the same server from 2 different emby servers. Same settings. One works, one doesn’t. 

[Luke 42077]

Hi there @Skywayman are you still having an issue with this?

[Skywayman 9]

I actually ended up cancelling all 3 of the servers. Continue having no issues with servers from other providers. Thanks for checking in. Was not able to resolve. 

[Luke 42077]

@Skywayman we're happy to work with you and help get you squared away if you'd like to try the new 4.6 release. Thanks.