Jump to content

(solved) Emby Server on Freenas Jail stalls when skipping


SpaceCabbi
 Share

Recommended Posts

SpaceCabbi

So new but happy user of emby now today something odd. When playing a movie and skipping the intro it stalled, 

As in not responding complete hookup it wil take a minute orso and i can get back onto the server.
Now interestingly enough there is no error none nada 0.

It look simulair to: https://emby.media/community/index.php?/topic/72236-emby-stops-during-playback/

So logs included although i doubt they are helpful it does seem to be related to remux ? Most plays are directstream.

embyserver.txt ffmpeg-remux-2baf661c-36e8-4ae1-b61a-a4b79208d1d2_1.txt

Edited by SpaceCabbi
Link to comment
Share on other sites

SpaceCabbi
1 hour ago, Luke said:

Hi, what do you have for a cpu?

CPU: Intel - Xeon E3-1230 V6 3.5GHz Quad-Core Processor
Motherboard: Supermicro - MBD-X11SSM-F Micro ATX LGA1151 Motherboard
Memory: Samsung - 32GB (2 x 16GB) DDR4-2133 Memory
Controllers: IBM-ServeRAID-M1015 SAS HBA (LSI9220-8i) flashed to IT mode
Chassis: HPE 24 Bay 3Gb SAS Expander in a EMC Avamar Gen4s DRBGP
Storage: 12x Hitachi HGST ultrastar 4tb Raidz2

It started to keep up at around 4Mbps 1080p This might be a jail not getting or using enough CPU First it thought ffmpeg wasn't using enough threads.
I haven't accepted the cpu is the bottleneck btw but more that there is something in my config that is the issue rather than Emby. I wish native dot.net would hurry the fuck up i am not a fan of mono. 

Link to comment
Share on other sites

6 hours ago, SpaceCabbi said:

CPU: Intel - Xeon E3-1230 V6 3.5GHz Quad-Core Processor
Motherboard: Supermicro - MBD-X11SSM-F Micro ATX LGA1151 Motherboard
Memory: Samsung - 32GB (2 x 16GB) DDR4-2133 Memory
Controllers: IBM-ServeRAID-M1015 SAS HBA (LSI9220-8i) flashed to IT mode
Chassis: HPE 24 Bay 3Gb SAS Expander in a EMC Avamar Gen4s DRBGP
Storage: 12x Hitachi HGST ultrastar 4tb Raidz2

It started to keep up at around 4Mbps 1080p This might be a jail not getting or using enough CPU First it thought ffmpeg wasn't using enough threads.
I haven't accepted the cpu is the bottleneck btw but more that there is something in my config that is the issue rather than Emby. I wish native dot.net would hurry the fuck up i am not a fan of mono. 

Your CPU should be fine for 1080p, but you should look at some more RAM. 

General rule of thumb on FreeNAS is 1Gb/Tb of storage. That's just for the file system. I would bet if you were to check your memory usage during playback you'd be maxed out or pretty close to it.

Link to comment
Share on other sites

SpaceCabbi
1 hour ago, MRobi said:

Your CPU should be fine for 1080p, but you should look at some more RAM. 

General rule of thumb on FreeNAS is 1Gb/Tb of storage. That's just for the file system. I would bet if you were to check your memory usage during playback you'd be maxed out or pretty close to it.

I have read that somewhere as well but I have 32Gb and 32TB Storage so should be fine.

But nevertheless has a look at history to confirm jrKdISx.png

Link to comment
Share on other sites

adrianwi

I have that processor in my FreeNAS box and have no problems at all with emby playing 1080p content.  I never had any issues with an earlier E3-1231v3 and 32GB RAM but I had less storage.

When I went to school 12 x 4 was a little more than 32, but the 1/1 rule of thumb starts to blur beyond 16GB RAM from my experience.  That said, FreeNAS loves RAM and that motherboard should allow another 32GB so it shouldn't be too expensive to upgrade?

I'm running the X11SSL-CF with 64GB RAM and 8x8TB drives.  I've also got 6 jails and 4 VMs running without any issues. 

Good luck!

 

 

Link to comment
Share on other sites

4 hours ago, SpaceCabbi said:

I have read that somewhere as well but I have 32Gb and 32TB Storage so should be fine.

You're definitely fine for running the filesystem. But what's left over for apps? Looks like as low as  1.84Gb free. And you've got how many jails running plus streaming video?

FreeNAS and ZFS is highly dependent on RAM. It'll take whatever you throw at it. I've personally got 96Gb of ram, 75 of which is being used by the ZFS file system 

15 minutes ago, adrianwi said:

When I went to school 12 x 4 was a little more than 32, but the 1/1 rule of thumb starts to blur beyond 16GB RAM from my experience.  That said, FreeNAS loves RAM and that motherboard should allow another 32GB so it shouldn't be too expensive to upgrade?

RAIDZ-2 so he should be somewhere aroudn 25-30Gb of usable storage. I always took the "rule of thumb" as total storage though since the duplication in RAIDZ-2 is still storage. Even if it's not "usable" the file system is still using it as storage.

Link to comment
Share on other sites

adrianwi

My understanding on the 'rule of thumb' was total storage, so as you've pointed out, 48TB with 32GB is stretching things without any services running.

I'm hardly one to talk though, as I've got 64TB with 64GB but am running several services which according to the dashboard are using 20GB!

preview

 

Link to comment
Share on other sites

SpaceCabbi
2 hours ago, adrianwi said:

I have that processor in my FreeNAS box and have no problems at all with emby playing 1080p content.  I never had any issues with an earlier E3-1231v3 and 32GB RAM but I had less storage.

When I went to school 12 x 4 was a little more than 32, but the 1/1 rule of thumb starts to blur beyond 16GB RAM from my experience.  That said, FreeNAS loves RAM and that motherboard should allow another 32GB so it shouldn't be too expensive to upgrade?

You would think. But no. at least 100 I don't have atm 😲

Ty for the feedback though, it confirms there is something else going on.  

2 hours ago, MRobi said:

You're definitely fine for running the filesystem. But what's left over for apps? Looks like as low as  1.84Gb free. And you've got how many jails running plus streaming video.
FreeNAS and ZFS is highly dependent on RAM. It'll take whatever you throw at it. I've personally got 96Gb of ram, 75 of which is being used by the ZFS file system 

RAIDZ-2 so he should be somewhere aroudn 25-30Gb of usable storage. I always took the "rule of thumb" as total storage though since the duplication in RAIDZ-2 is still storage. Even if it's not "usable" the file system is still using it as storage.

It runs 4 jails sonarr/radarr/nzbget and transmission/Emby. the 1.8 free was a exception normally there is plenty free I was updating to rc1 i suspect that was that. Normally it's around 12-16 atm even 22. free. But i get your point but its not the cause there is something odd going on i have yet to investigate due to lack of time.

And i might never will, look wat dropped yesterday:

https://github.com/truenas/truenas-installer/releases
Now if only they would bring it to Centos but debian wil have to do.

Edited by SpaceCabbi
Link to comment
Share on other sites

1 hour ago, SpaceCabbi said:

Normally it's around 12-16 atm even 22. free. But i get your point but its not the cause there is something odd going on

If you've got 22Gb of ram free, that right there is something odd going on ;)

In all seriousness, that's likely after a reboot. It's the only times you'll ever see those numbers of free ram. ZFS by design will use almost every available mb of ram you give it. It runs on the principle of "free memory is wasted memory". And "inactive" memory is actually cached but could become free if needed.

You'll see my services sitting at idle are using 16Gb. @adrianwi is using 20Gb. Yours is likely somewhere in that range as well. Then factor in what you need for the file system. I know it sucks spending more money, but with the amount of storage you've got you'll really need to bump up that ram because it's going to (and maybe already is) causing a bottleneck. At least a couple sticks of ram are one of the cheapest upgrades you can do.

Capture.PNG

Link to comment
Share on other sites

  • 1 month later...
SpaceCabbi
On 9/29/2020 at 5:33 AM, Luke said:

Are you still having an issue with this?

@Luke Sorry for my late response, no well yes (see below) I narrowed it down to file permissions, but what exactly goes wrong no clue tbh. I was using ACL for a while that worked, now back to default permissions the problem reappears but I can't figure out why some files will play normal but new created files are not.

bldbSxX.png

5cSxcqU.png

All rights are identical as far as I can see. emby runs as emby:media data store has basic permissions all set to 777 
Issue remains the same it will start playing then after roughly a minute it stalls and the whole server pretty much. Reloading page then brings you back to the main screen where you select sever, looks like it has reset. So yea freenas/freebsd stuff can't wait till truenas debian is released. But anyways if you have any tips by all means. Maybe mono demands ACL like permissions ?

 

Link to comment
Share on other sites

5 hours ago, SpaceCabbi said:

Issue remains the same it will start playing then after roughly a minute it stalls and the whole server pretty much.

If it starts playing then it is definitely not a permissions issue. It will either have access or it won't. There is no having access, then a minute into it the server realizes it shouldn't and takes it away.

My first hunch still says that it's resource related, but there's a few other things you could explore. I would recommend more ram even if you weren't running any jails at all.

What version of emby server are you running? What version of mono are you running? Are your packages for all your dependencies up to date?

The one point that's throwing me off a little here is when you said by reducing your quality to around 4mb/s it starts playing smoothly.... Direct playing is very resource friendly on the server.  Everything is handled by the end device (Shield TV, FireTV, AppleTV, Phone, etc... etc...). When you reduce quality the server will transcode the file to the lower quality which puts more strain on resources than direct play. So are you experiencing the same issue on ALL playback devices or just the 1? Have you tried playing the same file through the web player (browser)? Does it do it with all files you try to play or just certain ones or certain qualities like 1080p only? There's lots of variables that could come into play with end devices.

 

Link to comment
Share on other sites

SpaceCabbi
11 minutes ago, MRobi said:

If it starts playing then it is definitely not a permissions issue. It will either have access or it won't. There is no having access, then a minute into it the server realizes it shouldn't and takes it away.

My first hunch still says that it's resource related, but there's a few other things you could explore. I would recommend more ram even if you weren't running any jails at all.

What version of emby server are you running? What version of mono are you running? Are your packages for all your dependencies up to date?

The one point that's throwing me off a little here is when you said by reducing your quality to around 4mb/s it starts playing smoothly.... Direct playing is very resource friendly on the server.  Everything is handled by the end device (Shield TV, FireTV, AppleTV, Phone, etc... etc...). When you reduce quality the server will transcode the file to the lower quality which puts more strain on resources than direct play. So are you experiencing the same issue on ALL playback devices or just the 1? Have you tried playing the same file through the web player (browser)? Does it do it with all files you try to play or just certain ones or certain qualities like 1080p only? There's lots of variables that could come into play with end devices.

 

It is very clearly an issue with rights. How i do not know as you say it makes no sense whatsoever. 
All the info in the older post is old, Currently its quite simple all files downloaded upto 2 days ago wil play no issues no problems.
All users experience exactly the same.

New downloaded files after I changed back to basic permissions have this issue to resolve it all I have to do enable ACL again and set rights to an equivalent of 777.
That brings other issues inheritance etc won't go into that but suffice to say I hate ACL and prefer the default permissions.

If it were resource related then all movies should fail. 

Link to comment
Share on other sites

1 hour ago, SpaceCabbi said:

It is very clearly an issue with rights. How i do not know as you say it makes no sense whatsoever. 

It's not that it makes no sense, it's that it is impossible. File permissions cannot change after playback starts. The server will either have access to the file or it won't. 

2 hours ago, SpaceCabbi said:

All the info in the older post is old, Currently its quite simple all files downloaded upto 2 days ago wil play no issues no problems.

This however is good info and I have run into this myself numerous times after upgrading to a new version of Emby. The issue usually has to do with dependencies. Either a missing dependency or one with a version not compatible with the current version of Emby. For some reason it only affects new files added to the server after the upgrade.

Is your jail set to latest or quarterly packages? Someone correct me if I'm wrong, but I believe the latest stable should have packages in the jail set to latest not quarterly. Make that change then do a pkg update && pkg upgrade

Also make sure you have all of these dependencies installed. mono libass fontconfig freetype2 fribidi gnutls iconv opus samba48 sqlite3 libtheora libva libvpx libvorbis webp libx264 x265 dav1d libzvbi libraw ImageMagick6

Link to comment
Share on other sites

Baenwort
On 11/10/2020 at 9:54 AM, MRobi said:

It's not that it makes no sense, it's that it is impossible. File permissions cannot change after playback starts. The server will either have access to the file or it won't. 

This however is good info and I have run into this myself numerous times after upgrading to a new version of Emby. The issue usually has to do with dependencies. Either a missing dependency or one with a version not compatible with the current version of Emby. For some reason it only affects new files added to the server after the upgrade.

Is your jail set to latest or quarterly packages? Someone correct me if I'm wrong, but I believe the latest stable should have packages in the jail set to latest not quarterly. Make that change then do a pkg update && pkg upgrade

Also make sure you have all of these dependencies installed. mono libass fontconfig freetype2 fribidi gnutls iconv opus samba48 sqlite3 libtheora libva libvpx libvorbis webp libx264 x265 dav1d libzvbi libraw ImageMagick6

There isn't a samba48 package in the latest FreeBSD repo that FreeNAS/TrueNAS uses. 

Link to comment
Share on other sites

SpaceCabbi
6 hours ago, Baenwort said:

There isn't a samba48 package in the latest FreeBSD repo that FreeNAS/TrueNAS uses. 

Correct is that a big issue ? the rest is all there beside the files are are local via zfs

Link to comment
Share on other sites

SpaceCabbi

Solved.....
So in the end it turns out my SSD pool is fucked with a write performance of max 100mb/sec and my HD pool 4.3 GB/sec Ok fair the values do not make sense to me have to look into that but there is a severe différance between those 2 and if there is it should be reversed. 

So switched my transcoding dir to the storage pool mount and it started working again, there reason i had of and on success was most likely due to zone minder jail being of when testing since its a io heavy jail.
 

Link to comment
Share on other sites

12 hours ago, Baenwort said:

There isn't a samba48 package in the latest FreeBSD repo that FreeNAS/TrueNAS uses. 

You're right, didn't even notice. @Luke the dependency list should probably be updated on the beta install page. I'm using samba411 but it looks like 412 and 413 are released as well.

 

1 hour ago, SpaceCabbi said:

So switched my transcoding dir to the storage pool mount and it started working again, there reason i had of and on success was most likely due to zone minder jail being of when testing since its a io heavy jail.
 

If your issue was when direct playing, your transcode dir shouldn't have an effect on anything.  Regardless, if it works I'm not going to question it :)

Link to comment
Share on other sites

SpaceCabbi
3 hours ago, MRobi said:

If your issue was when direct playing, your transcode dir shouldn't have an effect on anything.  Regardless, if it works I'm not going to question it :)

And that solves why all series are working and movies didn't well not the ones I recently downloaded they were all transcoding for what ever reason.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...