Jump to content


Photo

Transcoding causes Emby server restart - Memory Leak

Emby server transcoding restart crash memory leak

Best Answer henriquefer , 05 November 2019 - 07:40 AM

It seems that the issue was caused by internal network latency between my RPi4 and Nvidia Shield. So, now I have all machines connected through a good switch, and the connection is working faster and no issue so far.

 

Thank you for your attention.

I am going to perform further tests during the next days. I let you know if anything goes wrong. 

Go to the full post


  • Please log in to reply
37 replies to this topic

#21 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted 10 November 2019 - 07:48 PM

Hi,

 

In fact, I also have 5 GB of swap in my Shield. It helps me to get a more fluid experience using Shield. Now, it works perfectly. 

This is the memory of my Shield at this moment, transcoding one file:

 

total        used        free      shared     buffers

Mem:             2.8G        2.8G         31M         47M        7.3M
-/+ buffers/cache:           2.8G         38M
Swap:            5.0G        1.1G        3.8G


#22 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 139927 posts
  • Local time: 07:42 PM

Posted 10 November 2019 - 11:09 PM

That's great info, thanks !



#23 softworkz OFFLINE  

softworkz

    Advanced Member

  • Developers
  • 1859 posts
  • Local time: 01:42 AM

Posted 11 November 2019 - 06:19 PM

@amyhimesama @henriquefer

 

How does the memory usage evolve over time?

Is it constantly rising? Or going high quickly?



#24 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted 12 November 2019 - 02:41 AM

@amyhimesama @henriquefer

 

How does the memory usage evolve over time?

Is it constantly rising? Or going high quickly?

 

Usually, Shield does not have a lot of free memory (3 GB is insufficient and even that, Nvidia release a new model with the same amount of RAM), but when I am running a media server with a big library, the issue becomes bigger. 

 

Usually the memory does not increase quickly. It progressively increases over time. During transcoding activities, the amount of memory required increases, and without SWAP, Shield runs out of memory and stops the transcoding or the interface becomes completely laggy. 



#25 prunhardt OFFLINE  

prunhardt

    Member

  • Members
  • 15 posts
  • Local time: 01:42 AM

Posted Yesterday, 05:30 AM

I am reluctant to root my Shield. Will simply expand the internal storage help in this matter?



#26 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted Yesterday, 06:42 AM

I am reluctant to root my Shield. Will simply expand the internal storage help in this matter?

 

 

Expanding the internal storage does not help. I also have an SSD to expand the internal storage, mas I have to increase swap in order to allow a good experience using Shield.

 

Nvidia need to increase the amount of memory available on Shield. 3GB is a very low number comparing to the power of this machine.



#27 prunhardt OFFLINE  

prunhardt

    Member

  • Members
  • 15 posts
  • Local time: 01:42 AM

Posted Yesterday, 01:57 PM

Expanding the internal storage does not help. I also have an SSD to expand the internal storage, mas I have to increase swap in order to allow a good experience using Shield.

 

Nvidia need to increase the amount of memory available on Shield. 3GB is a very low number comparing to the power of this machine.

Ok, clear. Can you leave some instruction on how to root and how to increase swap?



#28 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted Yesterday, 03:16 PM

Ok, clear. Can you leave some instruction on how to root and how to increase swap?

 

Yes. Of course.

 

1. You need a male-male USB cable: https://images-na.ss...QLL._SX425_.jpg

2. You need a computer running Windows or Linux and install ADB (ADB is used communicate between your computer and Shield) https://www.xda-deve...ws-macos-linux/.

3. Enable the "Developer Options" on your Shield (http://corochann.com...oid-tv-216.html).

4. Under the "Developer Options" enable USB Debugging and connect your Shield and computer using the male-male USB cable.

5. Go here: https://developer.nv...loper-os-images to download the correct image for you device and follow the provided instructions. IMPORTANTE: Be sure that you have picked the correction image otherwise you will brick your Shield.

6. Then, you can run "sudo adb shell" on your computer and you are in the shell of your Shield.

7. Run the following commands to enable swap (5 GB).

swapoff /dev/block/zram0

echo 1 > /sys/block/zram0/reset

fallocate -l 5G /data/swapfile

chmod 600 /data/swapfile

mkswap /data/swapfile

swapon /data/swapfile



#29 softworkz OFFLINE  

softworkz

    Advanced Member

  • Developers
  • 1859 posts
  • Local time: 01:42 AM

Posted Yesterday, 09:23 PM

Usually, Shield does not have a lot of free memory (3 GB is insufficient and even that, Nvidia release a new model with the same amount of RAM), but when I am running a media server with a big library, the issue becomes bigger. 

 

Usually the memory does not increase quickly. It progressively increases over time. During transcoding activities, the amount of memory required increases, and without SWAP, Shield runs out of memory and stops the transcoding or the interface becomes completely laggy. 

 

Which process is taking those huge amounts of memory? EmbyServer or ffmpeg?



#30 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted Today, 06:08 AM

Which process is taking those huge amounts of memory? EmbyServer or ffmpeg?

 

Here, you can see the "top" output of my Shield with Emby server on (no active play):

5dcd27752aac3_Image1.jpg

 

 

And here, there is the "top" output when I am watching a episode (remotely) that requires transcoding:

5dcd27a92af71_Image2.jpg



#31 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 47494 posts
  • Local time: 07:42 PM

Posted Today, 10:31 AM

airplaydmr2 (whatever that is) seems to be eating a bunch of resources...



#32 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted Today, 10:39 AM

airplaydmr2 (whatever that is) seems to be eating a bunch of resources...

 

This is an app that allows me to use AirPlay on Shield. I installed it a coupe of days ago.

However, the issues regarding the usage of memory have already happened before the installation of this app. If you need, I can uninstall the app and share the results with you.



#33 softworkz OFFLINE  

softworkz

    Advanced Member

  • Developers
  • 1859 posts
  • Local time: 01:42 AM

Posted Today, 06:34 PM

@henriquefer - Thanks for the outputs. No need to end the other tasks. Operating systems tend to be lazy about releasing memory unless there is memory pressure, so in fact it's good to have some pressure because it will lead to more meaningful results. 

 

It would be awesome if you could create the same outputs once again:

  1. Right after transcoding starts (the moment you see the first picture at the client)
  2. After 1 minute
  3. After 5 minutes

Leave throttling off and use a remote client (just as before)



#34 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted Today, 07:17 PM

@henriquefer - Thanks for the outputs. No need to end the other tasks. Operating systems tend to be lazy about releasing memory unless there is memory pressure, so in fact it's good to have some pressure because it will lead to more meaningful results. 

 

It would be awesome if you could create the same outputs once again:

  1. Right after transcoding starts (the moment you see the first picture at the client)
  2. After 1 minute
  3. After 5 minutes

Leave throttling off and use a remote client (just as before)

 

 

I have tested as before, and this is case (almost minute by minute):

 

Before starting:

5dcddf4cc9d23_Image.jpg

 

Start:

5dcddf65d06dd_Image1.jpg

 

5dcddf769d302_Image2.jpg

 

1 Minute:

 

5dcddfaa36644_Image3.jpg

 

5dcddfbbcfad3_Image4.jpg

 

4 Minutes:

5dcddff3e68e4_Image9.jpg

 

5dcde000a88b8_Image10.jpg

 

5 Minutes:

Some video freezing occurs on the client.

 

5dcde03b382a1_Image11.jpg

 

5dcde047e9337_Image12.jpg

 

7 Minutes:

 

5dcde07529f42_Image13.jpg

 

5dcde0863b7f3_Image14.jpg



#35 softworkz OFFLINE  

softworkz

    Advanced Member

  • Developers
  • 1859 posts
  • Local time: 01:42 AM

Posted Today, 07:41 PM

Excellent, thanks.

 

I've put the relevant lines in order here:

 

5dcde5ab00c43_android_memory.jpg

 

The hw accelerated ffmpeg run shows in fact excessive memory usage. Probably it crashes at some point (red line) and ffmpeg is restarted using sw transcoding.

 

Would you be able to post or PM the two ffmpeg logs from this test?


Edited by softworkz, Today, 07:41 PM.


#36 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted Today, 07:49 PM

Excellent, thanks.

 

I've put the relevant lines in order here:

 

5dcde5ab00c43_android_memory.jpg

 

The hw accelerated ffmpeg run shows in fact excessive memory usage. Probably it crashes at some point (red line) and ffmpeg is restarted using sw transcoding.

 

Would you be able to post or PM the two ffmpeg logs from this test?

 

 

I sent you the log files via PM.



#37 softworkz OFFLINE  

softworkz

    Advanced Member

  • Developers
  • 1859 posts
  • Local time: 01:42 AM

Posted Today, 08:03 PM

Before hw transcoding fails after 4-5 minutes, it has already transcoded half of the file.

 

I'm wondering whether it's a hard leak or whether it just doesn't have time for releasing memory.

 

What happens when you activate throttling? I would like to know whether it would fail after transcoding the same amount of video data (= about 24 minutes) or later (or not).

 

For this test you'll need to let it play, so it will take a while, but you can stop once it fails and starts sw transcoding.



#38 henriquefer ONLINE  

henriquefer

    Member

  • Members
  • 23 posts
  • Local time: 12:42 AM

Posted Today, 08:41 PM

Before hw transcoding fails after 4-5 minutes, it has already transcoded half of the file.

 

I'm wondering whether it's a hard leak or whether it just doesn't have time for releasing memory.

 

What happens when you activate throttling? I would like to know whether it would fail after transcoding the same amount of video data (= about 24 minutes) or later (or not).

 

For this test you'll need to let it play, so it will take a while, but you can stop once it fails and starts sw transcoding.

 

Hi,

 

You can find all the screenshots in attachment.

I am going to send you the log files by PM.

 

The behaviour is the same.

Attached Files







Also tagged with one or more of these keywords: Emby, server, transcoding, restart, crash, memory, leak

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users