Jump to content


Photo

Transcoding causes Emby server restart - Memory Leak

Emby server transcoding restart crash memory leak

Best Answer henriquefer , 05 November 2019 - 07:40 AM

It seems that the issue was caused by internal network latency between my RPi4 and Nvidia Shield. So, now I have all machines connected through a good switch, and the connection is working faster and no issue so far.

 

Thank you for your attention.

I am going to perform further tests during the next days. I let you know if anything goes wrong. 

Go to the full post


  • Please log in to reply
45 replies to this topic

#21 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 10 November 2019 - 07:48 PM

Hi,

 

In fact, I also have 5 GB of swap in my Shield. It helps me to get a more fluid experience using Shield. Now, it works perfectly. 

This is the memory of my Shield at this moment, transcoding one file:

 

total        used        free      shared     buffers

Mem:             2.8G        2.8G         31M         47M        7.3M
-/+ buffers/cache:           2.8G         38M
Swap:            5.0G        1.1G        3.8G


#22 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 146226 posts
  • Local time: 03:49 PM

Posted 10 November 2019 - 11:09 PM

That's great info, thanks !



#23 softworkz ONLINE  

softworkz

    Advanced Member

  • Developers
  • 2114 posts
  • Local time: 09:49 PM

Posted 11 November 2019 - 06:19 PM

@amyhimesama @henriquefer

 

How does the memory usage evolve over time?

Is it constantly rising? Or going high quickly?



#24 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 12 November 2019 - 02:41 AM

@amyhimesama @henriquefer

 

How does the memory usage evolve over time?

Is it constantly rising? Or going high quickly?

 

Usually, Shield does not have a lot of free memory (3 GB is insufficient and even that, Nvidia release a new model with the same amount of RAM), but when I am running a media server with a big library, the issue becomes bigger. 

 

Usually the memory does not increase quickly. It progressively increases over time. During transcoding activities, the amount of memory required increases, and without SWAP, Shield runs out of memory and stops the transcoding or the interface becomes completely laggy. 



#25 prunhardt OFFLINE  

prunhardt

    Member

  • Members
  • 20 posts
  • Local time: 09:49 PM

Posted 13 November 2019 - 05:30 AM

I am reluctant to root my Shield. Will simply expand the internal storage help in this matter?



#26 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 13 November 2019 - 06:42 AM

I am reluctant to root my Shield. Will simply expand the internal storage help in this matter?

 

 

Expanding the internal storage does not help. I also have an SSD to expand the internal storage, mas I have to increase swap in order to allow a good experience using Shield.

 

Nvidia need to increase the amount of memory available on Shield. 3GB is a very low number comparing to the power of this machine.



#27 prunhardt OFFLINE  

prunhardt

    Member

  • Members
  • 20 posts
  • Local time: 09:49 PM

Posted 13 November 2019 - 01:57 PM

Expanding the internal storage does not help. I also have an SSD to expand the internal storage, mas I have to increase swap in order to allow a good experience using Shield.

 

Nvidia need to increase the amount of memory available on Shield. 3GB is a very low number comparing to the power of this machine.

Ok, clear. Can you leave some instruction on how to root and how to increase swap?



#28 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 13 November 2019 - 03:16 PM

Ok, clear. Can you leave some instruction on how to root and how to increase swap?

 

Yes. Of course.

 

1. You need a male-male USB cable: https://images-na.ss...QLL._SX425_.jpg

2. You need a computer running Windows or Linux and install ADB (ADB is used communicate between your computer and Shield) https://www.xda-deve...ws-macos-linux/.

3. Enable the "Developer Options" on your Shield (http://corochann.com...oid-tv-216.html).

4. Under the "Developer Options" enable USB Debugging and connect your Shield and computer using the male-male USB cable.

5. Go here: https://developer.nv...loper-os-images to download the correct image for you device and follow the provided instructions. IMPORTANTE: Be sure that you have picked the correction image otherwise you will brick your Shield.

6. Then, you can run "sudo adb shell" on your computer and you are in the shell of your Shield.

7. Run the following commands to enable swap (5 GB).

swapoff /dev/block/zram0

echo 1 > /sys/block/zram0/reset

fallocate -l 5G /data/swapfile

chmod 600 /data/swapfile

mkswap /data/swapfile

swapon /data/swapfile


  • prunhardt likes this

#29 softworkz ONLINE  

softworkz

    Advanced Member

  • Developers
  • 2114 posts
  • Local time: 09:49 PM

Posted 13 November 2019 - 09:23 PM

Usually, Shield does not have a lot of free memory (3 GB is insufficient and even that, Nvidia release a new model with the same amount of RAM), but when I am running a media server with a big library, the issue becomes bigger. 

 

Usually the memory does not increase quickly. It progressively increases over time. During transcoding activities, the amount of memory required increases, and without SWAP, Shield runs out of memory and stops the transcoding or the interface becomes completely laggy. 

 

Which process is taking those huge amounts of memory? EmbyServer or ffmpeg?



#30 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 14 November 2019 - 06:08 AM

Which process is taking those huge amounts of memory? EmbyServer or ffmpeg?

 

Here, you can see the "top" output of my Shield with Emby server on (no active play):

5dcd27752aac3_Image1.jpg

 

 

And here, there is the "top" output when I am watching a episode (remotely) that requires transcoding:

5dcd27a92af71_Image2.jpg



#31 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 49019 posts
  • Local time: 03:49 PM

Posted 14 November 2019 - 10:31 AM

airplaydmr2 (whatever that is) seems to be eating a bunch of resources...



#32 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 14 November 2019 - 10:39 AM

airplaydmr2 (whatever that is) seems to be eating a bunch of resources...

 

This is an app that allows me to use AirPlay on Shield. I installed it a coupe of days ago.

However, the issues regarding the usage of memory have already happened before the installation of this app. If you need, I can uninstall the app and share the results with you.



#33 softworkz ONLINE  

softworkz

    Advanced Member

  • Developers
  • 2114 posts
  • Local time: 09:49 PM

Posted 14 November 2019 - 06:34 PM

@henriquefer - Thanks for the outputs. No need to end the other tasks. Operating systems tend to be lazy about releasing memory unless there is memory pressure, so in fact it's good to have some pressure because it will lead to more meaningful results. 

 

It would be awesome if you could create the same outputs once again:

  1. Right after transcoding starts (the moment you see the first picture at the client)
  2. After 1 minute
  3. After 5 minutes

Leave throttling off and use a remote client (just as before)



#34 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 14 November 2019 - 07:17 PM

@henriquefer - Thanks for the outputs. No need to end the other tasks. Operating systems tend to be lazy about releasing memory unless there is memory pressure, so in fact it's good to have some pressure because it will lead to more meaningful results. 

 

It would be awesome if you could create the same outputs once again:

  1. Right after transcoding starts (the moment you see the first picture at the client)
  2. After 1 minute
  3. After 5 minutes

Leave throttling off and use a remote client (just as before)

 

 

I have tested as before, and this is case (almost minute by minute):

 

Before starting:

5dcddf4cc9d23_Image.jpg

 

Start:

5dcddf65d06dd_Image1.jpg

 

5dcddf769d302_Image2.jpg

 

1 Minute:

 

5dcddfaa36644_Image3.jpg

 

5dcddfbbcfad3_Image4.jpg

 

4 Minutes:

5dcddff3e68e4_Image9.jpg

 

5dcde000a88b8_Image10.jpg

 

5 Minutes:

Some video freezing occurs on the client.

 

5dcde03b382a1_Image11.jpg

 

5dcde047e9337_Image12.jpg

 

7 Minutes:

 

5dcde07529f42_Image13.jpg

 

5dcde0863b7f3_Image14.jpg



#35 softworkz ONLINE  

softworkz

    Advanced Member

  • Developers
  • 2114 posts
  • Local time: 09:49 PM

Posted 14 November 2019 - 07:41 PM

Excellent, thanks.

 

I've put the relevant lines in order here:

 

5dcde5ab00c43_android_memory.jpg

 

The hw accelerated ffmpeg run shows in fact excessive memory usage. Probably it crashes at some point (red line) and ffmpeg is restarted using sw transcoding.

 

Would you be able to post or PM the two ffmpeg logs from this test?


Edited by softworkz, 14 November 2019 - 07:41 PM.


#36 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 14 November 2019 - 07:49 PM

Excellent, thanks.

 

I've put the relevant lines in order here:

 

5dcde5ab00c43_android_memory.jpg

 

The hw accelerated ffmpeg run shows in fact excessive memory usage. Probably it crashes at some point (red line) and ffmpeg is restarted using sw transcoding.

 

Would you be able to post or PM the two ffmpeg logs from this test?

 

 

I sent you the log files via PM.



#37 softworkz ONLINE  

softworkz

    Advanced Member

  • Developers
  • 2114 posts
  • Local time: 09:49 PM

Posted 14 November 2019 - 08:03 PM

Before hw transcoding fails after 4-5 minutes, it has already transcoded half of the file.

 

I'm wondering whether it's a hard leak or whether it just doesn't have time for releasing memory.

 

What happens when you activate throttling? I would like to know whether it would fail after transcoding the same amount of video data (= about 24 minutes) or later (or not).

 

For this test you'll need to let it play, so it will take a while, but you can stop once it fails and starts sw transcoding.



#38 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 14 November 2019 - 08:41 PM

Before hw transcoding fails after 4-5 minutes, it has already transcoded half of the file.

 

I'm wondering whether it's a hard leak or whether it just doesn't have time for releasing memory.

 

What happens when you activate throttling? I would like to know whether it would fail after transcoding the same amount of video data (= about 24 minutes) or later (or not).

 

For this test you'll need to let it play, so it will take a while, but you can stop once it fails and starts sw transcoding.

 

Hi,

 

You can find all the screenshots in attachment.

I am going to send you the log files by PM.

 

The behaviour was the same.

Attached Files


Edited by henriquefer, 14 November 2019 - 08:44 PM.


#39 softworkz ONLINE  

softworkz

    Advanced Member

  • Developers
  • 2114 posts
  • Local time: 09:49 PM

Posted 14 November 2019 - 08:51 PM

Thanks again. That seems to be a hard memory leak. I'll try to reproduce during the next days.

 

Last question: Do you experience that behavior with all kinds of source videos or only with specific ones?



#40 henriquefer OFFLINE  

henriquefer

    Member

  • Members
  • 27 posts
  • Local time: 08:49 PM

Posted 14 November 2019 - 08:59 PM

Thanks again. That seems to be a hard memory leak. I'll try to reproduce during the next days.

Last question: Do you experience that behavior with all kinds of source videos or only with specific ones?


all videos bigger than ~20 minutes (depending on bitrate) that requires transcoding.





Also tagged with one or more of these keywords: Emby, server, transcoding, restart, crash, memory, leak

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users