Jump to content


Photo

Version 4.1.0.5 bêta (netcore) Burst of packet loss in recordings

DVR netcore

  • Please log in to reply
42 replies to this topic

#1 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 31 January 2019 - 07:36 PM

Hello @solabc16

 

To debug the "potential" RAM memory leak, I switched this morning to the beta version running on netcore.  But since then, I can't record a single OTA stream without big glitches in it.  After some investigations, those glitches are from big burst of packet loss.

 

I have a HDhomerun Quatro and a Duo for a total of 6 tuners.  In the past, I saw this behavior, but only when I tried to record 5 or 6 streams at the same time with emby server on mono.  However, since the new version 4, I was able to record 6 streams without issue.  However, since I switched to netcore, I see big glitches even if I record only 1 stream.

 

To make sure that it's not related to my OTA setup, I fire up plex server on the same NAS using the same OTA gears and I can record 6 streams with plex without a single glitch in it.  (all validated using VideoRedo software)

 

I sent to you the logs ( sendlogs_NasDs418Play_synology_apollolake_418play_20190131T231601UTC.tgz)

 

If you need something else, I will let this emby server version running and I will use plex to record for the moment, otherwise I will be in trouble with the rest of the family :).  But I highly prefer using emby for my DVR needs than Plex.

 

Other observation.  OTA DVR recording take a lot of ressources.  On the mono version, recording 6 streams takes 90% of the CPU.  With the netcore, it's arround 70% (but with big burst of packet loss).  with plex, 6 streams take only 30%.  I don't convert on the fly the recordings.  I don't really understand this difference between emby and plex.

 

Thank you for your help


Edited by rouq, 01 February 2019 - 05:12 PM.


#2 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 134563 posts
  • Local time: 08:09 AM

Posted 01 February 2019 - 03:37 AM

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. thanks !



#3 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 01 February 2019 - 08:07 AM

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. thanks !

@Luke

It's not a playback issue, it's a recording issue. The files are corrupted (packets loss probably during write).

I already provided logs using the sendlogs utility.

Edited by rouq, 01 February 2019 - 08:08 AM.


#4 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 01 February 2019 - 02:45 PM

Hello @solabc16 @Luke

 

I may have found a lead.  The corruption of the recording files seems to happen only when the server is serving a client that need transcoding (doesn't need to be full transcoding, only transcoding the audio will trigger that).

 

If there is no client connected on the server or only direct play, my server can use all 6 tuners without a single glitch in any of them. 

 

However, if my son uses the roku ultra to play animations in the living room (this setup doesn't support AC3, so all my medias need at least audio transcoding), then no matter the number of tuners in use, all recordings will be corrupted.  I was able to recreate that several times in a row.

 

I wasn't that bad with mono.  I saw this corruption with mono, but at least 4 or 5 recordings were needed to create corruption.  So I never made the link with transcoding sessions.  However, with this version, no matter the number of recordings at the same time, if a transcoding session happens during those recodings, corruption will happen.

 

EDIT:  other test case results:

 

1.  Record 2 streams with emby, record the same streams with plex (running on the same NAS).  Playback a media through emby webapp that forces transcoding while the recordings are going on:

     Results: both recordings in emby are corrupted.  Both recordings in plex are clean.

 

2. Record 2 streams with emby, record the same streams with plex (running on the same NAS). Playback a media through plex web app that forces transcoding while the recordings are going on.

  Results: both recordings in emby are clean.  Both recordings in plex are clean.

 

I hope that may help you understand what is going on

 

Thank you for your help.


Edited by rouq, 01 February 2019 - 05:16 PM.


#5 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 01 February 2019 - 05:20 PM

Hell @rouq

Thanks for the update, that’s useful information and should help track down the cause of the problem.

Please can you attach the server logs that cover the above test scenarios.

Best
- James

#6 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 01 February 2019 - 05:30 PM

Hello @solabc16

 

I ran the sendlogs utility.  Here the file name: sendlogs_NasDs418Play_synology_apollolake_418play_20190201T212846UTC.tgz

 

Is it what you need?

 

Thank you


Edited by rouq, 01 February 2019 - 05:34 PM.


#7 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 01 February 2019 - 06:58 PM

That's great, thanks @rouq - what was the time window (roughly) of your tests above?

 

Best

- James



#8 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 01 February 2019 - 07:04 PM

Hello @solabc16

 

I recorded 6 streams between 11am and 12pm.  Those were successfull without corruption (no transcode sessions)

 

Then I did several short recordings testing, I would say between 12:40pm to 4pm.  The ones using emby and plex at the same time would be arround 3:30 and 4:15pm I think.  Those are all very short recordings.  No need to record hours to see the impairments when I start a transcode session.

 

Thank you


Edited by rouq, 01 February 2019 - 07:05 PM.


#9 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 02 February 2019 - 02:35 PM

Hello @rouq

 

As a test, can you disable the following setting : Server Manger -> Transcoding -> Enable Throttling

 

Make sure to save the configuration.

 

To make it easier to grab the most relevant log, can you restart Emby Server : Server Manager -> Dashboard -> Restart

 

Can you then re-run the test you used above to reproduce the issue and let me know what happens.

 

Best

- James



#10 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 02 February 2019 - 03:39 PM

Hello @solabc16

 

I performed this test case.  I disabled throttling as requested and rebooted emby server at 1:57pm.  But no change, the recordings are badly dammaged.

 

Test case #1: 2 recordings started at 1:59PM, stopped at 2:13pm.  Recorded both channels from emby and plex at the same time.  During those recordings, started playback in emby webapp (full transcoding).

Both recordings from emby are badly dammaged.  Both recordings from plex are clean.

 

Test case #2: 2 recordings started at 2:17pm, stopped at 2:32.  Recorded both channels from emby and plex at the same time.

time.  During those recordings, started playback in plex webapp (full transcoding).

Both recordings from emby and plex are clean.

 

I ran the Sendlogs utility (filename: sendlogs_NasDs418Play_synology_apollolake_418play_20190202T193849UTC.tgz)

 

thank you


Edited by rouq, 02 February 2019 - 03:39 PM.


#11 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 02 February 2019 - 03:47 PM

Hello @rouq

 

Thanks, I picked something up in the logs related to the above, so wanted to eliminate it from the equation; I can follow that up later.

 

Have you had an opportunity to notice what the CPU is doing when you start the transcoding? Is it spiking or running at 100% continually?

 

Having said that, if Plex is managing to complete the recordings at the same time without issue, it is unlikely to be a factor.

 

Thanks for the latest logs, I'll take a look at these shortly...

 

Best

- James



#12 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 02 February 2019 - 05:04 PM

Hello @solabc16

 

When the transcode session starts, with 2 recordings, the cpu doesn't spike.  The CPu is arround 50% load in those conditions.

 

Regards



#13 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 02 February 2019 - 05:30 PM

Hello @rouq

 

Thanks, you mentioned some comparisons with utilisation on Plex earlier, what do you see CPU-wise with 2 recordings?

 

Best

- James



#14 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 02 February 2019 - 05:56 PM

Hello @rouq

 

As another test, can you try the above with h/w assisted transcoding disabled : Server Manger -> Transcoding -> Enable hardware acceleration when available -> Off

 

You'll see higher CPU utilisation, but you should have enough headroom for the above test and to see if this has an impact.

 

Best

- James



#15 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 02 February 2019 - 07:07 PM

Hello @solabc16

 

For 2 recordings without anything else going on:

 

Emby takes an average of 28% CPU.  Plex takes an average of 6%.

 

For 6 recordings at the same time

Emby takes an average of 65% and plex takes an average of 18%

 

Without hardware acceleration, this system doesn't have enough room.  The CPU is at 99% and the transcode session is at 17fps.  So not enough for real time transcoding.  With 2 recordings, both are corrupted in emby.  But even with a CPU at 99%, plex was still able to record it without any loss.  I don't like their DVR, but I have to give them that, the recording process is very efficient.

 

Regards



#16 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 02 February 2019 - 07:19 PM

Ok, thanks @rouq.

 

The actual I/O won't be responsible for generating any significant CPU utilisation, so we must be doing more processing on the stream than Plex.

 

I'll come back to this again tomorrow. 

 

@Luke - is there any way to disable h/w transcoding and/or disable deinterlacing for Live TV only?

 

Best

- James



#17 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 134563 posts
  • Local time: 08:09 AM

Posted 03 February 2019 - 02:16 AM

 is there any way to disable h/w transcoding and/or disable deinterlacing for Live TV only?

 

No, there isn't.



#18 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 134563 posts
  • Local time: 08:09 AM

Posted 03 February 2019 - 02:17 AM

Hello @solabc16

 

For 2 recordings without anything else going on:

 

Emby takes an average of 28% CPU.  Plex takes an average of 6%.

 

For 6 recordings at the same time

Emby takes an average of 65% and plex takes an average of 18%

 

Without hardware acceleration, this system doesn't have enough room.  The CPU is at 99% and the transcode session is at 17fps.  So not enough for real time transcoding.  With 2 recordings, both are corrupted in emby.  But even with a CPU at 99%, plex was still able to record it without any loss.  I don't like their DVR, but I have to give them that, the recording process is very efficient.

 

Regards

 

We'd have to look at some log examples, thanks.



#19 rouq ONLINE  

rouq

    Advanced Member

  • Members
  • 117 posts
  • Local time: 08:09 AM

Posted 03 February 2019 - 08:25 AM

We'd have to look at some log examples, thanks.

I already sent several log sets to @solabc16.

Thanks

#20 solabc16 OFFLINE  

solabc16

    Advanced Member

  • ForumMod
  • 1967 posts
  • Local time: 01:09 PM

Posted 03 February 2019 - 08:42 AM

Hello @rouq

 

The CPU utilisation itself is likely a secondary factor, from your tests what we need to understand is why starting a transcoding session to a client device reproducibly breaks the recordings.

 

From the logs I've looked at, there are no errors as such - nothing is outright failing.

 

What happens if you have a single recording running and start a playback that requires a transcode?

 

No, there isn't.

 

Ok, we can use the custom launcher to do this for the purpose of troubleshooting.

 

Best

- James







Also tagged with one or more of these keywords: DVR, netcore

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users