Downloading large series seems to get "stuck"

[pbathuk 20]

I am using the latest emby beta, however this happened before the beta as well.

If I use the download function to have some offline movies / shows etc, when downloading a larger show (I will use Bluey for this, as it has 154 items, across 3 seasons), it will work perfectly, converting & transfering up to a point and then just sit on the "Awaiting Transfer" and does not seem to move on. 

I have tried leaving for hours and nothing moves forwards, however sometimes it can be 5 minutes and get a little spike of activity, then nothing for ages again, but if I, on the client itself, click into manage downloads, click into the show and click Save it starts the transfer for a little bit longer.

Observations from myself :

It seems to get "stuck" once all the conversion tasks are completed, but there are a backlog of transfers, in my case the conversions of the 154 does not take long, but the transfer also does not take long (1gbps wifi connection gets saturated when it is working).

I have so far tried:

• Whole show in one go - seems to get "stuck" around the 30% mark for me.
  • Click Save will work to an extent, but is very manual.
  • Every so often some will transfer (Just noticed after 5 minutes I just had a couple more episodes transfer).
• Downloading each season (50 ish episodes per season)
  • doing all at the same time
  • seems to get "stuck" after season 1 has completed (all conversions have completed as well).
  • Click Save on the next season sparks it off again.. and then it pauses for a minute, but started back up again 1 minute later, then pauses again. 
• Downloading each season
  • but not at the same time - so waiting for season 1 to complete and then doing season 2 etc
  • Season 2 seemed to get stuck... which is weird. 
    • After seeing that it was Season 2 getting stuck I then deleted the request for Season 2 and started Season 3 - incase it was a file in Season 2 that was causing this... 
    • So I have Season 1 (finished and no restarting / pauses noticed)
    • Season 2 "stuck" - removed request
    • Season 3 "stuck" as well - seemed to follow the same as Season 2 - gets to a few episodes in and then stalls.
• So I then thought I would try a different show, incase it was just this show that was causing it, so used Cuphead, which is only 36 items, but is split across seasons. For this I started with just downloading the whole show in one go
  • The conversion - per show - did take longer than before, so there were natuaral pauses in transfer
  • 100% completed with zero interventions
  • Extra time taken because of conversions
  • So multiple seasons appears to work
• Another test I have run - using my android phone, the latest Google Play app & downloading the whole of bluey
  • I watched it transfer 90%+ until it said queued as it was waiting to convert the last 10% 
  • 100% converted & transfered without intervention.. infact I moved off the app and used something else while it was doing the last 10%

Personally I would like the transfer to keep going until all shows are completed and it seems to be a notification issue from the client to the server.. I can only assume that its setup as a request from the client saying "hey i'm ready for item X" and the message gets lost so just sits there, as the server can't force the download to the client etc.

Just sat on the client while doing a season by season check and saw the attached - so it seems like it is transfer alot in one go (when it can).

 

 

Sorry for the long post - happy to provide logs etc, if someone can point me to where the windows client saves its logs for the beta version? 

Or if you want other testing completed. 

 

[softworkz 5067]

@pbathuk- Is your Emby Server a beta version or are you running a release version? 

Currently, it's supposed to work with the Beta server only.

[pbathuk 20]

I'm on stable, but use docker, so can make a copy and spin it up on beta and test if you feel that would resolve this weird bug?

[softworkz 5067]

It's not a bug if you're on the stable server. The stable server doesn't have the corresponding implementation yet.

It would be great if you could try with the beta server.

Thanks!

[pbathuk 20]

spinning it up as we speak

[pbathuk 20]

ok, going to run the same tests that failed (imho) before.

[pbathuk 20]

So, from what I can whole show in one go is doing the same thing as before, its transfered some, but then because the conversions have finished it is now "stuck"..

Going to attempt the open and save to see if that restarts everything and its just a weird one off issue. - It grabbed a few more and then sat on "ready to transfer" again i'm afraid.

 

Happy to share logs if that helps?

[softworkz 5067]

On 10/12/2024 at 10:40 AM, pbathuk said:

however this happened before the beta as well.

Sorry, I had totally missed that part, which might rather indicate a server issue then.

But lets start small and test whether the Emby Beta client (plus Beta Server) are working as intended.
Here's a quick overview about how the sync feature is working in case of the new beta client (other clients are working similar):

1. Polling
   While the client app is running, it regularly queries the server about whether any items are ready for download
   The interval is 5 minutes, because the polling is just a fallback for cases when the actual mechanism doesn't work for some reason
2. Push Notifications
   This uses the MS push notification service to communicate to the Emby Beta app, even when it's not running, so we're having two cases here:
   1. Emby app not running
      • When the Emby server has an item ready to transfer, it contacts the MS push notification service and submits a message for the client app
      • The app gets started by Windows invisibly, supplying the push message to the app
      • The  app starts up in a background mode and connects to the Emby server
      • It queries the server for items that can be transferred and submits them as download jobs to the Windows BITS (Background intelligent transfer service)
      • Then it exits again (after running for just 10-30 seconds)
      • Once the BITS has completed a download, it activates (starts) the app again in background mode
      • The app acknowledges the download, and copies the file(s) to the target location
      • It marks the items as downloaded in the local database and connects to Emby server to confirm the download is completed
      • Then it exits again
   2. Emby app running
      • This works pretty much the same, just that the app keeps running
      • Also it stays connected to the BITS, so that it can show the actual downloads with completion percentage 
        (so far the only Emby app which does that - only in the client-side UI though)

 

Now, the first thing we should do is to test whether 2.1 is working properly. Please try this:

• Close the client application
• Use the web client of your (beta) server to initiate some downloads to the Emby Beta app
• You can use an album or a song or a short video and choose "Original Quality", so you don't have to wait for the conversion
• At the item, use the "Download To.." command and select the beta client app
• Then go to the server settings and there to "Downloads"
• Wait a bit and see whether the item you chose will get marked as "downloaded"
• Then you can open the app and see whether the item got downloaded
  => If yes, then we'd have confirmed that push notifications are working

Thanks

[pbathuk 20]

Hi, sorry it took so long, but I have followed the steps above, and yes it downloaded in the background

[softworkz 5067]

@pbathuk- Great! Thanks a lot for trying.

I think we rather need to take a look at the server side in this case. To get some useful logs you would (ideally):

• Run a fresh and totally blank Emby Server (beta) installation 
• Add a TV Shows library which contains just one of those large series
• Wait until the server has scanned and retrieved metadata
• Connect a fresh client like the Emby Server beta - it get's fresh by uninstalling and then installing again
• Stop the server, delete all log files and start again
• From the client, initiate sync/download
• Let everything run until it hangs
• Wait 10 minutes more
• Then zip up
  • all client log files
  • all server log files
  • library.db

 

Sorry that's a bit of work. And I already hate having to dig through all those logs, but it's the only way to find out what's happening.

[pbathuk 20]

Hi, 

I have a busy weekend this weekend, but will try to get that sorted next week.

 

When you say fresh install, I take it a new docker container + volume area (so nothing connected from the old dB etc) would suffice? 

 

[softworkz 5067]

3 minutes ago, pbathuk said:

I have a busy weekend this weekend, but will try to get that sorted next week.

No hurry, just when you got time.

3 minutes ago, pbathuk said:

hen you say fresh install, I take it a new docker container + volume area (so nothing connected from the old dB etc) would suffice? 

Perfect!

PS: I've moved this topic to the server forum.

[pbathuk 20]

Hey, just a heads up that I have a new server instance (used the same premier key - hopefully that does not break anything), reinstalled the windows client and set one of the shows (bluey) to install. 

I am sat here waiting for a 10 min gap. but thus far it basically transfers a few episodes, pauses for 4 mins and then transfers a few more..

So thus far I can't recreate the error. I'm going to leave this going to see if anything changes.. but it seems to be working.

I personally would argue slowly, since I have a full 1gbp connection and can transfer ALOT in the 5 minutes.. so maybe there is some server tweaks to up the queue amount, so it pushes more through in every 5 mins

Since I can't recreate it, I'm going to try this on the normal server again and leave it to see if suddenly it is working fine.

The key differences between this and the normal server:

http vs reverse proxy https - the main server sits behind a nginx reverse proxy to enable exposure to the internet via https. The test one is just on http with no exposure to the internet.

 

[pbathuk 20]

Hey, so I am pleased to say on the normal server (not the one I started from scratch) it has completed 100% of the transfer.

This was on a different machine (hardwired rather than wifi), but the other one is working and just seems to be taking time.

 

I did notice there was a new client updatre 2 days ago + I am on server version Version 4.9.0.30 beta.

 

So I am happy to report that (for now) the issue has disappeared.. I will therefore stick to the beta channel for now

[softworkz 5067]

I'm glad to hear that.

Thanks a lot for the feedback and let us know in case you'd encounter any more issues.