HTTP Response are Slow

[killride 47]

Hi,

 

I post you the server log file :

2018-01-08 14:58:16.587 Info HttpServer: HTTP Response 200 to 195.20.147.130. Time: 19438ms (slow). https://dataghost.ddns.net:8920/emby/Users/a89d2504b90a498ba050d5e68962524d/Items?format=json&Fields=Path%2CGenres%2CSortName%2CStudios%2CWriter%2CProductionYear%2CTaglines%2CCommunityRating%2COfficialRating%2CCumulativeRunTimeTicks%2CMetascore%2CAirTime%2CDateCreated%2CMediaStreams%2CPeople%2COverview%2CCriticRating%2CCriticRatingSummary%2CEtag%2CShortOverview%2CProductionLocations%2CTags%2CProviderIds%2CParentId%2CRemoteTrailers%2CSpecialEpisodeNumbers%2CMediaSources%2CVoteCount%2CItemCounts&Ids=f7d7a1f2759e2ad4cb2037ad04ea8ff7%2Cf01a20658bbe95ee439288214ecfcb85%2C16c6564136e15af9213bfc5479d431f5%2C476c3ed10ccbcf7bb70443b97adfa975%2C2f784c7cce53990597bb0577c0347fbb%2C6062e68db72c05a9486d354072afa44c%2C1d7df978de5706d79212c385c1ff95a4%2Ccabd25fbf68f110f4ce0ae3c3cea2812%2Ce71ee4ccc03e7b46202980d76e0000e6%2Cb1c0605139a6dbb61428e04f68a9e36d%2C278d239f268b8aca279220992b93934d%2Cf54e05eb3760f4d743798e27895cad48%2C6f67527fc0e94d7e02270e063bfbb302%2C99816c07d7cc01ab0f027273d3e3d462%2Cbe621d15e07453f4b9e93436e0663cca 
2018-01-08 14:58:16.597 Info HttpServer: HTTP Response 200 to 178.39.82.241. Time: 6826ms (slow). https://dataghost.ddns.net:8920/emby/Users/809e0d46da6c4be293b91f52822d95e2/Items?format=json&Fields=Path%2CGenres%2CSortName%2CStudios%2CWriter%2CProductionYear%2CTaglines%2CCommunityRating%2COfficialRating%2CCumulativeRunTimeTicks%2CMetascore%2CAirTime%2CDateCreated%2CMediaStreams%2CPeople%2COverview%2CCriticRating%2CCriticRatingSummary%2CEtag%2CShortOverview%2CProductionLocations%2CTags%2CProviderIds%2CParentId%2CRemoteTrailers%2CSpecialEpisodeNumbers%2CMediaSources%2CVoteCount&Ids=ba79e98f46618eba9d5749476db1a569%2C206a2cab584d8ec79fa4699ffc51dca2%2Cba752fb43bf24a7d7d479d5625db04cc%2C97b7cb6f529da26a25f9b7d007c624a7%2C0ceb49eed5ff1cfeaec5de3ddfe5c525%2Ca74d76e9ed86801b73db61897fc476ab%2Cc57d20ed63ebf7639ca9122af98e2b28%2C439a52bc98040b298e020e68ccd217c3%2Cc25d94ff2a071a78c10d721fb5625727%2C2fc0d7eed89e897c864478d8197663cf%2C6126c00518e95e3c02a914df9f713f0a%2Ce9bf0a36c000cb102e9510820b66c552%2C66e2829faf1e4a3dd5b5c9a3774cf01a%2Cd2d2c3588cb5338c37ad949a7b7d3e45%2Ca42e2f0149bf799173c641902fe33b51 
2018-01-08 14:58:16.618 Info HttpServer: HTTP Response 200 to 80.219.96.2. Time: 2713ms (slow). https://dataghost.ddns.net:8920/emby/Users/a558a99e540c44ee904810ce46315123/Items?format=json&Recursive=True&Fields=Etag&IsMissing=False&SortOrder=Ascending&ParentId=d1a619a6a967a808973ae77cb091d9fa&IsVirtualUnaired=False&IncludeItemTypes=Episode&LocationTypes=FileSystem%2CRemote%2COffline&StartIndex=375&EnableTotalRecordCount=False&CollapseBoxSetItems=False&Limit=15&SortBy=SortName 
2018-01-08 14:58:16.619 Info HttpServer: HTTP Response 200 to 88.163.192.184. Time: 2507ms (slow). https://dataghost.ddns.net:8920/emby/Users/dc5336eda06e40b09ce3b4b9999c8340/Items?format=json&Recursive=True&Fields=Etag&IsMissing=False&SortOrder=Ascending&ParentId=d1a619a6a967a808973ae77cb091d9fa&IsVirtualUnaired=False&IncludeItemTypes=Episode&LocationTypes=FileSystem%2CRemote%2COffline&StartIndex=5460&EnableTotalRecordCount=False&CollapseBoxSetItems=False&Limit=15&SortBy=SortName 
2018-01-08 14:58:16.626 Info HttpServer: HTTP Response 200 to 86.200.42.206. Time: 3051ms (slow). https://dataghost.ddns.net:8920/emby/Users/fb6a7c5d65ee4f5690a7533ac72d39ec/Items?format=json&Recursive=True&Fields=Etag&IsMissing=False&SortOrder=Ascending&ParentId=d1a619a6a967a808973ae77cb091d9fa&IsVirtualUnaired=False&IncludeItemTypes=Episode&LocationTypes=FileSystem%2CRemote%2COffline&StartIndex=3840&EnableTotalRecordCount=False&CollapseBoxSetItems=False&Limit=15&SortBy=SortName 
2018-01-08 14:58:16.663 Info HttpServer: HTTP Response 200 to 77.57.20.47. Time: 8583ms (slow). https://dataghost.ddns.net:8920/emby/Users/7de18288711c42b7b9f8ea93b6333ebd/Items?format=json&Fields=Path%2CGenres%2CSortName%2CStudios%2CWriter%2CProductionYear%2CTaglines%2CCommunityRating%2COfficialRating%2CCumulativeRunTimeTicks%2CMetascore%2CAirTime%2CDateCreated%2CMediaStreams%2CPeople%2COverview%2CCriticRating%2CCriticRatingSummary%2CEtag%2CShortOverview%2CProductionLocations%2CTags%2CProviderIds%2CParentId%2CRemoteTrailers%2CSpecialEpisodeNumbers%2CMediaSources%2CVoteCount&Ids=c0f7052a9070ba6ff211d09f2d1619cf%2Ca67ad9fc8e9b2731d3d39f89a81e5d04%2C3b9b657bd1ef79867d191ed778d6bba0%2Cba3d4898ff2c3cbf1e0c76478be1c33a%2C59b6cd3489ea697d042c127f5f6e34d7%2C7591ab0b498ea85b109d91e70ca93d5c%2C69c9c49de415105cfa2dbf4221f36a79%2C7cf97e5502260882047fc2c0c14df46b%2C30a6c139f60394f4c4e7258c5a82cc04%2C04b7a21a32285af37ec71d980fae331a%2Cd97cead45783dc46151dc12276e63db7%2C7f6efcb9b4fb4dcbcac0823bb0864d4c%2Ca9a35089be674f52c033d1cf4f8b1b5e%2C11280c1efb8fc82818c8d9e252af3709%2C5273084631549ab806c3312a069bb9d2 
2018-01-08 14:58:16.665 Info HttpServer: HTTP Response 200 to 46.126.213.6. Time: 16532ms (slow). https://dataghost.ddns.net:8920/emby/Users/0fb385f961884b59afc81842092139e7/Items?format=json&Recursive=True&Fields=Path%2CGenres%2CSortName%2CStudios%2CWriter%2CProductionYear%2CTaglines%2CCommunityRating%2COfficialRating%2CCumulativeRunTimeTicks%2CMetascore%2CAirTime%2CDateCreated%2CMediaStreams%2CPeople%2COverview%2CCriticRating%2CCriticRatingSummary%2CEtag%2CShortOverview%2CProductionLocations%2CTags%2CProviderIds%2CParentId%2CRemoteTrailers%2CSpecialEpisodeNumbers%2CMediaSources%2CVoteCount&IsMissing=False&SortOrder=Ascending&ParentId=d1a619a6a967a808973ae77cb091d9fa&IsVirtualUnaired=False&IncludeItemTypes=Series&LocationTypes=FileSystem%2CRemote%2COffline&StartIndex=90&EnableTotalRecordCount=False&CollapseBoxSetItems=False&Limit=15&SortBy=SortName 
2018-01-08 14:58:16.811 Info HttpServer: HTTP Response 200 to 195.20.147.130. Time: 17153ms (slow). https://dataghost.ddns.net:8920/emby/Users/a89d2504b90a498ba050d5e68962524d/Items?format=json&Fields=Path%2CGenres%2CSortName%2CStudios%2CWriter%2CProductionYear%2CTaglines%2CCommunityRating%2COfficialRating%2CCumulativeRunTimeTicks%2CMetascore%2CAirTime%2CDateCreated%2CMediaStreams%2CPeople%2COverview%2CCriticRating%2CCriticRatingSummary%2CEtag%2CShortOverview%2CProductionLocations%2CTags%2CProviderIds%2CParentId%2CRemoteTrailers%2CSpecialEpisodeNumbers%2CMediaSources%2CVoteCount%2CItemCounts&Ids=8ebff3ecf609f4c2f9cf26cba8776ec4%2C658a2d39a2d440407f0bf3ea4dec1a63%2C56d81af6103d6a0f5cbc9890e879bfe7%2C964d6a7e6327f5254f003c5cf02381c7%2C88fe0dbf409339e36c529c16abfd8745%2Ca976d7c95f98a42201b66843cf9e0d52%2C720241cb5bc30f462b8ba500cd5efd9c%2C7809c3bec72b177d4194b9fe4a9f7639%2Ccf3f09df1a1b09e6d579d6d46e61efcc%2C9aa7a187c06df83f4868a1c7f6023ce6%2C656f3cd9230ceebe2a3e9024fc30d09e%2C24cc965f5a8a75818010d6bc4928c11c%2Cf3ed11e9a40fdb55445917d2ed28cd9e%2C158a3b564d2573245766a4a0fe21689e%2C77118d4e8e336c6315e6955be39cc723 

As you can see I have really slow response. My emby server is running on a Xeon E3-1246 v3 and he's using 5% CPU during this capture.

 

Do you know why the response is so slow?

[ebr 16172]

What app are those coming from?

 

If you attach the entire log so we can see the whole context we can probably provide more answers.

 

Thanks.

[killride 47]

I cannot attach the entire log cause if to many SSL exception, my log file increase really fast. 

 

It's Android Boxes with Emby for Kodi running manual sync.

[killride 47]

@@Luke Do you have any idea about how to make the server more performant?

Edited January 8, 2018 by killride

[Happy2Play 9780]

This is just https connections and not http correct?

Edited January 8, 2018 by Happy2Play

[killride 47]

I think so, It's HTTPS cause HTTP (8096) is not forwarded on my router and force HTTPS on external connection is enable on my emby configuration.

Edited January 8, 2018 by killride

[Luke 42077]

Please attach the complete emby server log, thanks.

[killride 47]

Here

server-63651019938.txt

[Luke 42077]

hi there @@killride, i would suggest removing the kodi sync plugin, and then also disconnect any instances of kodi that are currently connected to your emby server. 

 

Please see if this improves the performance of your emby server. thanks !

[ebr 16172]

I think that is the process he is trying to improve the performance of .

 

Killride - that particular process can be quite expensive depending on several factors.

 

One question, do the SSL errors you reported only occur during this sync process?

[killride 47]

Let me try this later, I will come back to you soon

[killride 47]

@ebr  Yes it's when Kodi client try to sync via HTTPS

 

@Luke  I blocked the https port and restart the server, I start my local Kodi (using http) It seems to work but obviously I'm alone syncing at the time.

 

You will see at the end of the logs when I deblocked to HTTPS port than the SSL exception occur and http response is going slowly.

 

I think we should fix the SSL exception first to see if it's that or if it's only due to overcharge.

server-63651041475.txt

Edited January 8, 2018 by killride

[Luke 42077]

I don't think there's anything we can do server side. The log seems to indicate the client is terminating the connection:

System.IO.IOException: Authentication failed because the remote party has closed the transport stream.

Maybe the client doesn't like the SSL cert?

[killride 47]

how it's possible?

 

Do I have to enable verify SSL on Emby for kodi? Or it's obvious when SSL cert is up on the server?

Edited January 8, 2018 by killride

[killride 47]

Do you think this exception will overload the server and occured high latency?

[Luke 42077]

Probably not, however I'm not sure why Kodi is aborting connections. Actually maybe one way it could cause a performance issue is if it is getting stuck in a cycle of abort -> retry -> abort, etc.

[killride 47]

This is my kodi logs during manual sync right now.

 

With Verify SSL enabled

kodi.old.log

Edited January 8, 2018 by killride

[Luke 42077]

I'm guessing you probably want to turn that off. What kind of SSL cert is this?

[killride 47]

Let's encrypt with pdx password

[killride 47]

It's also the same without Verify SSL...

[killride 47]

@@Luke

 

I would like to understand why the http response is slow.

 

I restart my server and connect only 2 kodi on it and start a manual sync.

 

Log server attached.

 

I cannot imagine than the server is raming with only 2 clients on it.

 

Usely I have around 50 Kodi clients connected at the same time and those slowly response cause timeout and then It's impossible to sync the whole library.

 

We are in http and not https on this test so it's not a SSL issue as we talked about

server-63651545708.txt

Edited January 14, 2018 by killride

[ebr 16172]

Hi.  Your responses are slow due to the nature of the queries that plug-in has to perform and the size of your database I believe.  I don't think this is related to the number of users as much as those factors.

 

You might consider using the EmbyCon solution instead which does not require a sync.

[killride 47]

I tried EmbyCon and I had timeout and impossible to open my tv show library and I don't want to change to EmbyCon for many reason. How can we improve that?

[killride 47]

Do you think It will change something if I create few folders on my emby server for tv shows? instead of one with 30k episodes?

Edited January 14, 2018 by killride

[ebr 16172]

How does your library perform using one of the other apps (web, mobile, theater, etc.)?