Missing lots of guide data

[kanipek 224]

Can we PLEASE get some logic put in Emby so that the guide can retain data if there are issues with SD?

 

I know I can delete the scheduled task in Emby but I have no idea how to re-add it afterwards even this would be of some help.

 

I understand that this is not really an Emby problem, but it is a problem for Emby users.

 

My guide is nearly empty of data after the last update. So pretty much useless to me at the moment.

 

I am attaching logs in hopes someone can tell me where the problem is. I have done a couple of refreshes of the guide without change.

 

Thanks for reading and any help!

server-63646001912.txt

server-63645955199.txt

[maegibbons 1267]

I am also seeing the same thing this evening here in the UK so it is probably an SD transient issue unless other changes have been made in emby server related to EPG in .7 but i dont think so.

 

HOWEVER, It does appear that valid guide data gets deleted from emby when there is an SD issue.  This should not happen.

 

The existing guide data should be retained.

 

Krs

 

Mark

[kanipek 224]

So for the moment I have added XMLTV guide data from Schedules Direct. This seems to have solved my guide data problem for the moment.

 

Will the refresh guide task automatically update the guide data - as long as the xmltv source file has been updated? I am thinking yes, but I do not know for sure.

 

1. Installed XMLTV

2. Ran the --> xmltv tv_grab_sd_json --configure command - pretty self explanatory a few steps/options to setup

3. Ran xmltv tv_grab_sd_json  -  Output to xmltv.cache

4. Pointed Emby to this file in LiveTV add XMLTV source

 

Had to redo a handful of items that I had chosen to not record using the Schedules Direct json data and it looks back to normal

[Latchmor 584]

Hi, I had this issue last night too but it's back to normal this morning after a guide refresh.

[maegibbons 1267]

Yes - all back this morning.  My comments above still very much stand.

 

Emby should NOT delete existing guide data when there is no guide data at SD or other collection problems..

 

Krs

 

Mark

[Guest asrequested]

I agree. It's annoying. I've contacted SD about it, and they say that everything is working, correctly. I'm going to send a link to this thread. Half my guide was gone the other night. I shouldn't have to manually refresh over and over. What if I couldn't access my server? Some of my shows wouldn't be recorded.

Edited November 12, 2017 by Doofus

[bungee91 98]

I find this extremely annoying as well, and seems to be happening more frequently lately. While I understand it's SD's fault, the way Emby handles this absolutely requires attention. I'm refreshing my guide now, hoping it will fix it.

[Guest asrequested]

My response from SD

 

 

 

So the issue is if the SD-JSON service returns an error  wipes the data?   Sounds like an Emby bug to me.

If the SD-JSON server returns a error status code, and not returning bogus data, then it's doing things correctly.  We provide data many days in advance... we shouldn't be expected to provide 99.999% uptime.  Client apps should be able to deal with (even multi-day) outages.

In addition, the whole point of the SD-JSON protocol is to provide a *differential* download..  it only downloads shows that change.  Is every Emby user downloading a full 20+ days of data multiple times a day?  If that's a case, we may have to implement bandwidth limits (we pay for bandwidth).  Plugin developers need to write efficient code.   I don't run the SD-JSON service, so I'm not sure if that's what's happening... it's something we (SD) may need to look into.

Robert(E)
 

[kanipek 224]

"In addition, the whole point of the SD-JSON protocol is to provide a *differential* download..  it only downloads shows that change.​...."

 

This differential download is exactly what Emby needs to be doing.  EPG123 is doing this, at least after the initial run it is doing differential updates.  Wish it was built for Emby, love EPG123.

[SikSlayer 197]

My response from SD:

 

So the issue is if the SD-JSON service returns an error  wipes the data?   Sounds like an Emby bug to me.

 

If the SD-JSON server returns a error status code, and not returning bogus data, then it's doing things correctly.  We provide data many days in advance... we shouldn't be expected to provide 99.999% uptime.  Client apps should be able to deal with (even multi-day) outages.

 

In addition, the whole point of the SD-JSON protocol is to provide a *differential* download..  it only downloads shows that change.  Is every Emby user downloading a full 20+ days of data multiple times a day?  If that's a case, we may have to implement bandwidth limits (we pay for bandwidth).  Plugin developers need to write efficient code.   I don't run the SD-JSON service, so I'm not sure if that's what's happening... it's something we (SD) may need to look into.

 

Robert(E)

 

And in one fell swoop, this has gone from a major issue to a major priority.

 

Now then, by default, I believe Emby only downloads guide data every 24 hours, so the "every Emby user downloading a full 20+ days of data multiple times a day" isn't true, but damage is being done.

 

In the other thread, Luke already mentioned that he would implement this feature after it was requested, but now it's gotta be a priority. This feature needs to be implemented post haste. We don't need SD being mad at Emby.

Edited November 13, 2017 by SikSlayer

[rmeden 10]

(I might as well chime in directly)

"We don't need SD being mad at Emby"

 

Your loader wiping data is a bigger concern than SD getting "mad".  We're headed by Open Source devs and totally understand OSS developer priorities (*life!*).  If Emby only updates once a day it shouldn't be a problem, and if it does become an issue, we can work with the Emby community to mitigate any problems. 

 

I have not heard of any load/bandwidth issues this from RobertK. (he runs the SD-JSON platform)

 

Emby could get more accurate data with mid-day updates... and the system is designed to allow that to be done efficiently.  I do consider this an Emby bug... problems on our service, should take a while to cause problems on client apps.

 

Robert Eden

SD President/CEO

XMLTV

[revengineer 122]

(I might as well chime in directly)

"We don't need SD being mad at Emby"

 

Your loader wiping data is a bigger concern than SD getting "mad".  We're headed by Open Source devs and totally understand OSS developer priorities (*life!*).  If Emby only updates once a day it shouldn't be a problem, and if it does become an issue, we can work with the Emby community to mitigate any problems. 

 

Thank you for the insight. The problem I am having is that I have another IPTV service for which the provider only gives 24 h of guide data in advance. So I need to refresh the guide several times per day. Unfortunately, the guide update refreshes all guides at that cadence even though it would not be necessary to update SD guides as often. Until the guides can be refreshed independently, I apologize for taxing your servers/bandwidth.

[jasonmcroy 315]

I only had this happen once about 2 weeks ago. However, now I have seen it 3 times in the last few days. I get concerned about missing recordings so I have been checking every day about an hour or two before the evening shows start that I record.  

 

Tonight I see another channel missing it's data. I can't chance this anymore and it looks like no one on either side feels it's an issue on their end. I guess I will go back to my earlier set up using the Silicondust DVR. At least that one has never failed me on it's recordings. I just don't like that I can't look within their app and easily see what is set up to be recorded for the night or upcoming nights. That is why I liked using Emby and be able to keep everything within one App. I also feel like functionality I used without issue (able to remux to MKV on the fly as it records) has been taken away so I have to use MCEBuddy regardless  of which DVR I choose to use. 

 

Maybe this will get resolved in the future at which time I may try the DVR portion again. 

[rmeden 10]

If Emby supports XMLTV formatted data, you can use the XMLTV project's tv_grab_zz_sdjson or tv_grab_zz_sdjson_sqlite to download data from SD and generate and efficiently generate an XMLTV file.  If the IPTV data is also in XMLTV format, you can use tv_cat to combine them and then load them both into Emby.

 

One advantage of this solution is you can see the XMLTV formatted file to troubleshoot problem... (SD vs Emby) on missing stations.

 

Robert

Schedules Direct

XMLTV (but I didn't write the XMLTV JSON stuff)

 

[Guest asrequested]

Hang in there, guys. I think Luke has made some adjustments for the next release.

[kanipek 224]

Hang in there, guys. I think Luke has made some adjustments for the next release.

 

This is great news. But the resounding silence from the Emby team is what concerns me. We get a little bit of finger pointing and then nothing. Acknowledge the issue. Tell us it will be resolved - no matter where the issue is.

 

Sounds a little snarky and I apologize for that, very frustrated over this issue. Even more so than the bad old days of watching the guide data in WMC dwindle down to a day or two before Microsoft fixed whatever the issue was each time - again resounding silence from MS as well.

 

For me personally I resolved the issue by setting up an XMLTV source to use as a fallback.

 

Stepping down off my soapbox now...

[kanipek 224]

If Emby supports XMLTV formatted data, you can use the XMLTV project's tv_grab_zz_sdjson or tv_grab_zz_sdjson_sqlite to download data from SD and generate and efficiently generate an XMLTV file.  If the IPTV data is also in XMLTV format, you can use tv_cat to combine them and then load them both into Emby.

 

One advantage of this solution is you can see the XMLTV formatted file to troubleshoot problem... (SD vs Emby) on missing stations.

 

Robert

Schedules Direct

XMLTV (but I didn't write the XMLTV JSON stuff)

 

How does one convert the json data to xml for use in Emby?  I have not found a way to accomplish this - I know there is a way just can't find it.

 

For now I am using the tv_grab_na_dd tool to grab an xmltv formatted xml file.

 

Thanks for reading and any help!

[rmeden 10]

How does one convert the json data to xml for use in Emby?  I have not found a way to accomplish this - I know there is a way just can't find it.

 

For now I am using the tv_grab_na_dd tool to grab an xmltv formatted xml file.

 

Simply use the XMLTV grabbers  tv_grab_zz_sdjson or tv_grab_zz_sdjson_sqlite  instead of tv_grab_na_dd

 

 

Edited November 18, 2017 by rmeden

[kanipek 224]

Simply use the XMLTV grabbers  tv_grab_zz_sdjson or tv_grab_zz_sdjson_sqlite  instead of tv_grab_na_dd

 

 

 

I am trying to do that. But when I add an xml source in Emby using that file I don't get any channels in the channel list (map channels).

 

I do the same process with tv_grab_na_dd and I get a full channel list - I can even edit the file and replace the channel list section so that the mapping is automatic.

 

I know the output of tv_grab_na_dd is in xml but using the json grabber file (emby.cache) it is not. I am guessing it's json but I don't know.  I would love to get this working because I believe the channel ID's and series ID's of the json grabber will

match what I have using the traditional schedules direct source in Emby.

 

Does that make sense?

[CBers 6766]

If someone wants to a create a new thread detailing exactly how to use XMLTV, then it can be pinned in the Live TV forum.

[Luke 37024]

I wouldn't use xml tv as a workaround. We're working on making sure this won't happen anymore for the next release of emby server, thanks.

[CBers 6766]

But xmltv is free isn't it, compared with SD?

 

Perhaps someone wants to use a free service.

[Luke 37024]

Ok sorry I thought the discussion here was converting sd data to XML and then feeding the XML into emby.

[rmeden 10]

XMLTV is lots of things.. in this case 2 things.. a file format (xml schema) and a set of tools (XMLTV project).  Both are open source and free.  The data service used by the a XMLTV grabber may or may not be without cost (in the case of Schedules Direct it isn't.. we have to pay for the data, servers, etc)

 

Sounds like this is under control for now,  but I think the problem using zz_sdjson is the channel format is different from na_dd and that's the expected format.

[kanipek 224]

XMLTV is lots of things.. in this case 2 things.. a file format (xml schema) and a set of tools (XMLTV project).  Both are open source and free.  The data service used by the a XMLTV grabber may or may not be without cost (in the case of Schedules Direct it isn't.. we have to pay for the data, servers, etc)

 

Sounds like this is under control for now,  but I think the problem using zz_sdjson is the channel format is different from na_dd and that's the expected format.

 

So would selecting the na_dd format when running --configure for tv_grab_sd_json fix my issue? 

 

Also what should the output file type be (called)? .cache? .xml?

 

Thanks for any help - sorry to be such a pain, never really messed with this grabber before. I know it is something I am just not grasping because the na_dd grabber is working fine.