Channel Mapping (again)

[danmarcoux 6]

OK, I've checked the Knowledge Base and according to it Channel Mapping should just automatically work.   I've looked here on the forums and there are lots questions about how Mapping works, but I haven't seen a good answer.  There are several "guess and by golly" messages.   

Simple question: How Does It Work?

Simple example:

One entry in my M3U file to test:

#EXTM3U url-tvg=""
#EXTINF:-1 tvg-id="AMC.us" tvg-name="AMC (AMC)" tvg-logo="http://www.ourbunch.net/tv_logos/1-AMC.png" group-title="United States",AMC (AMC)
http:/bleah/bleah/bleah/12345

What it matched to:

 

I've got several Guide Data Sources setup, AMC should be in the first one.  GBR-10000193-DEFAULT is second to last.  (The last one, my own XmlTV EPG, also contains AMC.us, so it would match there, too.)

The log file is HUGE (It's recording a TV Show, somebody is watching a movie, and it's searching for missing subtitles), but these look like the relevant entries:

Quote

2023-03-07 12:59:05.994 Info Server: http/1.1 POST http://localhost:8096/emby/LiveTv/TunerHosts?X-Emby-Client=Emby Web&X-Emby-Device-Name=Google Chrome Windows&X-Emby-Device-Id=TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwOC4wLjAuMCBTYWZhcmkvNTM3LjM2fDE2NzIyNTc4NDA3NzE1&X-Emby-Client-Version=4.7.11.0&X-Emby-Language=en-us&reqformat=json. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36
2023-03-07 12:59:05.998 Info Server: http/1.1 Response 200 to ::1. Time: 3ms. http://localhost:8096/emby/LiveTv/TunerHosts?X-Emby-Client=Emby Web&X-Emby-Device-Name=Google Chrome Windows&X-Emby-Device-Id=TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwOC4wLjAuMCBTYWZhcmkvNTM3LjM2fDE2NzIyNTc4NDA3NzE1&X-Emby-Client-Version=4.7.11.0&X-Emby-Language=en-us&reqformat=json
2023-03-07 12:59:06.006 Info HttpClient: GET http://192.168.0.2:34400/x_path8_x/x_path9_x
2023-03-07 12:59:06.213 Info LiveTV: Found epg channel in XmlTV  AMC.us AMC AMC.us  (that XmlTV is the last)
...
2023-03-07 12:59:06.218 Info LiveTV: Found epg channel in Emby Guide Data USA-NC32414-X AMC 10021 (that Emby Guide Data is the first)
...
2023-03-07 12:59:06.607 Info LiveTV: Found epg channel in Emby Guide Data USA-DISH501-DEFAULT AMC 10021 (that Emby Guide is the third from the last)
---
2023-03-07 12:59:06.623 Info LiveTV: Found epg channel in Emby Guide Data GBR-1000193-DEFAULT 3SATGEH 66621  (The incorrect match!  That Emby guide is the second from the last.  And, by the way, has no AMC entry in it.)

And it doesn't stop there.... it continues processing GBR-1000193 before finally stopping.

(All of the entries say "Found epg channel in Emby Guide Data (or XmlTV)..." and lists every channel in that guide.

 

So I'm at a lost.   I would love to use Emby Guide Data, but if it's not going to match a large percentage of the channels, why would I?  And Emby, for a reason I have yet to determine, sometimes decides to run the Map Channel routine again and wipes out all of the manual mapping I had to do. 

 

Any clues?  Please.

 

[Luke 37066]

HI, is the amc channel in that list of channels to pick from when mapping?

[Luke 37066]

And secondly, how did you configure the m3u tuner?

[danmarcoux 6]

If I just select "Guide data channel" before changing "Guide data source", then no it isn't there (there is an AMCROI).   If I change "Guide data source" to a source that I know has the AMC channel, then I can find it (by channel number, not by name only. ugh )

[danmarcoux 6]

2 minutes ago, Luke said:

And secondly, how did you configure the m3u tuner?

Live TV -> Setup -> Add TV Source -> M3U -> File or url: = <my playlist, local file - just one channel (AMC) for testing, as in the first post)>

 

[Luke 37066]

6 minutes ago, danmarcoux said:

Live TV -> Setup -> Add TV Source -> M3U -> File or url: = <my playlist, local file - just one channel (AMC) for testing, as in the first post)>

 

Can you show a screenshot?

[Luke 37066]

9 minutes ago, danmarcoux said:

If I just select "Guide data channel" before changing "Guide data source", then no it isn't there (there is an AMCROI).   If I change "Guide data source" to a source that I know has the AMC channel, then I can find it (by channel number, not by name only. ugh )

So based on the channels that are in your guide sources, what channel did you expect it to map to?

[danmarcoux 6]

4 minutes ago, Luke said:

Can you show a screenshot?

No problemo....

 

 

[danmarcoux 6]

4 minutes ago, Luke said:

So based on the channels that are in your guide sources, what channel did you expect it to map to?

Well, ideally, I would have expected Emby to look in the first Guide Source I added (USA-NC32414-X), found that the channel AMC existed, and mapped to that one.

 

[Luke 37066]

It does, assuming the guide source is enabled for the tuner, but it's looking for an exact name match and so that's why that happened.

[danmarcoux 6]

OK, so kinda weird results.   I added several Emby Guides for some random OTA USA Zip Codes and Canada (Montreal).  I added Spectrum, Dish, DirecTV, and UK Sky Network (London).  I also added 2 XMLTV local files (one for just the channels I subscribe to, and one for all of the Plex "Free" Channels (just because I had it handy)).  Finally I added a XMLTV http file from xTeVe.   The xTeVe Guide was set for only the xTeVe tuner, the rest were set for All Tuners.

This was the order:

1.) Spectrum
2.) OTA 99201
3.) OTA 60007
4.) Custom XML
5.) OTA 64030
6.) xTeVe XML
7.) Plex.tv XML
8.) OTA Canada
9.) DirectTV
10) Dish
11) UK Sky

(Several were pre-existing, some I just added for this test.)

I then added my dummy playlist.  I changed it a bit to ensure nothing was cached (as far the playlist went) from the last time I tried it.
Looking at the Emby logs, it searched the Guides in this order:

4.) Custom XML
1.) Spectrum
2.) OTA 99201
3.) OTA 60007
5.) OTA 64030
6.) xTeVe XML
7.) Plex.tv XML
8.) OTA Canada
9.) DirectTV
10) Dish
11) UK Sky

(The first digit is the order that they are listed in Emby.  So, basically, it searched the Custom XML first, then searched the rest of the Guides in order.  Including the xTeVe Guide, which was *not* marked for all tuners.)

Then I took the Custom XML out (and Emby did remapping, with most going to random channels in the UK Guide and changing all of my beautiful custom logos  ) and then added it back in.  My new order was now:

1.) Spectrum
2.) OTA 99201
3.) OTA 60007
4.) OTA 64030
5.) xTeVe XML
6.) Plex.tv XML
7.) OTA Canada
8.) DirectTV
9.) Dish
10) UK Sky
11) Custom XML

I removed my dummy playlist.  Again, I changed it to avoid any caching, then added it back in.   This time Emby searched in the order expected, skipping over the xTeVe Guide.  But, for good measure I guess, it added the xTeVe guide to the end of the search list.  So it still search xTeVe.

According to the logs, Emby only looks for an exact match in the first <display name> entry for XML files.  In Emby Guides it looks like also looks for an exact match.  In either case, it looks like Emby is looking for the Broadcast ID (a 2 to 7 identifier for the channel.  There's probably a lookup somehwere, but I haven't been able to find it), so unless somebody can find the lookup, you need to know that "NIK" is "Nickelodeon" or that "STZK" (or "STZKHD" - I'll touch on that in a moment) is "Starz Kids & Family".   (So, taking "Starz Kids & Family" as an example, Spectrum, Dish, and DirecTV all carries it.   STZK was the orginal name, when broadcasters fully embraced HD they starting adding HD channles and STZK was available in HD on STZKHD. Spectrum only lists STZK, DirecTV only lists STZKHD, Dish lists both STZK and STZKHD.  Wonderful.)   

Also, as far local channels go, Spectrum only lists the station id - WBTV, for example.  The OTA feeds list the new(!) digital channel ID: WBTVDT.  DirecTV and Dish list both WBTV and WBTVDT.

I didn't go any further than OTA, Spectrum, DirectTV, and Dish.  I have no idea what the other carriers (Comcast, AT&T, Verizon, etc.) do.

 

[danmarcoux 6]

FYI, "Broadcast ID" is also known as "Callsign" and "Call Letters".   This looks like a pretty accurate list for the US, but I don't now how current it is.

https://www.ascap.com/help/royalties-and-payment/payment/cable-tv-service-abbreviations

 

[Luke 37066]

Quote

Looking at the Emby logs, it searched the Guides in this order:

4.) Custom XML
1.) Spectrum

why do you think it searched it in that order?

[Luke 37066]

Actually we look for exact matches against a list of possible fields:

• name
• callsign
• alternate names (supported by some xmltv guides).

[danmarcoux 6]

11 hours ago, Luke said:

why do you think it searched it in that order?

The log file is pretty good about telling you.  The times stamp for XmlTV entries were before the Emby Guide Data entries.

 

2023-03-07 16:09:05.629 Info LiveTV: Found epg channel in XmlTV  5StarMax.us 5MAX 5StarMax.us
2023-03-07 16:09:05.629 Info LiveTV: Found epg channel in XmlTV  ABCNewsLive.us KABCDT2 ABCNewsLive.us
2023-03-07 16:09:05.629 Info LiveTV: Found epg channel in XmlTV  AMC.us AMC AMC.us
....
2023-03-07 16:09:05.758 Info LiveTV: Found epg channel in Emby Guide Data GBR-1000193-DEFAULT 123TVGR 53409
2023-03-07 16:09:05.758 Info LiveTV: Found epg channel in Emby Guide Data GBR-1000193-DEFAULT 123TVGH 90494
2023-03-07 16:09:05.758 Info LiveTV: Found epg channel in Emby Guide Data GBR-1000193-DEFAULT 3SATGEH 66621

 

[danmarcoux 6]

11 hours ago, Luke said:

Actually we look for exact matches against a list of possible fields:

• name
• callsign
• alternate names (supported by some xmltv guides).

So using a XMLTV guide, it *looks* to be pretty simple.  Modify the first <display name> entry to be the exact match of the callsign.  The guide will look fine because because besides the TV Channel icon (or if you change "Channel display" to be Title instead of Both), it will display the Description given in the M3U file (tvg-name).

 But for Emby Guides, if one were to change the tvg-name to just be the call sign, that *could* be problematic.  Most local channels, and a lof of cable/satellite channels are pretty descriptive, but there are some that take a little deciphering.

OR - do you use the tvg-id to match the callsign?  In the US, a lot of tvg-id IDs end with .us - "AMC.us" for example.  Does the matching algorythm strip the .us (.uk .gr etc.), or does it expect the tvg-id to not have the extension.  (That totally goes against the specification by the way, which specifies that the channel id should look like a Top Level Domain (TLD), even though it isn't one.)

OR - is there a "special" field that only Emby looks for that will guarantee a match?  Now that would be great.   

 

Also, the "dark image" for the channels seems to corresponed with the  tvg-logo field in M3U or the "icon src" field for the channel in the XMLTV file. (I haven't figured out yet which one takes precedence, but I will.  ).  Is there any way to specify the "light" and "light with color" images?  Emby seems to prefer those in the default "dark" mode.

 

[Luke 37066]

Quote

 

Also, the "dark image" for the channels seems to corresponed with the  tvg-logo field in M3U or the "icon src" field for the channel in the XMLTV file. (I haven't figured out yet which one takes precedence, but I will.  https://cdn.jsdelivr.net/gh/twitter/twemoji@14.0.2/assets/72x72/1f600.png).  Is there any way to specify the "light" and "light with color" images?  Emby seems to prefer those in the default "dark" mode.

For xml, no. It is only emby guide data that supports multiple logo variations. It's not that we couldn't do it in xml, it's just that it would have to be something proprietary.

[Luke 37066]

Quote

OR - do you use the tvg-id to match the callsign?  In the US, a lot of tvg-id IDs end with .us - "AMC.us" for example.  Does the matching algorythm strip the .us (.uk .gr etc.), or does it expect the tvg-id to not have the extension.  (That totally goes against the specification by the way, which specifies that the channel id should look like a Top Level Domain (TLD), even though it isn't one.)

We match name against name, callsign against name, and guide id (tvg-id) against id. We might be able to improve this a little by cross matching name and id together in case the name is treated like id and vice versa.

We don't remove data from the id's, but all of the matching normalizes symbol characters so that the presence or lack thereof of a dash, period, etc, doesn't cause matches to not happen.

[danmarcoux 6]

11 minutes ago, Luke said:

For xml, no. It is only emby guide data that supports multiple logo variations. It's not that we couldn't do it in xml, it's just that it would have to be something proprietary.

Having something propietary in there is likely to break several editors and such.   Probably not a good idea. 
Looking at the DTD, there also isn't support for mulitiple "icon src" lines, which would be ideal.

One option might be use the "icon src" line, but the src it references doesn't have to be a graphic file.  It could be a CGI script that returns up to three different files.
<icon src="http://domain.whatever/logos/ParamountNetwork.png" />
would become:
<icon src="http://domain.whatever/logos/LogoServer.cgi?LogoDark=ParaDark.png&LogoLight=ParaLight.png&LogoColor=ParaColor.png" />
(or whatever)

Emby would want to make that public knowledge so that developers could support that (maybe even go so far as providing LogoServer.cgi for serveral OSes).  First to market, I bet even your competitors might look at implementing that too....

 

[danmarcoux 6]

21 minutes ago, Luke said:

We match name against name, callsign against name, and guide id (tvg-id) against id. We might be able to improve this a little by cross matching name and id together in case the name is treated like id and vice versa.

We don't remove data from the id's, but all of the matching normalizes symbol characters so that the presence or lack thereof of a dash, period, etc, doesn't cause matches to not happen.

OK, I will test this later, but just to make sure I have this right... assuming the provider provides a name (which might not actually be shown in the emby logfile, just the callsign is in the log file) and the ID is the number in the log file directly after the callsign (I notice that number is the same regardless of what the provider is).

Log file entry for Animal Planet:
2023-03-07 16:09:05.616 Info LiveTV: Found epg channel in Emby Guide Data USA-DISH501-DEFAULT APL 16331

From the Emby Guide (going from the log file):
Name: Animal Planet (Not seen in the log file, so kind of guessing here)
CallSign: APL
ID: 16331

The following in the M3U should all provide matches:
#EXTINF:-1 tvg-id="AnimalPlanet.us" tvg-name "Animal Planet" tvg-logo="https://xyz.com/ap.png" group-title="",Animal Planet
(Name matches tvg-name)

#EXTINF:-1 tvg-id="AnimalPlanet.us" tvg-name "APL" tvg-logo="https://xyz.com/ap.png" group-title="",APL
(CallSign matches tvg-name)

#EXTINF:-1 tvg-id="16331" tvg-name "The Animal Planet Channel" tvg-logo="https://xyz.com/ap.png" group-title="",The Animal Plant Chanel
(ID matches tvg-id)

 

Does that look right?

 

[danmarcoux 6]

On 3/8/2023 at 1:51 PM, danmarcoux said:

 

Log file entry for Animal Planet:
2023-03-07 16:09:05.616 Info LiveTV: Found epg channel in Emby Guide Data USA-DISH501-DEFAULT APL 16331

CallSign: APL
ID: 16331

#EXTINF:-1 tvg-id="16331" tvg-name "The Animal Planet Channel" tvg-logo="https://xyz.com/ap.png" group-title="",The Animal Plant Chanel
(ID matches tvg-id)

 

Finally got around to doing some quick testing this afternoon.  In my test M3U containing 15 channels, I'm happy to report this one had a 100% accuracy rate!

Changing the "tvg-name" to the Name was maybe 30% accurate.  I just don't know what the name is supposed to be.

I didn't get a chance to try changing the "tvg-name" to the CallSign.  I don't like/want to change my description of the channel in tvg-name, but I will for this test.  

I expect to go with the ID/tvg-id route.  I see wee bit of scripting in my future, but it really shouldn't be tough since I already have the CallSigns - I'll probably just write a BASH script instead of Python.

 

Edited March 9, 2023 by danmarcoux

[Luke 37066]

Thanks for testing that. I think for another round of improvements to the matching we should make name, callsign and id's all interchangeable in the event that we're dealing with either an m3u or xml data that is put together unexpectedly.

[danmarcoux 6]

5 hours ago, danmarcoux said:

Finally got around to doing some quick testing this afternoon.  In my test M3U containing 15 channels, I'm happy to report this one had a 100% accuracy rate!

Changing the "tvg-name" to the Name was maybe 30% accurate.  I just don't know what the name is supposed to be.

I didn't get a chance to try changing the "tvg-name" to the CallSign.  I don't like/want to change my description of the channel in tvg-name, but I will for this test.  

I expect to go with the ID/tvg-id route.  I see wee bit of scripting in my future, but it really shouldn't be tough since I already have the CallSigns - I'll probably just write a BASH script instead of Python.

 

 

On 3/8/2023 at 1:51 PM, danmarcoux said:

Log file entry for Animal Planet:
2023-03-07 16:09:05.616 Info LiveTV: Found epg channel in Emby Guide Data USA-DISH501-DEFAULT APL 16331

From the Emby Guide (going from the log file):
Name: Animal Planet (Not seen in the log file, so kind of guessing here)
CallSign: APL
ID: 16331

Sample Line in Original M3U Playlist:
#EXTINF:-1 tvg-id="AnimalPlanet.us" tvg-name "Animal Planet" tvg-logo="https://xyz.com/ap.png" group-title="",Animal Planet

 

OK, did testing on 4 different scenarios.  Each time I tested I reset the M3U file back to the original so I was only changing one attribute at a time.  My sample M3U file had 15 entries.

1) Change tvg-name to what I would think would be the name (so Animal Planet in the above scenario)
2) Change tvg-id to ID
3) Change tvg-id to CallSign
4) Change tvg-name to CallSign

Scenario 1: Just under 50% accuracy rate.  Like I mentioned in a previous post, not knowing what the Emby is looking for as far as a name goes hurts this.

Scenario 2: 100% accuracy rate.   Finding the IDs (Channel ID?) either requires getting all entries for Live TV out of the Emby log file.  Or, for the more adventurous, you can find this information using utilties like mc2xml or zap2xml.  There may be other online resources, too.  The ID seems to be universal.  Not knowing the CallSign makes this next to impossible.

Scenario 3: 0% accuracty rate.   Just doesn't work.

Scenario 4: 100% accuracy rate.  Finding the CallSign will require a little bit of detective work.  The CallSign seems to be universal (like the ID), but be warned that there are multiple variants of the CallSig: Local channels will like have DT appended to the station name so WSOC will be WSOCDT or KREM will be KREMDT, not all cable or satellite providers will have the DT variant, or only the DT variant, and some may have both.  And there are can be HD variants, too: CH5 and CH5HD, for example.  Again some cable or satellite providers will have the HD variant, or only the HD variant, and some may have both.  (They could, in rare instances, also have slightly different programming - so the guide data may not be accurate.)

So, in the end, "guess and by golly" rules the roost.  But, I think knowing this much will help. I know it'll help me, but I only have about 150 channels in my playlist - I know some people have a whole lot more. I also did some footwork and I have Call Signs for all of my channels.  I would like to think there is a lookup tool online somewhere where you could be a channel name in (or a partial name) and have it return out the Call Sign and the ID.  I know it would have saved me a ton of time.   I'd build one, but that would take even *more* time!  The list of channels worldwide has got to be huge. The file that I built contains about 4000 Call Signs/IDs - no names - and it only has channels for US, Canada (partial list), and Great Britain (may be partial list).

 

[danmarcoux 6]

FYI - I found TV Listings on zap2it.com show the CallSign.  Just select a zip/postal code, provider, and boom.  It doesn't show the name of the channel, but it does have the channel logo (logos for a channels where either a logo doesn't exist or they don't have one, well, the "missing logo" logo looks *awfully* familiar.  .  Should help people a lot.

 

[Luke 37066]

 Support for tvg-url and url-tvg is in Emby Server 4.8.3+: