Jump to content

Automated Commercial Removal from TV Recordings


PhilWhite

Recommended Posts

sluggo45

Bumping this back in thanks for the useful tips. I got my own variant working and converting videos + removing the .ts from the new file name but could use some pointers for the next idea:

 

I'd like to use ffprobe to check the converted video file then delete the original .ts if all is well (or at least ffprobe thinks all is well). Trying to think of an easy way to do this via the script. Any ideas?

Link to comment
Share on other sites

Baenwort

Bumping this back in thanks for the useful tips. I got my own variant working and converting videos + removing the .ts from the new file name but could use some pointers for the next idea:

 

I'd like to use ffprobe to check the converted video file then delete the original .ts if all is well (or at least ffprobe thinks all is well). Trying to think of an easy way to do this via the script. Any ideas?

 

No ideas but please share your process if you documented it.

 

Or at least the automated .ts removal as I've never gotten PhilWhite's method to work for myself.

Link to comment
Share on other sites

sluggo45

Here's my script, which is really WarrentC3's which he based on Phil's OP in this thread.

 

I cleaned some stuff up for example:

set filename = `printf "%s " $1:t:r:r`

Works better if you use 'basename', which does what WarrentC3 intended (extracts just the file name from the path string returned) but doesn't choke on unexpected characters in the file name. I had a problem with a couple shows that had . in their name (i.e. House Hunters St. Croix) using WarrentC3's script as is. This works better:

set filename = `basename "$1" .ts`

Cleaned up some other minor stuff as well. I was going to just re-do it in Bash because C Shell is ancient, but then, Bash is no spring chicken either and to do some of the more complicated stuff I am thinking of doing next will probably require a more robust scripting environment than a shell....that is a bigger project so in the meantime my full script posted below works fine.

 

  • It will cut commercials and create a _cut.ts file
  • Convert the video (adjust the settings to your liking if you don't want .mp4, etc.)
  • Extract the first embedded CC track if one is present (in the cut file obviously, so they sync correctly with that) and write it out to an .srt file - note I've only tested this with EIA-608/708 North America closed captioning, no idea if it works for other types. This is one function I plan to make a lot more robust...just not using csh :)
  • Checks to see if an mp4 file was created by HandbrakeCLI and if so deletes the _cut.ts and .ts files.

 

Pretty basic but works. In addition to extracting subtitles more elegantly I am also working on checking the mp4 file via ffprobe to insure codec is correct and the duration matches - Handbrake will usually just fail if it encounters a problem (and thus checking for the presence of the mp4 is "good enough" for the basics) but it would be better to actually do a quick diagnostic on the file itself before deleting.

 

Anyway I'll noodle on improving this process but it works well enough now - use at your own risk of course. Script is pretty self explanatory as are my changes:

#!/bin/csh
set path = ($path /usr/local/bin)
set w1 = `jot -r 1 2 15`
set filename = `basename "$1" .ts`
set grab = `printf "%s " $1:h`
set a = `echo "$filename".ts`
set b = `echo "$grab/$filename"_cut.ts`
set c = `echo "$grab/$filename".mp4`
set d = `echo "$grab/$filename".srt`
set e = `echo "$grab/$filename".ts`
sleep $w1

while (1)
if ( ! -e "$b" ) then
echo "Running Comcut"
comcut --ffmpeg=/usr/bin/ffmpeg --comskip=/usr/bin/comskip --lockfile=/tmp/comchap.lock --comskip-ini=/media/Comskip/comskip.ini "$1" "$b"
echo "Comcut complete"
else
echo "Comcut already done"
endif
break
end

while (1)
if ( -e "$b" ) then
echo "Extracting subtitles"
ffmpeg -f lavfi -i "movie=${b}[out0+subcc]" -map 0:1 "$d"
echo "Subtitle extraction complete"
else
echo "No input file found for sub extraction"
endif
break
end

while (1)
if (! `pgrep HandBrakeCLI` ) then
break
endif
echo "HandBrakeCLI is still running. Will try again in a minute."
sleep 60
end

HandBrakeCLI --input "$b" --output "$c" --preset-import-file /media/comchap/hb-dvr.json

while (1)
if (! -e "$c" ) then
echo "Conversion appears to have failed no file found"
break
endif
echo "Cleaning up source files"
sleep 5
rm "$b"
rm "$e"
end
  • Like 1
Link to comment
Share on other sites

  • 3 months later...
Baenwort

So I'm setting this up again on a rebuilt iocage jail and I'm having trouble with the argtable file.

 

When I run:

nano usr/local/libdata/pkgconfig/argtable2.pc

I get a nano window that complains that:

   [ Directory 'usr/local/libdata/pkgconfig' does not exist ]

and attempting to save anyway results in the expected file doesn't exist error:

   [ Error writing usr/local/libdata/pkgconfig/argtable2.pc: No such file or directory ]

I suspect I should create the required directory but I wanted to alert you in case I should not or if it works out you can update the OP.

Link to comment
Share on other sites

Baenwort

Figured out the cause!

 

When you use:

iocage console emby(or your jail name)

you aren't actually at root. You need to enter:

cd /

and you'll be at the point in your jail structure that the setup guide assumes and now your nano command will take you to the right file and location.

Link to comment
Share on other sites

Baenwort

 

59bd95da14bc5_convert.jpg

Update for any other Premier subscribers. It seems that to make this work you have to turn off the built in Emby "Automatically convert recordings to a streaming friendly format

Recordings will be converted on the fly to MP4 or MKV for easy playback on your devices." as it results in a .mkv file being fed into post.sh instead of the .ts file.
 
Now that I've found that problem I'm still testing to see if I can actually get the commercials removed.

 

 

So I can't seem to find this setting any longer. Does anyone know where it moved to? I'm having the files convert to h264 successfully but am still trying to get the commercial cutting to work.

Link to comment
Share on other sites

That option was removed and replaced with the media convert feature, to convert after the recording has completed.

Link to comment
Share on other sites

Baenwort

That option was removed and replaced with the media convert feature, to convert after the recording has completed.

 

So I'm having a stupid moment as I'm not finding any such configurable option?  I've looked under DVR (as that seems most logical under Live TV) and elsewhere.

 

Or is this a case of looking where I don't need as Emby is now smart enough to run my post.sh to convert media and not use an auto convert option that has to be turned off?

Link to comment
Share on other sites

PhilWhite

So I can't seem to find this setting any longer. Does anyone know where it moved to? I'm having the files convert to h264 successfully but am still trying to get the commercial cutting to work.

 

 

Hi,

 

I don't know where that setting is either.  I'm not sure what all is wrong with the commercial cutting, but a question -- are you recording over the air or through a provider?  This will only work for OTA.

 

I also found another problem in the line for removing the .ts.mkv file

 

It had read:

find /path/to/recording -type f iname “*.ts.mkv -exec sh -c mv $0 S{0%.ts.mkv}.mkv”’ {} \;

 

But should actually be:

find /path/to/recording -type f iname “*.ts.mkv -exec sh -c mv $0 ${0%.ts.mkv}.mkv”’ {} \;

 

(the change in red).

 

I thought that I had corrected the other issues, but maybe I missed something?

 

Let me go back and see if there is something equally stupid in the commercial cutting line.

 

 

ETA: one other error in the line to get the script to sleep briefly (in post.sh).  There should have been 2 $$ in that line.  I have corrected both that and the above issue in the original post.  I didn't see anything obviously wrong in the commercial cutting line.

 

Thanks.

 

ETA:  Last issue I noticed -- if you created the comskiplinux directory, then there is an issue in the commercial cutting line:

 

/comchap/comcut --ffmpeg=/usr/local/bin/ffmpeg --comskip=/Comskip-master/comskip --lockfile=/tmp/comchap.lock --comskip-ini=/Comskip-master/comskip.ini "$1"

 

should read /comchap/comcut --ffmpeg=/usr/local/bin/ffmpeg --comskip=/comskiplinux/Comskip-master/comskip --lockfile=/tmp/comchap.lock --comskip-ini=/comskiplinux/Comskip-master/comskip.ini "$1"

 

 

Sorry, I don't generally create the comskiplinux directory, so I left that out.  But if that is where you put Comskip-master, then it needs to be there as part of the path.

 

 

Sorry for all the typos.  I thought I had corrected them, but there were still more.

 

ETA:  Please disregard anything about post_install.sh -- which I have removed.  That has nothing to do with any of this, now that I took a look at it.

Edited by PhilWhite
  • Like 1
Link to comment
Share on other sites

UnderMiner

Hello,

Luke mentioned above that "That option was removed and replaced with the media convert feature, to convert after the recording has completed."

 

I found the option at the recorded program or folder level in the Emby player.

 

May this also be managed using the File Sync plugin and if so, do the global settings for transcode apply to the plugin?

 

Thanks,

Edited by UnderMiner
Link to comment
Share on other sites

PhilWhite

Baenwort,

 

I have these suggestions to make after redoing the commercial cutting process in my newest iocage jail with Freenas 11.2 today.

 

I think many of the needed changes that you noticed and suggested are now in the first post, so that should help.  But I also noticed that commercials were not being cut in my new trial until I did two things -- I think only the first is really necessary but it probably won't hurt to do both.

 

It will require you to ssh into your freenas box unless you set things up in a particular way, but I haven't played with that.  Make sure ssh is turned on under services in Freenas, and get one of the programs that allow you to ssh into your system.  A good option is WinSCP if you use windows.  Putty is great too, but WinSCP allows you to move files really easily.

 

Make sure that your post.sh file is in the same directory/folder as the comchap folder and Comskip-master (I ended up deleting comskiplinux since it isn't really necessary -- just a way to segregate where comskip is compiled).  In my case, I have all three in root, with the pathway being -- /mnt/iocage/jails/emby-server/root/

 

The other thing I noticed, and I don't know if this matters, but the comskip file was not executable (not sure it needs to be), so I gave it full permissions, which if you use WinSCP you can do easily by highlighting the file, clicking properties, and then checking permissions for everything.  Or you can use chmod +x /Comskip-master/comskip in the cli of your jail.  I bet you will need to do cd / first since there are two levels of 'root' in the jail structure, and different parts of this set up were placed in different sections (hence the need to move them).

 

With those alterations I was able to get commercial cutting working today.

 

I hope something in there helps.

 

I think the other option, if you wanted to start from scratch, would be -- once you are in your iocage jail -- to send a cd / command at the beginning and then perform all the other commands in the setup guide.  I didn't have to do this with the warden jails and I didn't run into any problems with the iocage jails until later -- not sure if this was a Freenas or Emby change.  But I think that would work if you wanted to start from scratch.  I just don't think it is necessary since you have already built everything necessary.

 

One thing I would absolutely do, though, if you haven't made a backup copy of your comskip.ini file, on whatever computer you use to ssh in, do so now -- that way you won't need to recreate it.  It is much easier to copy that from one system to another than rewrite it.  I also made a backup copy of my post.sh file.

 

Good luck.

  • Like 1
Link to comment
Share on other sites

PhilWhite

By the way, for anyone interested, Silicondust's new streaming service (which, for right now does not include Food or HGTV -- cable service fights) works great with Emby; and I was able to cut commercials from that stream and convert the file to mkv.  So, it basically allows you to do for 'cable programs' what we have been able to do with OTA.

  • Like 1
Link to comment
Share on other sites

PhilWhite

@sluggo45  quick question.  I was just wondering if this might be worth being converted into a plugin?   

 

I was just the messenger moving it from the Freenas forum, where it was worked out for Plex, to here -- and I screwed up plenty of the lines in translation.  I don't know how adept you are coding, but I was just wondering if it might be worth it?

 

My gut tells me that not that many people are interested.  But what changed my mind, at least a bit, was discovering that it works very well with Silicondust's streaming service.  It's really nice to cut commercials from recorded 'cable' shows.

 

I don't know enough to be able to do it.  Just asking since you've made your own adjustments and might know how to go about it.

 

It would be nice for folks to not have to go to so much trouble to get a commercial cutting program working.  I guess the bigger issue might be getting something to work across different OSs.  Having it integrated into Emby in any way would make the TV experience better, I think.

Link to comment
Share on other sites

Baenwort

At one point @@Luke mentioned that he was interested in the feature being added but he didn't think it was a high priority.

 

I'd say a plug-in would be the first step as that would make it easy enough that a better judgement of usage could be figured out. It seems to be straight forward enough that if someone who had the time wanted to take it up it could be done. Once we found the right options it seems to be stable and other than when I moved from Warden to iocage I never had an issue.

  • Like 1
Link to comment
Share on other sites

  • 3 weeks later...

Hi All,

 

I completed install OK.

 

I'm trying to test by running manually (from terminal) first.

 

Initially, only doing the comskip\comcut step of the post.sh

I have commented out the Handbrake stuff.

 

The comskip portion appears to run OK, but when I get to next part (I'm guessing something with ffmpeg) I'm having issues.

 

I'm seeing a bunch of lines (guessing 50-100) with either "mpeg buffer underflow" or "mpeg packet too large".

 

My source files came from my HDHomeRun Prime, and I'm using ChannelsDVR.

My server is FreeNAS 11.2, the ChannelsDVR server is hosted within 11.2 iocage jail 

 

Any suggestions/ideas I can try to resolve?

Edited by djronh1
  • Like 1
Link to comment
Share on other sites

PhilWhite

Hi All,

 

I completed install OK.

 

I'm trying to test by running manually (from terminal) first.

 

Initially, only doing the comskip\comcut step of the post.sh

I have commented out the Handbrake stuff.

 

The comskip portion appears to run OK, but when I get to next part (I'm guessing something with ffmpeg) I'm having issues.

 

I'm seeing a bunch of lines (guessing 50-100) with either "mpeg buffer underflow" or "mpeg packet too large".

 

My source files came from my HDHomeRun Prime, and I'm using ChannelsDVR.

My server is FreeNAS 11.2, the ChannelsDVR server is hosted within 11.2 iocage jail 

 

Any suggestions/ideas I can try to resolve?

Are you recording through Emby and producing a .ts file or recording through Channels DVR?  If the latter, what type of file does it produce?  

Link to comment
Share on other sites

Are you recording through Emby and producing a .ts file or recording through Channels DVR?  If the latter, what type of file does it produce?  

 

 

I'm using ChannelsDVR, and it produces an mpg file

 

using mediainfo within jail, it list 

 

Format                                   : MPEG-TS

File size                                : 22.8 GiB

Duration                                 : 3 h 35 min

Overall bit rate mode                    : Variable

Overall bit rate                         : 15.1 Mb/s

 

Format                                   : MPEG Video

Format version                           : Version 2

Edited by djronh1
Link to comment
Share on other sites

PhilWhite

It produces an mpg file

 

using mediainfo within jail, it list 

 

Format                                   : MPEG-TS

File size                                : 22.8 GiB

Duration                                 : 3 h 35 min

Overall bit rate mode                    : Variable

Overall bit rate                         : 15.1 Mb/s

 

Format                                   : MPEG Video

Format version                           : Version 2

Thanks.  So, the file format is fine.  It's a big file, but I don't know that would be a problem (though it might be).

 

From the comskip site:

 

Input files

Comskip can operate on 3 types of files. The first file type is an MPEG file. The extension must be mpg, mpeg (for mpeg Program Streams), ts (for mpeg Transport Streams) or dvr-ms for DVR-MS). Maximum resolution is 2000 by 1200 and the content must be MPEG2 video and MPEG2 or AC3 audio as recorded by most mpeg encoders. When an mpeg file is provided as input Comskip will do a full decoding and commercial detection.  When the demux switch is used Comskip will demux the mpeg input file into a separate audio and video elemtary stream suitable for many editor tools. A second input file type is a csv file. This file can be created by a previous run of Comskip by setting output_framearry=1 or using the –csvout parameter. This type of input file is useful when Comskip must be run multiple times to tune certain parameters. This skips the mpeg decoding and results in a substantial faster execution. The third type of input file is a .txt file with the commercial cutpoints as generated by a previous run of Comskip. When this type of input file is used Comskip will open the debugwindow and it is possible to review quickly the detected commercials and modify the cutpoints if required and write a new .txt file.

 

 

So, it may depend on what type of audio is recorded as well.  I doubt there is a mismatch though.

 

Do you have a smaller file to run through it?  Say, something that is 30 min or 1 hour?

 

ETA:  The other thing to check is what your comskip.ini file looks like.  If there are wrap-arounds in that file it tends to throw all sorts of error messages.  All commands in that file need to be on their own line.

Edited by PhilWhite
Link to comment
Share on other sites

 

So, it may depend on what type of audio is recorded as well.  I doubt there is a mismatch though.

 

Do you have a smaller file to run through it?  Say, something that is 30 min or 1 hour?

 

ETA:  The other thing to check is what your comskip.ini file looks like.  If there are wrap-arounds in that file it tends to throw all sorts of error messages.  All commands in that file need to be on their own line.

 

 

Here's my audio info - 

 

Audio #1

ID                                       : 4220 (0x107C)

Menu ID                                  : 10 (0xA)

Format                                   : AC-3

Format/Info                              : Audio Coding 3

Commercial name                          : Dolby Digital

Codec ID                                 : 129

Duration                                 : 3 h 35 min

Bit rate mode                            : Constant

Bit rate                                 : 384 kb/s

Channel(s)                               : 6 channels

Channel layout                           : L R C LFE Ls Rs

Sampling rate                            : 48.0 kHz

Frame rate                               : 31.250 FPS (1536 SPF)

Bit depth                                : 16 bits

Compression mode                         : Lossy

Delay relative to video                  : -759 ms

Stream size                              : 592 MiB (3%)

Language                                 : English

Service kind                             : Complete Main

 

While I could try recording a shorter program (30 mins - 1 hour), that would be a deal breaker for me (if that is only option). Since the only thing I DVR is sports (mainly football), with most events running minimum of 2 1/2 - 3 hours.

Edited by djronh1
Link to comment
Share on other sites

PhilWhite

Here's my audio info - 

 

Audio #1

ID                                       : 4220 (0x107C)

Menu ID                                  : 10 (0xA)

Format                                   : AC-3

Format/Info                              : Audio Coding 3

Commercial name                          : Dolby Digital

Codec ID                                 : 129

Duration                                 : 3 h 35 min

Bit rate mode                            : Constant

Bit rate                                 : 384 kb/s

Channel(s)                               : 6 channels

Channel layout                           : L R C LFE Ls Rs

Sampling rate                            : 48.0 kHz

Frame rate                               : 31.250 FPS (1536 SPF)

Bit depth                                : 16 bits

Compression mode                         : Lossy

Delay relative to video                  : -759 ms

Stream size                              : 592 MiB (3%)

Language                                 : English

Service kind                             : Complete Main

 

While I could try recording a shorter program (30 mins - 1 hour), that would be a deal breaker for me (if that is only option). Since the only thing I DVR is sports (mainly football), with most events running minimum of 2 1/2 - 3 hours.

I doubt the problem is the length, just trying to trouble shoot.

 

The audio is also fine, so comskip should be able to deal with the file.  One other potential issue -- do you know if the streams you get through Channels DVR have copy flags on them -- like most cable providers provide?  If a program has a copy flag on it, like copy 0 or copy 1 then you can't cut commercials from it.

 

That might be your problem since you have a prime and the prime uses a cable card.

Link to comment
Share on other sites

One other potential issue -- do you know if the streams you get through Channels DVR have copy flags on them -- like most cable providers provide?  If a program has a copy flag on it, like copy 0 or copy 1 then you can't cut commercials from it.

 

That might be your problem since you have a prime and the prime uses a cable card.

 

 

How would I check for copy flag?

 

BTW - I previously had a MythTV server setup, which also has a commercial removal feature, and never had an issue with that (using same my HDHomerun Prime).

 

Only time I had issue trying to DVR stuff (on MythTV) was when trying to DVR premium channels (like HBO, Showtime, etc).

 

Also, not sure if this means anything, but I'm able to view all my ChannelsDVR shows on Plex, and have also had no issues convert mpg file to X264 or X265.

 

Lastly, here something I found on ChannelsDVR FAQ section - https://getchannels.com/faq/

 

I verified, none of my channels have DRM

 

 

Does Channels support DRM protected cable channels?

DRM channels are not currently supported. We are working on DTCP-IP certification and plan to support DRM channels in the future.

DRM, or copy-once, is generally only enabled on premium content channels like Showtime and HBO. However, some cable providers are known to DRM their entire lineup (notabily TWC, but only in some regions). As of mid-2015, Verizon FIOS has enabled DRM on FOX affiliate channels like Fox News and Fox Sports.

To see which of your channels are laced with DRM:

  • visit my.hdhomerun.com
  • click on your PRIME
  • click on “Channel Lineup”
  • look for any channels with the DRM badge.
Edited by djronh1
Link to comment
Share on other sites

PhilWhite

 

How would I check for copy flag?

 

BTW - I previously had a MythTV server setup, which also has a commercial removal feature, and never had an issue with that (using same my HDHomerun Prime).

 

Only time I had issue trying to DVR stuff (on MythTV) was when trying to DVR premium channels (like HBO, Showtime, etc).

 

Also, not sure if this means anything, but I'm able to view all my ChannelsDVR shows on Plex, and have also had no issues convert mpg file to X264 or X265.

 

Lastly, here something I found on ChannelsDVR FAQ section - https://getchannels.com/faq/

 

I verified, none of my channels have DRM

 

 

Does Channels support DRM protected cable channels?

DRM channels are not currently supported. We are working on DTCP-IP certification and plan to support DRM channels in the future.

DRM, or copy-once, is generally only enabled on premium content channels like Showtime and HBO. However, some cable providers are known to DRM their entire lineup (notabily TWC, but only in some regions). As of mid-2015, Verizon FIOS has enabled DRM on FOX affiliate channels like Fox News and Fox Sports.

To see which of your channels are laced with DRM:

  • visit my.hdhomerun.com
  • click on your PRIME
  • click on “Channel Lineup”
  • look for any channels with the DRM badge.

 

OK, good, so none of that is an issue.

 

What does your comskip.ini file look like? and what is the path to it?

 

 

ETA:  I tried to upload my copy of comskip.ini but it won't let me upload that file type.  It could be a problem with ffmpeg, but I would bet the issue is with comskip.ini.  Regarding ffmpeg, which version of Emby are you using?  If it is 3.6 or above, then there shouldn't be a problem with ffmpeg and Freenas 11.2.

Edited by PhilWhite
Link to comment
Share on other sites

OK, good, so none of that is an issue.

 

What does your comskip.ini file look like? and what is the path to it?

 

 

I'm using the comskip.ini file for you initial post, and verified my paths (as well as permissions) are fine.

 

Anyways, I just took a closer look at one of the files I ran post.sh against, and it looks like it actually worked fine.

 

So I'm guessing I can probably just ignore the various warnings, since end result is good.

 

Next step will be to uncomment that Handbrake lines, but I'm guessing that won't be an issue, since I've been using HandbrakeCLI without issues for months now.

 

Thanks again for all your help, and sorry for the false alarm

Edited by djronh1
Link to comment
Share on other sites

PhilWhite

I'm using the comskip.ini file for you initial post, and verified my paths (as well as permissions) are fine.

 

Anyways, I just took a closer look at one of the files I ran post.sh against, and it looks like it actually worked fine.

 

So I'm guessing I can probably just ignore the various warnings, since end result is good.

 

Next step will be to uncomment that Handbrake lines, but I'm guessing that won't be an issue, since I've been using HandbrakeCLI without issues for months now.

 

Thanks again for all your help, and sorry for the false alarm

 

Great, glad it works. 

 

Googling a bit, I think I found the source of the problem.  It appears that ffmepg interprets .mpg extensions as an MPEG-1 system stream and expects that particular buffer size.  Your files are all MPEG-2 which, I think, has a different buffer size, so that is probably the source of the error messages.

 

But if it works, I guess problem solved.

  • Like 1
Link to comment
Share on other sites

Great, glad it works. 

 

Googling a bit, I think I found the source of the problem.  It appears that ffmepg interprets .mpg extensions as an MPEG-1 system stream and expects that particular buffer size.  Your files are all MPEG-2 which, I think, has a different buffer size, so that is probably the source of the error messages.

 

But if it works, I guess problem solved.

 

 

Hmm ... thank for the info.

 

So what extension should I use so that ffmpeg knows that files are MPEG-2?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...