Jump to content
ShotToTheDome

Emby post-processing dvr script for linux

Recommended Posts

ShotToTheDome

I thought I'd post my emby post-processing dvr script. Tested and working on Debian Linux 8.9

Files processed end up being named *originalvideoname*.ts.mkv so i can search by that and make sure recordings have no issues. After I look at them, I manually rename to just .mkv. I do not have automatically convert recordings checked or the two preserve boxes checked under live tv --> settings.

Script flow:

  • see if lockfile exists. if so wait 60 sec
  • create lockfile
  • commerical scan and commerical cutting
  • ccextractor to pull out closed captions and convert to srt file
  • Encode to h264 using ffmpeg and combine in srt file from previous step
  • Trim off first 60 sec from recording (due to iptv delay you get a minute from previous show. I have recordings set to last 1 min longer in emby record. The shows using my OTA tuners start early by 1 min to compensate)
  • Remove srt/extra files
  • Overwrite original ts recording
  • delete lockfile

Depends on: comskip, comcut (comchap optional), ccextractor, ffmpeg

I eventually want to fool around with a nvidia gtx 1050 and see how nevnc with hevc/h265 works on the ffmpeg encoding line. Software h265 encoding on very fast saves an additional 30-40% space on same setting as h264 on medium but my encoding framerate goes from 420fps (14x) to 69fps (2.3x) on dual Xeon e5-2670s. The picture looks pretty much identical when i look at them side by side

post.sh emby script

#!/bin/csh

set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/path/to/embydvr/dvrProcessing.log'
set dvrLockLog = '/path/to/embydvr/dvrLock.log'


#Wait if post processing is already running
while ( -f $lockFile )
    echo "'$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end

#Create lock file to prevent other post-processing from running simultaneously
echo "Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
echo "cut from '$origFile'" | tee -a $dvrPostLog
/path/to/embydvr/comchap/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/path/to/embydvr/comskip.ini "$origFile"

#Pull CC from file to SRT file
echo "Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/usr/bin/ffmpeg -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -acodec copy -scodec copy -c:v libx264 -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"

#Trim off first minute
echo "Remove first 60 sec of file" | tee -a $dvrPostLog
/usr/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"

#Remove tmpEncode file
echo "Remove tmpEncode file" | tee -a $dvrPostLog
rm -f "$tmpEncode"

#Rename transcoded file to temp file in case no subtitles
echo "Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode2" "$tmpFile"

#Overwrite original ts file with the transcoded file
echo "Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"

#Rename .ts file to .ts.mkv
echo "Renaming '$origFile' to .mkv" | tee -a $dvrPostLog
mv -f "$origFile" "$1.mkv"

#Remove lock file
echo "Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile

exit 0
Edited by ShotToTheDome
  • Like 2

Share this post


Link to post
Share on other sites
ShotToTheDome
Installing the programs needed by the script assuming Ubuntu/Debian

 

CCextractor


clone ccextractor from git source

go to directory you want to use. I'll call mine /media/ssd/



cd /media/ssd
git clone https://github.com/CCExtractor/ccextractor.git


This will pull in the ccextractor code to new directory called ccextractor



cd ccextrator


for ubuntu/debian - download dependencies: 



sudo apt-get install -y gcc libcurl4-gnutls-dev tesseract-ocr tesseract-ocr-dev libleptonica-dev


now we need to go into the build directory



cd /media/ssd/ccextractor/linux
./build
 
# test your build
./ccextractor


now we need to place ccextractor where we can call it from a built in path

sudo cp ccextractor /usr/local/bin/ccextractor

 

Comskip


clone comskip from git source



cd /media/ssd




git clone https://github.com/erikkaashoek/Comskip.git


go to new Comskip directory

install build requirements



cd Comskip
sudo apt-get install -y autoconf automake libtool


build comskip



./autogen.sh
./configure


here i ran into a missing package argtable2



cd /tmp
wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-0_12-1.1_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-docs_12-1.1_all.deb
wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-dev_12-1.1_amd64.deb
sudo dpkg -i libargtable2-0_12-1.1_amd64.deb libargtable2-docs_12-1.1_all.deb libargtable2-dev_12-1.1_amd64.deb


also more missing packages:

sudo apt-get install -y libavutil-dev libavformat-dev libavcodec-dev

unfortunately the version were too low so I had to install from jessie-backports repo. You'll have to setup backports for your version of linux if you run into same problem in your /etc/apt/sources.list



sudo apt-get -t jessie-backports install libavutil-dev libavformat-dev libavcodec-dev


now finally can run ./configure



cd /media/ssd/Comskip
make


again copy comskip to a path you can call from anywhere



sudo cp comskip /usr/local/bin/comskip


 

Comchap/Comcut



cd /media/ssd
git clone https://github.com/BrettSheleski/comchap.git
cd comchap
make
sudo cp comchap /usr/local/bin/comchap
sudo cp comcut /usr/local/bin/comcut


 

Make sure you have newest version of ffmpeg installed

I've installed mine from jessie-backports but probably whatever you have in your current repo is fine (I wanted NVENC for hevc so i needed a newer version)

 

If you want to use comchap you'll need to adjust some of the code in the original script. Attached comskip.ini I use. Also version 2 of comskip that may be more accurate

Also if you don't have it you need csh package to run dvr script



sudo apt-get install csh

comskip.ini.txt

Edited by ShotToTheDome
  • Like 1

Share this post


Link to post
Share on other sites
mastrmind11

well done.

  • Like 1

Share this post


Link to post
Share on other sites
ShotToTheDome

New post processing script for use if you want to "automatically convert recordings to a streaming friendly format" also check preserve original video and preserve original audio. After the recording completes, the script changes recording mkv ext back to ts, pulls out closed captions and then goes through things as usual. At the end you are left with a the original name in .mkv and the .srt file if it exists. I've found for IPTV using these settings allows for more variance in stream without it stopping or starting a new "xxxx-1.ts" stream for a recording

 

59b4110427bc7_embylivetvsettings.jpg

#!/bin/csh


set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/path/to/embydvr/dvrProcessing.log'
set dvrLockLog = '/path/to/embydvr/dvrLock.log'


#Wait if post processing is already running
while ( -f $lockFile )
    echo "'$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end


#Create lock file to prevent other post-processing from running simultaneously
echo "Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile


#Rename .ts file to .ts.mkv
echo "Renaming '$origFile' to .ts" | tee -a $dvrPostLog
cp -f "$origFile" "$1.ts"


#Mark and cut commercials
echo "cut from '$origFile'" | tee -a $dvrPostLog
/path/to/embydvr/comchap/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/path/to/embydvr/comskip.ini "$1.ts"


#Pull CC from file to SRT file
echo "Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
ccextractor "$1.ts" -o "$tmpSrt"


#Encode file to H.264 with mkv container using ffmpeg
echo "Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/usr/bin/ffmpeg -i "$1.ts" -map 0 -acodec copy -scodec copy -c:v libx264 -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"


#Trim off first minute
echo "Remove first 60 sec of file" | tee -a $dvrPostLog
/usr/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"


#Rename encodes to tmpfile
echo "Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode" "$tmpFile"
mv -f "$tmpEncode2" "$tmpFile"


#Overwrite original ts file with the transcoded file
echo "Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"


#Remove leftover .ts file
echo "Remove leftover .ts file" | tee -a $dvrPostLog
rm -f "$1.ts"


#Remove lock file
echo "Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile


exit 0

 

Edited by ShotToTheDome
  • Like 1

Share this post


Link to post
Share on other sites
tdiguy

I have not tried this yet but it looks quite amazing.

 

  • Like 1

Share this post


Link to post
Share on other sites
ShotToTheDome

I've been using a more aggressive comskip.ini It is using all available detection methods so it takes a lot longer to run, but it hasn't missed any commercials yet in my usage. Also sticking with TS files works better for me with the iptv setup I have along with auto-looping on. I think overall TS is just a more stable recording method after a lot of usage with both

comskip.ini.txt

Edited by ShotToTheDome
  • Like 1

Share this post


Link to post
Share on other sites
ginsengbomb

Fairly new to working with scripts like this, and probably making a terrible simple mistake. I'm attempting to run this on Linux Mint. Have all dependencies (or seem to). The script is instantly failing on launch though. Logs below. Wondering if this is a permission error? I have the script sitting in my /home/me/Videos folder, and emby is set to owner of everything in the folder and subfolders. Log below:

 

2017-11-22 23:38:09.056 Info App: Refreshing recording parent /home/barry/Videos/The Late Show With Stephen Colbert/Season 3
2017-11-22 23:38:09.057 Info App: Deleting temp file /var/lib/emby/transcoding-temp/7acfba132be5429d927e9695296cce6a.ts
2017-11-22 23:38:09.068 Info App: Running recording post processor /home/barry/Videos/post.sh "/home/barry/Videos/The Late Show With Stephen Colbert/Season 3/The Late Show With Stephen Colbert S03E47 John Leguizamo; Elton John.ts"
2017-11-22 23:38:09.069 Error App: Error running recording post processor
    *** Error Report ***
    Version: 3.2.40.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.8.0.53
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Processor count: 8
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.ComponentModel.Win32Exception (0x80004005): Permission denied
     at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd)
     at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
     at System.Diagnostics.Process.Start()
     at Emby.Server.Implementations.Diagnostics.CommonProcess.Start()
     at Emby.Server.Implementations.LiveTv.EmbyTV.EmbyTV.PostProcessRecording(TimerInfo timer, String path)
    System.ComponentModel.Win32Exception
     at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd)
     at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
     at System.Diagnostics.Process.Start()
     at Emby.Server.Implementations.Diagnostics.CommonProcess.Start()
     at Emby.Server.Implementations.LiveTv.EmbyTV.EmbyTV.PostProcessRecording(TimerInfo timer, String path)
    
2017-11-22 23:38:09.069 Info HttpServer: HTTP Response 204 to ::1. Time: 36ms. 

Share this post


Link to post
Share on other sites
Luke

yea that would suggest it does not have permission to execute.  the server is running under a user called emby so you will want to make sure it is able to execute that script.

Share this post


Link to post
Share on other sites
dcrdev

The default acls will prevent other users (besides root) from accessing anything in someone else's home directory, regardless of whether the containing folder/file is owned by someone else.

 

You need to do two things:

  • Move the script somewhere outside of your home directory like /usr/local/bin
  • Make the script executable i.e. chmod +x filename.ext

Share this post


Link to post
Share on other sites
mwongjay

I've been using this script for about a year and it's solid. Just need to ensure log rotation is set up correctly as this causes the syslog and daemon.log files to grow rapidly. In addition to rotating daily I also set them to rotate with a maxsize of 5GB.

 

Below is a partial logrotate config

/var/log/syslog
/var/log/daemon.log
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        maxsize 5G
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

I also modified the original script to add timestamps to the log file and support hardware acceleration using my Nvidia gpu

#!/bin/csh
set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/opt/scripts/commercial-cutting/dvrProcessing.log'
set dvrLockLog = '/opt/scripts/commercial-cutting/dvrLock.log'

#Wait if post processing is already running
while ( -f $lockFile )
    echo "`date '+%Y-%m-%d %H:%M:%S'` '$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end

#Create lock file to prevent other post-processing from running simultaneously
echo "`date '+%Y-%m-%d %H:%M:%S'` Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
echo "`date '+%Y-%m-%d %H:%M:%S'` cut from '$origFile'" | tee -a $dvrPostLog
/usr/local/bin/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/opt/scripts/commercial-cutting/comskip.ini "$origFile"

#Pull CC from file to SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
/usr/local/bin/ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "`date '+%Y-%m-%d %H:%M:%S'` Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/opt/emby-server/bin/ffmpeg -c:v h264_cuvid -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -acodec copy -scodec copy -c:v h264_nvenc -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"

#Trim off first minute
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove first 60 sec of file" | tee -a $dvrPostLog
/opt/emby-server/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"

#Remove tmpEncode file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove tmpEncode file" | tee -a $dvrPostLog
rm -f "$tmpEncode"

#Rename transcoded file to temp file in case no subtitles
echo "`date '+%Y-%m-%d %H:%M:%S'` Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode2" "$tmpFile"

#Overwrite original ts file with the transcoded file
echo "`date '+%Y-%m-%d %H:%M:%S'` Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"

#Rename .ts file to .ts.mkv
echo "`date '+%Y-%m-%d %H:%M:%S'` Renaming '$origFile' to .mkv" | tee -a $dvrPostLog
mv -f "$origFile" "$1.mkv"

#Remove lock file
echo "`date '+%Y-%m-%d %H:%M:%S'` Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile

exit 0

Thanks @@ShotToTheDome for the hard work. 

  • Like 2

Share this post


Link to post
Share on other sites
ShotToTheDome

I'm still using the script too. making the updates for the log files. How come you are not using the cuda h265 conversion or is card not that new? That's what i would want to use if i had access

Share this post


Link to post
Share on other sites
alexpaton

I've just tried setting this up on Ubuntu 18.04, but got stuck at the comskip stage.  When I try to run autogen.sh, it comes up with the following:

Preparing the Comskip build system...please wait

Found GNU Autoconf version 2.69
Found GNU Automake version 1.15.1
Found GNU Libtool version 2.4.6

Automatically preparing build ... Warning: autoreconf failed
Attempting to run the preparation steps individually

Preparing build ... ERROR: aclocal failed

I would very much appreciate it if anyone could offer any help.

Share this post


Link to post
Share on other sites
mark-in-dallas

Just thought that I'd revive this thread to say that I've just set up Comskip on my server, running Linux Mint 20, based on Ubuntu 20.04, and it seems to be working perfect!

The only hitch that I had during set up was when running the command below it indicated that tesseract-ocr-dev was not available, but suggested a replacement to use instead.  I replaced it with the suggestion and the command ran fine..

sudo apt-get install -y gcc libcurl4-gnutls-dev tesseract-ocr tesseract-ocr-dev libleptonica-dev

Replaced with

sudo apt-get install -y gcc libcurl4-gnutls-dev tesseract-ocr libtesseract-dev libleptonica-dev

 

Edited by mark-in-dallas
  • Like 1

Share this post


Link to post
Share on other sites
mikegpds

I just wanted to say thanks for the script.  I put it to use on my Emby server running in Mac OSX.  I used Homebrew to set up ffmpeg and compiled comcut and comchap from source code.

Homebrew ffmpeg:

https://trac.ffmpeg.org/wiki/CompilationGuide/macOS

Comcut/Comchap:

https://www.danielcolomb.com/2017/03/29/setting-up-comchaps-comcut-with-comskip-ffmpeg-and-handbrakecli/

It's working great!  Thanks @@ShotToTheDome for sharing. 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...