Emby post-processing dvr script for linux

September 2, 2017

I thought I'd post my emby post-processing dvr script. Tested and working on Debian Linux 8.9

Files processed end up being named *originalvideoname*.ts.mkv so i can search by that and make sure recordings have no issues. After I look at them, I manually rename to just .mkv. I do not have automatically convert recordings checked or the two preserve boxes checked under live tv --> settings.

Script flow:

see if lockfile exists. if so wait 60 sec
create lockfile
commerical scan and commerical cutting
ccextractor to pull out closed captions and convert to srt file
Encode to h264 using ffmpeg and combine in srt file from previous step
Trim off first 60 sec from recording (due to iptv delay you get a minute from previous show. I have recordings set to last 1 min longer in emby record. The shows using my OTA tuners start early by 1 min to compensate)
Remove srt/extra files
Overwrite original ts recording
delete lockfile

Depends on: comskip, comcut (comchap optional), ccextractor, ffmpeg

I eventually want to fool around with a nvidia gtx 1050 and see how nevnc with hevc/h265 works on the ffmpeg encoding line. Software h265 encoding on very fast saves an additional 30-40% space on same setting as h264 on medium but my encoding framerate goes from 420fps (14x) to 69fps (2.3x) on dual Xeon e5-2670s. The picture looks pretty much identical when i look at them side by side

post.sh emby script

#!/bin/csh

set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/path/to/embydvr/dvrProcessing.log'
set dvrLockLog = '/path/to/embydvr/dvrLock.log'


#Wait if post processing is already running
while ( -f $lockFile )
    echo "'$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end

#Create lock file to prevent other post-processing from running simultaneously
echo "Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
echo "cut from '$origFile'" | tee -a $dvrPostLog
/path/to/embydvr/comchap/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/path/to/embydvr/comskip.ini "$origFile"

#Pull CC from file to SRT file
echo "Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/usr/bin/ffmpeg -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -acodec copy -scodec copy -c:v libx264 -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"

#Trim off first minute
echo "Remove first 60 sec of file" | tee -a $dvrPostLog
/usr/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"

#Remove tmpEncode file
echo "Remove tmpEncode file" | tee -a $dvrPostLog
rm -f "$tmpEncode"

#Rename transcoded file to temp file in case no subtitles
echo "Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode2" "$tmpFile"

#Overwrite original ts file with the transcoded file
echo "Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"

#Rename .ts file to .ts.mkv
echo "Renaming '$origFile' to .mkv" | tee -a $dvrPostLog
mv -f "$origFile" "$1.mkv"

#Remove lock file
echo "Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile

exit 0

Edited September 3, 2017 by ShotToTheDome

September 8, 2017

Installing the programs needed by the script assuming Ubuntu/Debian

CCextractor

git for ccextractor - https://github.com/CCExtractor/ccextractor

clone ccextractor from git source

go to directory you want to use. I'll call mine /media/ssd/



cd /media/ssd

git clone https://github.com/CCExtractor/ccextractor.git

This will pull in the ccextractor code to new directory called ccextractor



cd ccextrator

for ubuntu/debian - download dependencies:



sudo apt-get install -y gcc libcurl4-gnutls-dev tesseract-ocr tesseract-ocr-dev libleptonica-dev

now we need to go into the build directory



cd /media/ssd/ccextractor/linux

./build

 

# test your build

./ccextractor

now we need to place ccextractor where we can call it from a built in path

sudo cp ccextractor /usr/local/bin/ccextractor

Comskip

git for comskip - https://github.com/erikkaashoek/Comskip

clone comskip from git source



cd /media/ssd



git clone https://github.com/erikkaashoek/Comskip.git

go to new Comskip directory

install build requirements



cd Comskip

sudo apt-get install -y autoconf automake libtool

build comskip



./autogen.sh

./configure

here i ran into a missing package argtable2



cd /tmp

wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-0_12-1.1_amd64.deb

wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-docs_12-1.1_all.deb

wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-dev_12-1.1_amd64.deb

sudo dpkg -i libargtable2-0_12-1.1_amd64.deb libargtable2-docs_12-1.1_all.deb libargtable2-dev_12-1.1_amd64.deb

also more missing packages:

sudo apt-get install -y libavutil-dev libavformat-dev libavcodec-dev

unfortunately the version were too low so I had to install from jessie-backports repo. You'll have to setup backports for your version of linux if you run into same problem in your /etc/apt/sources.list



sudo apt-get -t jessie-backports install libavutil-dev libavformat-dev libavcodec-dev

now finally can run ./configure



cd /media/ssd/Comskip

make

again copy comskip to a path you can call from anywhere



sudo cp comskip /usr/local/bin/comskip

Comchap/Comcut



cd /media/ssd

git clone https://github.com/BrettSheleski/comchap.git

cd comchap

make

sudo cp comchap /usr/local/bin/comchap

sudo cp comcut /usr/local/bin/comcut

Make sure you have newest version of ffmpeg installed

I've installed mine from jessie-backports but probably whatever you have in your current repo is fine (I wanted NVENC for hevc so i needed a newer version)

If you want to use comchap you'll need to adjust some of the code in the original script. Attached comskip.ini I use. Also version 2 of comskip that may be more accurate

Also if you don't have it you need csh package to run dvr script



sudo apt-get install csh

comskip.ini.txt

Edited September 8, 2017 by ShotToTheDome

September 9, 2017

well done.

September 9, 2017

New post processing script for use if you want to "automatically convert recordings to a streaming friendly format" also check preserve original video and preserve original audio. After the recording completes, the script changes recording mkv ext back to ts, pulls out closed captions and then goes through things as usual. At the end you are left with a the original name in .mkv and the .srt file if it exists. I've found for IPTV using these settings allows for more variance in stream without it stopping or starting a new "xxxx-1.ts" stream for a recording

#!/bin/csh


set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/path/to/embydvr/dvrProcessing.log'
set dvrLockLog = '/path/to/embydvr/dvrLock.log'


#Wait if post processing is already running
while ( -f $lockFile )
    echo "'$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end


#Create lock file to prevent other post-processing from running simultaneously
echo "Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile


#Rename .ts file to .ts.mkv
echo "Renaming '$origFile' to .ts" | tee -a $dvrPostLog
cp -f "$origFile" "$1.ts"


#Mark and cut commercials
echo "cut from '$origFile'" | tee -a $dvrPostLog
/path/to/embydvr/comchap/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/path/to/embydvr/comskip.ini "$1.ts"


#Pull CC from file to SRT file
echo "Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
ccextractor "$1.ts" -o "$tmpSrt"


#Encode file to H.264 with mkv container using ffmpeg
echo "Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/usr/bin/ffmpeg -i "$1.ts" -map 0 -acodec copy -scodec copy -c:v libx264 -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"


#Trim off first minute
echo "Remove first 60 sec of file" | tee -a $dvrPostLog
/usr/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"


#Rename encodes to tmpfile
echo "Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode" "$tmpFile"
mv -f "$tmpEncode2" "$tmpFile"


#Overwrite original ts file with the transcoded file
echo "Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"


#Remove leftover .ts file
echo "Remove leftover .ts file" | tee -a $dvrPostLog
rm -f "$1.ts"


#Remove lock file
echo "Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile


exit 0

Edited September 10, 2017 by ShotToTheDome

September 12, 2017

I have not tried this yet but it looks quite amazing.

September 12, 2017

I've been using a more aggressive comskip.ini It is using all available detection methods so it takes a lot longer to run, but it hasn't missed any commercials yet in my usage. Also sticking with TS files works better for me with the iptv setup I have along with auto-looping on. I think overall TS is just a more stable recording method after a lot of usage with both

comskip.ini.txt

Edited September 12, 2017 by ShotToTheDome

November 23, 2017

Fairly new to working with scripts like this, and probably making a terrible simple mistake. I'm attempting to run this on Linux Mint. Have all dependencies (or seem to). The script is instantly failing on launch though. Logs below. Wondering if this is a permission error? I have the script sitting in my /home/me/Videos folder, and emby is set to owner of everything in the folder and subfolders. Log below:

2017-11-22 23:38:09.056 Info App: Refreshing recording parent /home/barry/Videos/The Late Show With Stephen Colbert/Season 3
2017-11-22 23:38:09.057 Info App: Deleting temp file /var/lib/emby/transcoding-temp/7acfba132be5429d927e9695296cce6a.ts
2017-11-22 23:38:09.068 Info App: Running recording post processor /home/barry/Videos/post.sh "/home/barry/Videos/The Late Show With Stephen Colbert/Season 3/The Late Show With Stephen Colbert S03E47 John Leguizamo; Elton John.ts"
2017-11-22 23:38:09.069 Error App: Error running recording post processor
    *** Error Report ***
    Version: 3.2.40.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.8.0.53
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Processor count: 8
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.ComponentModel.Win32Exception (0x80004005): Permission denied
     at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd)
     at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
     at System.Diagnostics.Process.Start()
     at Emby.Server.Implementations.Diagnostics.CommonProcess.Start()
     at Emby.Server.Implementations.LiveTv.EmbyTV.EmbyTV.PostProcessRecording(TimerInfo timer, String path)
    System.ComponentModel.Win32Exception
     at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd)
     at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
     at System.Diagnostics.Process.Start()
     at Emby.Server.Implementations.Diagnostics.CommonProcess.Start()
     at Emby.Server.Implementations.LiveTv.EmbyTV.EmbyTV.PostProcessRecording(TimerInfo timer, String path)

2017-11-22 23:38:09.069 Info HttpServer: HTTP Response 204 to ::1. Time: 36ms.

November 23, 2017

yea that would suggest it does not have permission to execute. the server is running under a user called emby so you will want to make sure it is able to execute that script.

November 23, 2017

The default acls will prevent other users (besides root) from accessing anything in someone else's home directory, regardless of whether the containing folder/file is owned by someone else.

You need to do two things:

Move the script somewhere outside of your home directory like /usr/local/bin
Make the script executable i.e. chmod +x filename.ext

January 15, 2019

I've been using this script for about a year and it's solid. Just need to ensure log rotation is set up correctly as this causes the syslog and daemon.log files to grow rapidly. In addition to rotating daily I also set them to rotate with a maxsize of 5GB.

Below is a partial logrotate config

/var/log/syslog
/var/log/daemon.log
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        maxsize 5G
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

I also modified the original script to add timestamps to the log file and support hardware acceleration using my Nvidia gpu

#!/bin/csh
set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/opt/scripts/commercial-cutting/dvrProcessing.log'
set dvrLockLog = '/opt/scripts/commercial-cutting/dvrLock.log'

#Wait if post processing is already running
while ( -f $lockFile )
    echo "`date '+%Y-%m-%d %H:%M:%S'` '$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end

#Create lock file to prevent other post-processing from running simultaneously
echo "`date '+%Y-%m-%d %H:%M:%S'` Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
echo "`date '+%Y-%m-%d %H:%M:%S'` cut from '$origFile'" | tee -a $dvrPostLog
/usr/local/bin/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/opt/scripts/commercial-cutting/comskip.ini "$origFile"

#Pull CC from file to SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
/usr/local/bin/ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "`date '+%Y-%m-%d %H:%M:%S'` Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/opt/emby-server/bin/ffmpeg -c:v h264_cuvid -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -acodec copy -scodec copy -c:v h264_nvenc -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"

#Trim off first minute
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove first 60 sec of file" | tee -a $dvrPostLog
/opt/emby-server/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"

#Remove tmpEncode file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove tmpEncode file" | tee -a $dvrPostLog
rm -f "$tmpEncode"

#Rename transcoded file to temp file in case no subtitles
echo "`date '+%Y-%m-%d %H:%M:%S'` Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode2" "$tmpFile"

#Overwrite original ts file with the transcoded file
echo "`date '+%Y-%m-%d %H:%M:%S'` Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"

#Rename .ts file to .ts.mkv
echo "`date '+%Y-%m-%d %H:%M:%S'` Renaming '$origFile' to .mkv" | tee -a $dvrPostLog
mv -f "$origFile" "$1.mkv"

#Remove lock file
echo "`date '+%Y-%m-%d %H:%M:%S'` Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile

exit 0

Thanks @@ShotToTheDome for the hard work.

January 20, 2019

I'm still using the script too. making the updates for the log files. How come you are not using the cuda h265 conversion or is card not that new? That's what i would want to use if i had access

March 13, 2019

I've just tried setting this up on Ubuntu 18.04, but got stuck at the comskip stage. When I try to run autogen.sh, it comes up with the following:

Preparing the Comskip build system...please wait

Found GNU Autoconf version 2.69
Found GNU Automake version 1.15.1
Found GNU Libtool version 2.4.6

Automatically preparing build ... Warning: autoreconf failed
Attempting to run the preparation steps individually

Preparing build ... ERROR: aclocal failed

I would very much appreciate it if anyone could offer any help.

September 15, 2020

Just thought that I'd revive this thread to say that I've just set up Comskip on my server, running Linux Mint 20, based on Ubuntu 20.04, and it seems to be working perfect!

The only hitch that I had during set up was when running the command below it indicated that tesseract-ocr-dev was not available, but suggested a replacement to use instead. I replaced it with the suggestion and the command ran fine..

sudo apt-get install -y gcc libcurl4-gnutls-dev tesseract-ocr tesseract-ocr-dev libleptonica-dev

Replaced with

sudo apt-get install -y gcc libcurl4-gnutls-dev tesseract-ocr libtesseract-dev libleptonica-dev

Edited September 17, 2020 by mark-in-dallas

October 13, 2020

I just wanted to say thanks for the script. I put it to use on my Emby server running in Mac OSX. I used Homebrew to set up ffmpeg and compiled comcut and comchap from source code.

Homebrew ffmpeg:

https://trac.ffmpeg.org/wiki/CompilationGuide/macOS

Comcut/Comchap:

https://www.danielcolomb.com/2017/03/29/setting-up-comchaps-comcut-with-comskip-ffmpeg-and-handbrakecli/

It's working great! Thanks @@ShotToTheDome for sharing.

October 27, 2020

Not sure why, but I didn't think anyone had comskip working on Linux, so was giving my thumb a workout. Another thread brought this to my attention, so I gave it a shot. Centos made a few package dependencies a challenge, but once past those, it works like a charm.

I did tweak the script to use BASH. With that I set it to trim the .ts from the target file, and added basic error checking. I had one failure during testing, and rerunning the script with no changes allowed it to convert. So I check to see if ffmpeg errors out, or if the new mkv exists, but is zero bytes. If either is true, the conversion is retried, up to three times, and logged it if ultimately fails.

I left the code to trim the 1st minute, but commented it out, since comskip seems to handle my recordings that I had set to start a minute early. Also commented out is a test to delete the original if the conversion worked. For now I want to keep both copies.

This script assumes you have a separate recording library and viewing library. In my case no one has access to the recording library and depends on this script to move the recordings to the viewing library. I am working on a script that works with one library that can hopefully avoid emby trying to update media info before the task completes.

#!/bin/bash
#Set these values for your setup
base=/home/common/Media/LiveTV/         # Where is the TV library
srcbase=/home/common/Media/tmp/         # Where recordings are stored
comProc=comchap                         # comcut or comchap for cut vs chapter marks
comskip=/usr/local/bin/comskip          # In case you put it somewhere emby won't look
ffmpegProc=/usr/bin/ffmpeg              # Use system ffmpeg if run manually
svcUser=emby                            # Who is the emby service running as
tgtRatio=13                             # H265 w/comcut 18~20% of original, H264 20~22%
                                        # Pick a number just under the target

origFile="$1"
tmpFile="$origFile.tmp"
tgtFile="${origFile%.*}.mkv"
tmpEncode="$origFile.mkv"
tmpEncode2="$origFile.2.mkv"
tmpSrt="$origFile.srt"
dvrPostLog='/var/log/dvrProcessing.log'
dvrLockLog='/var/log/dvrLock.log'
lockFile='/tmp/dvrProcessing.lock'

failed=1
((tryCount=3))

#Get Source file size
origFilesize=`stat -c %s "$origFile"`

#Check to see if the emby server called this and use the correct ffmpeg
if [ "$USER" = "$svcUser" ];
then
        echo "Running inside of emby as $USER" | tee -a $dvrPostLog
        ffmpegProc=/opt/emby-server/bin/ffmpeg

        #Extract series name
        stmp=`echo $1 |sed "s|^$srcbase\(.*\)\/.*|\1|"`
        series=`echo $stmp |sed 's/^\(.*\)\/.*$/\1/'`/
        #Extract Season
        season=`echo $stmp |sed 's/^.*\/\(.*\)$/\1/'`/
        #cleanup temp structure
        srcDir=`echo $1 |sed 's/\(.*\/\).*\/.*\/.*$/\1/'`
        #Pull just the file name ending in MKV for the final move
        tgtFile=test`echo $1 |sed 's/\(.*\)\/\(.*\)$/\2/'`

        if [ "$series" = "$season" ];
        then
                season=""
        fi

        tgtFinal=`echo $tgtFile |sed 's/\(.*\)\/\(.*\)$/\2/'`

fi

#Wait if post processing is already running
while [ -f $lockFile ] ;
do
    echo "`date '+%Y-%m-%d %H:%M:%S'` '$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
done

#Create lock file to prevent other post-processing from running simultaneously
echo "`date '+%Y-%m-%d %H:%M:%S'` Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
if [ "$comProc" = "comcut" ];
then
        echo "`date '+%Y-%m-%d %H:%M:%S'` cut from '$origFile'" | tee -a $dvrPostLog
        /usr/local/bin/$comProc --lockfile=/tmp/comchap.lock --comskip-ini=/etc/comskip.ini --comskip=$comskip --comskip=$comskip "$origFile"
fi

#Pull CC from file to SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
/usr/local/bin/ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "`date '+%Y-%m-%d %H:%M:%S'` Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
while [ $tryCount -ne 0 ]
do
         $ffmpegProc -nostdin -hide_banner -hwaccel_output_format cuda -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -ignore_unknown -acodec copy -scodec copy -c:v hevc_nvenc -preset medium -deinterlace "$tgtFile" | tee -a $dvrPostLog


        if [[ $? -ne 0 || ! -s "$tgtFile" ]];
        then
                rm -f "$tgtFile"
                echo "Trying $tryCount more time(s)..." | tee -a $dvrPostLog
                ((tryCount=tryCount-1))
                sleep 15
        else
                ((tryCount=0))
                failed=0
        fi
done

if [ $failed -eq 1 ];
then
        echo "`date '+%Y-%m-%d %H:%M:%S'` Re-encoding '$origFile' to MKV file failed!" | tee -a $dvrPostLog
fi

#Trim off first minute
#echo "`date '+%Y-%m-%d %H:%M:%S'` Remove first 60 sec of file" | tee -a $dvrPostLog
#/opt/emby-server/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"


#Calculate MKV size
if [ -f "$tgtFile" ];
then
        newFilesize=`stat -c %s "$tgtFile"`
        MKVratio=$(( 100*"$newFilesize"/"$origFilesize" ))
fi

#Delete .ts file h265 MKV is typically 18~20% original
if [[ $failed -eq 0 && $MKVratio -ge $tgtRatio ]];
then
        #Mark commercials as chapters, must be done after conversion to MKV
        if [ "$comProc" = "comchap" ];
        then
                echo "`date '+%Y-%m-%d %H:%M:%S'` cut from '$origFile'" | tee -a $dvrPostLog
                mv "$tgtFile" "$tgtFile"-tmp.mkv
                /usr/local/bin/$comProc --verbose --lockfile=/tmp/comchap.lock --comskip-ini=/etc/comskip.ini "$tgtFile"-tmp.mkv  > /tmp/comchap.log
                #Comchap seems to have a knack for minor corruption that prtevents playing. Quick repair
                $ffmpegProc -i "$tgtFile"-tmp.mkv -c copy "$tgtFile"
                rm -f "$tgtFile"-tmp.mkv
        fi

        mv   "$origFile" "$origFile"-orig  #backup copy

        if [ "$USER" = "$svcUser" ];            #Don't move Season/series if running manually
        then
                if [ ! -d "$base$series" ];
                then
                        echo "`date '+%Y-%m-%d %H:%M:%S'` Creating series and season" | tee -a $dvrPostLog
                        mv "$srcDir$series" $base
                fi

                if [ ! -d "$base$series$season" ];
                then
                        echo "`date '+%Y-%m-%d %H:%M:%S'` Creating season" | tee -a $dvrPostLog
                        mv "$srcDir$series$season" "$base$series"
                else
                        if [ -f "$tgtFile" ];
                        then
                                echo "`date '+%Y-%m-%d %H:%M:%S'` Moving episode files" | tee -a $dvrPostLog
                                mv "$srcDir$series$season*" "$base$series$season"
                        fi
                fi
        fi

        #mv "$tgtFile" "$base$series/$season/$tgtFinal"

        #This WILL eat your system if the recording path is wrong, better to clean manually
        if [[ "$srcDir$series" != ""  && "$srcDir$series" != "/"  && "$srcDir$series" != "/home"  && "$srcDir$series" != "/usr"  && "$srcDir$series" != "/var"  && "$srcDir$series" != "/bin" ]];
        then
                echo "`date '+%Y-%m-%d %H:%M:%S'` Remove item directories '$srcDir$series'" | tee -a $dvrPostLog
                #rm -rf "$srcDir$series"
        fi

else
        mv   "$origFile" "$base$series$season$origFile" #Conversion failed
        rm -f "$tgtFile"
fi

#Remove lock file
echo "`date '+%Y-%m-%d %H:%M:%S'` Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm -f $lockFile

exit 0

Edit: made script dual purpose, can be run manually or called from emby, uses the appropriate ffmpeg in either case. Also added a vairable to switch from cutting commercials to just marking chapters

Edit 2: Add target file size check to avoid grossly invalid conversions. Also move chapter marking until after MKV conversion, cuts remain before conversion to speed up conversion

Edit 3: The script now assumes the recording path and Library are different and moves the final conversions to the Library. It can clean up the temporary work area, but I consider that high risk and left it commented out.

Edit 4: Found a think-o blocking comchap from even running. Improved file relocation using mv for the whole series/season if either is new. Less code and faster if the source and destination are on the same filesystem

Edit 5: Dealt with recordings without a season and attempt at better detection of the base path

Edited October 31, 2020 by Dan_Austin
Script updates

October 27, 2020

Tip: -crf does not apply to _nvenc use -qp <value>

20 to 30 are sensible values

25 is a good compromise

20 - higher bitrate > bigger files

30 - lower bitrate > smaller files

depending on you other ffmpeg settings but a h264 to h265 would give you a file of approx a third of the size and generally be very difficult to tell from the original - but garbage in garbage out applies

October 27, 2020

Well shucks. FFMPEG didn't complain. Another round of testing :

Original	Comskip Only	No QP	H264	H265
3GB	1.8GB	326MB	532MB	493MB

The H264 conversion took 1:03 and the H265 took 1:42

I haven't watched the last two tests, but the 'No QP' test appeared artifact free and looked good to me. I'll update the last post now...

October 27, 2020

I am using built in emby options to convert my files to h256. I have freed up over almost 500gb of space on my 3tb library. This makes me want to install comskip just to free up even more space. Cutting a files size almost in half by eliminating commercials, now thats something.

October 28, 2020

7 hours ago, Dan_Austin said:

Well shucks. FFMPEG didn't complain. Another round of testing :

Original Comskip Only No QP H264 H265
3GB 1.8GB 326MB 532MB 493MB
The H264 conversion took 1:03 and the H265 took 1:42

I haven't watched the last two tests, but the 'No QP' test appeared artifact free and looked good to me. I'll update the last post now...

ffmpeg - if you save the console output has a couple of lines in it about using qp with nvenc - which is how i found out about it

-crf is cpu specific for when using the cpu - IIRC then nvenc will be using its defaults

another thing you can add is the preset values - slow, medium or fast - only for nvenc

this is my ffmpeg argument list - its for powershell but you should be able to convert easily

$ArgumentList = '-hide_banner -loglevel +level -hwaccel_device 0 -hwaccel d3d11va -i "{0}" -vf yadif -map 0:v:0 -c:v:0 hevc_nvenc -preset {2} -qp 25 -level 5.1 -tier high -map 0:a -c:a copy -map 0:s? -c:s copy -map_chapters 0 -map_metadata 0 "{1}"' -f $oldVideo, $newVideo, $GPUPreset

"-hwaccel_device 0 -hwaccel d3d11va" is for hardware decoding on windows - there should be equivalents for vaapi etc

device 0 is the first gpu in your system should you have more than one change the number to 1,2,3 etc to target that gpu

yadif is for interlaced content - only works if interlaced content is found so harmless if say 1080p

map 0:s? - copy subtitles if they exist - but be careful as MP4 does not support some subtitle formats and ffmpeg will have a strop about it - so i always convert to mkv to avoid this

map_x maintain chapter markers and metadata set to -1 if you want them removed

etc

i have used the above on over 15k of movies/episodes - saving well over 10TB of space - and 99.99999999999% it works fine - and remaining ones were crap recordings and usually a trip through mkvtoolnix would sort the issue or get another copy etc

October 28, 2020

8 hours ago, Dan_Austin said:

Well shucks. FFMPEG didn't complain. Another round of testing :

Original Comskip Only No QP H264 H265
3GB 1.8GB 326MB 532MB 493MB
The H264 conversion took 1:03 and the H265 took 1:42

I haven't watched the last two tests, but the 'No QP' test appeared artifact free and looked good to me. I'll update the last post now...

one thing i forgot to say

if you go to say qp 20 - and the orig file is already at a lowish bitrate say below 3Mb/s qp 20 is very likely to make the file significantly bigger as it will increase the bitrate of the output

so experiment with the values you use - 25 works for me most of the time

also not figured this out yet but 50fps files - BBC HD for some programs - tend to be bigger in output - not sure why might be a setting i need to change or its a nvenc issue - but for me its only for things like live sport which i delete after watching so not got around to investigating it yet

October 28, 2020

At the end of the day - how much you care about TV recordings - is down to you - i rarely keep them more than a few days - so a simple remux to h265 - will usually fix any errors in the stream - 90% of live streams have errors which ffmpeg can clean up as part of the remux

as for comskip its OK but its not that accurate with UK tv at least and is usually within 10s of the transition from and to commercials - so i just set chapter markers for the change points it finds and skip the commercials using the chapters and rwd/ffw as necessary if i think i have missed something - so i watch 99% of tv from the processed recording not the raw ts file or live etc

your mileage may vary but i dont see any benefit in chopping up tv recordings to remove the commercials - but in the UK its only every 15 minutes for 2-3 minutes so not a huge saving in space - so not worth the effort and the chapter markers are not accurate enough from comskip.

October 28, 2020

U.S. TV used to have a ratio closer to that, but has been introducing longer and more frequent commercials.

I only have a couple actual programs that were processed by the script with Emby launching it. I have about a dozen that I processed manually while I tuned and tweaked the script. For the 'benchmark' part I would reprocess a copy of the same file over and over, otherwise of the dozen or so programs, all were 32 minutes long with a mix of live action and animation. All were programs I had watched. I do not notice any missing content, but do find the commercial cuts slightly jarring, but only slightly. The resulting durations were between 19:00 and 22:00, a significant reduction.

I do tend to keep series I like, either to watch again later or share when I have company if their interests mirror my own, especially if it is a still running series. Going back to rewatch an old episode that contributes to a current story line beats the 3 minute 'previously on...'

Thanks for sharing your option list, it does already show a couple improvements I can make and suggest areas I should look into to understand.

October 30, 2020

I decided that it was a better idea to move the original .ts file to trash instead of deleting the file with rm. I lost a couple shows because I wasn't error checking to be sure the MKV file was actually created before deleting the original file. Moving to trash gives me another opportunity to process the file. I can live with cleaning out my trash every so often. I changed the rm commands in the script to mv.

#rm -f "$origFile"
mv "$origFile" ~/.Trash

Just a suggestion. Enjoy!

October 30, 2020

Updated the script further. There is a bit of code redundancy. Chapters cannot be saved in TS files, so chapter marking must be after the conversion to MKV, but for commercial cutting, it still makes sense to do it before the conversion.

Also added a test to see if the resulting file is a reasonable size. My one 'complete' conversion failure was where the video was not copied/transcode and the target only had audio, with a result the file size was about 1% of the original. I choose 15%, but made it a config option, since ymmv.

October 30, 2020

you can add the chapters as part of the conversion no need to do two things separately

Sign In

Emby post-processing dvr script for linux

Recommended Posts

ShotToTheDome 7

ShotToTheDome 7

mastrmind11 722

ShotToTheDome 7

tdiguy 97

ShotToTheDome 7

ginsengbomb 5

Luke 39651

dcrdev 253

mwongjay 72

ShotToTheDome 7

alexpaton 10

mark-in-dallas 87

mikegpds 1

Dan_Austin 62

PenkethBoy 2066

Dan_Austin 62

tdiguy 97

PenkethBoy 2066

PenkethBoy 2066

PenkethBoy 2066

Dan_Austin 62

mikegpds 1

Dan_Austin 62

PenkethBoy 2066

Create an account or sign in to comment

Create an account

Sign in

Activity