Jump to content


Photo

Emby post-processing dvr script for linux

dvr post-processing ffmpeg ccextractor comskip comcut srt linux

  • Please log in to reply
11 replies to this topic

#1 ShotToTheDome OFFLINE  

ShotToTheDome

    Advanced Member

  • Members
  • 39 posts
  • Local time: 05:18 AM

Posted 02 September 2017 - 12:56 PM

I thought I'd post my emby post-processing dvr script. Tested and working on Debian Linux 8.9

Files processed end up being named *originalvideoname*.ts.mkv so i can search by that and make sure recordings have no issues. After I look at them, I manually rename to just .mkv. I do not have automatically convert recordings checked or the two preserve boxes checked under live tv --> settings.

Script flow:

  • see if lockfile exists. if so wait 60 sec
  • create lockfile
  • commerical scan and commerical cutting
  • ccextractor to pull out closed captions and convert to srt file
  • Encode to h264 using ffmpeg and combine in srt file from previous step
  • Trim off first 60 sec from recording (due to iptv delay you get a minute from previous show. I have recordings set to last 1 min longer in emby record. The shows using my OTA tuners start early by 1 min to compensate)
  • Remove srt/extra files
  • Overwrite original ts recording
  • delete lockfile

Depends on: comskip, comcut (comchap optional), ccextractor, ffmpeg

I eventually want to fool around with a nvidia gtx 1050 and see how nevnc with hevc/h265 works on the ffmpeg encoding line. Software h265 encoding on very fast saves an additional 30-40% space on same setting as h264 on medium but my encoding framerate goes from 420fps (14x) to 69fps (2.3x) on dual Xeon e5-2670s. The picture looks pretty much identical when i look at them side by side

post.sh emby script

#!/bin/csh

set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/path/to/embydvr/dvrProcessing.log'
set dvrLockLog = '/path/to/embydvr/dvrLock.log'


#Wait if post processing is already running
while ( -f $lockFile )
    echo "'$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end

#Create lock file to prevent other post-processing from running simultaneously
echo "Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
echo "cut from '$origFile'" | tee -a $dvrPostLog
/path/to/embydvr/comchap/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/path/to/embydvr/comskip.ini "$origFile"

#Pull CC from file to SRT file
echo "Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/usr/bin/ffmpeg -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -acodec copy -scodec copy -c:v libx264 -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"

#Trim off first minute
echo "Remove first 60 sec of file" | tee -a $dvrPostLog
/usr/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"

#Remove tmpEncode file
echo "Remove tmpEncode file" | tee -a $dvrPostLog
rm -f "$tmpEncode"

#Rename transcoded file to temp file in case no subtitles
echo "Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode2" "$tmpFile"

#Overwrite original ts file with the transcoded file
echo "Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"

#Rename .ts file to .ts.mkv
echo "Renaming '$origFile' to .mkv" | tee -a $dvrPostLog
mv -f "$origFile" "$1.mkv"

#Remove lock file
echo "Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile

exit 0

Edited by ShotToTheDome, 03 September 2017 - 04:35 PM.

  • Jdiesel likes this

#2 ShotToTheDome OFFLINE  

ShotToTheDome

    Advanced Member

  • Members
  • 39 posts
  • Local time: 05:18 AM

Posted 08 September 2017 - 04:13 PM

Installing the programs needed by the script assuming Ubuntu/Debian
 
CCextractor
clone ccextractor from git source
go to directory you want to use. I'll call mine /media/ssd/
cd /media/ssd
git clone https://github.com/CCExtractor/ccextractor.git
This will pull in the ccextractor code to new directory called ccextractor
cd ccextrator
for ubuntu/debian - download dependencies: 
sudo apt-get install -y gcc libcurl4-gnutls-dev tesseract-ocr tesseract-ocr-dev libleptonica-dev
now we need to go into the build directory
cd /media/ssd/ccextractor/linux
./build
 
# test your build
./ccextractor
now we need to place ccextractor where we can call it from a built in path
sudo cp ccextractor /usr/local/bin/ccextractor
 
Comskip
clone comskip from git source
cd /media/ssd
git clone https://github.com/erikkaashoek/Comskip.git
go to new Comskip directory
install build requirements
cd Comskip
sudo apt-get install -y autoconf automake libtool
build comskip
./autogen.sh
./configure
here i ran into a missing package argtable2
cd /tmp
wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-0_12-1.1_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-docs_12-1.1_all.deb
wget http://ftp.us.debian.org/debian/pool/main/a/argtable2/libargtable2-dev_12-1.1_amd64.deb
sudo dpkg -i libargtable2-0_12-1.1_amd64.deb libargtable2-docs_12-1.1_all.deb libargtable2-dev_12-1.1_amd64.deb
also more missing packages:
sudo apt-get install -y libavutil-dev libavformat-dev libavcodec-dev
unfortunately the version were too low so I had to install from jessie-backports repo. You'll have to setup backports for your version of linux if you run into same problem in your /etc/apt/sources.list
sudo apt-get -t jessie-backports install libavutil-dev libavformat-dev libavcodec-dev
now finally can run ./configure
cd /media/ssd/Comskip
make
again copy comskip to a path you can call from anywhere
sudo cp comskip /usr/local/bin/comskip
 
Comchap/Comcut
cd /media/ssd
git clone https://github.com/BrettSheleski/comchap.git
cd comchap
make
sudo cp comchap /usr/local/bin/comchap
sudo cp comcut /usr/local/bin/comcut
 
Make sure you have newest version of ffmpeg installed
I've installed mine from jessie-backports but probably whatever you have in your current repo is fine (I wanted NVENC for hevc so i needed a newer version)
 
If you want to use comchap you'll need to adjust some of the code in the original script. Attached comskip.ini I use. Also version 2 of comskip that may be more accurate
Also if you don't have it you need csh package to run dvr script
sudo apt-get install csh

Attached Files


Edited by ShotToTheDome, 08 September 2017 - 04:25 PM.


#3 mastrmind11 OFFLINE  

mastrmind11

    Advanced Member

  • Members
  • 2613 posts
  • Local time: 06:18 AM
  • LocationLong Island, NY

Posted 09 September 2017 - 10:08 AM

well done.


  • ShotToTheDome likes this

#4 ShotToTheDome OFFLINE  

ShotToTheDome

    Advanced Member

  • Members
  • 39 posts
  • Local time: 05:18 AM

Posted 09 September 2017 - 12:04 PM

New post processing script for use if you want to "automatically convert recordings to a streaming friendly format" also check preserve original video and preserve original audio. After the recording completes, the script changes recording mkv ext back to ts, pulls out closed captions and then goes through things as usual. At the end you are left with a the original name in .mkv and the .srt file if it exists. I've found for IPTV using these settings allows for more variance in stream without it stopping or starting a new "xxxx-1.ts" stream for a recording

 

59b4110427bc7_embylivetvsettings.jpg

#!/bin/csh


set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/path/to/embydvr/dvrProcessing.log'
set dvrLockLog = '/path/to/embydvr/dvrLock.log'


#Wait if post processing is already running
while ( -f $lockFile )
    echo "'$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end


#Create lock file to prevent other post-processing from running simultaneously
echo "Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile


#Rename .ts file to .ts.mkv
echo "Renaming '$origFile' to .ts" | tee -a $dvrPostLog
cp -f "$origFile" "$1.ts"


#Mark and cut commercials
echo "cut from '$origFile'" | tee -a $dvrPostLog
/path/to/embydvr/comchap/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/path/to/embydvr/comskip.ini "$1.ts"


#Pull CC from file to SRT file
echo "Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
ccextractor "$1.ts" -o "$tmpSrt"


#Encode file to H.264 with mkv container using ffmpeg
echo "Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/usr/bin/ffmpeg -i "$1.ts" -map 0 -acodec copy -scodec copy -c:v libx264 -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"


#Trim off first minute
echo "Remove first 60 sec of file" | tee -a $dvrPostLog
/usr/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"


#Rename encodes to tmpfile
echo "Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode" "$tmpFile"
mv -f "$tmpEncode2" "$tmpFile"


#Overwrite original ts file with the transcoded file
echo "Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"


#Remove leftover .ts file
echo "Remove leftover .ts file" | tee -a $dvrPostLog
rm -f "$1.ts"


#Remove lock file
echo "Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile


exit 0

 


Edited by ShotToTheDome, 09 September 2017 - 08:21 PM.


#5 tdiguy OFFLINE  

tdiguy

    Advanced Member

  • Members
  • 1119 posts
  • Local time: 06:18 AM

Posted 12 September 2017 - 06:41 AM

I have not tried this yet but it looks quite amazing.

 


  • ShotToTheDome likes this

#6 ShotToTheDome OFFLINE  

ShotToTheDome

    Advanced Member

  • Members
  • 39 posts
  • Local time: 05:18 AM

Posted 12 September 2017 - 02:07 PM

I've been using a more aggressive comskip.ini It is using all available detection methods so it takes a lot longer to run, but it hasn't missed any commercials yet in my usage. Also sticking with TS files works better for me with the iptv setup I have along with auto-looping on. I think overall TS is just a more stable recording method after a lot of usage with both

Attached Files


Edited by ShotToTheDome, 12 September 2017 - 04:15 PM.


#7 ginsengbomb OFFLINE  

ginsengbomb

    Member

  • Members
  • 25 posts
  • Local time: 06:18 AM

Posted 23 November 2017 - 12:41 AM

Fairly new to working with scripts like this, and probably making a terrible simple mistake. I'm attempting to run this on Linux Mint. Have all dependencies (or seem to). The script is instantly failing on launch though. Logs below. Wondering if this is a permission error? I have the script sitting in my /home/me/Videos folder, and emby is set to owner of everything in the folder and subfolders. Log below:

 

2017-11-22 23:38:09.056 Info App: Refreshing recording parent /home/barry/Videos/The Late Show With Stephen Colbert/Season 3
2017-11-22 23:38:09.057 Info App: Deleting temp file /var/lib/emby/transcoding-temp/7acfba132be5429d927e9695296cce6a.ts
2017-11-22 23:38:09.068 Info App: Running recording post processor /home/barry/Videos/post.sh "/home/barry/Videos/The Late Show With Stephen Colbert/Season 3/The Late Show With Stephen Colbert S03E47 John Leguizamo; Elton John.ts"
2017-11-22 23:38:09.069 Error App: Error running recording post processor
    *** Error Report ***
    Version: 3.2.40.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Unix 4.8.0.53
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Processor count: 8
    Program data path: /var/lib/emby
    Application directory: /opt/emby-server/system
    System.ComponentModel.Win32Exception (0x80004005): Permission denied
     at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd)
     at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
     at System.Diagnostics.Process.Start()
     at Emby.Server.Implementations.Diagnostics.CommonProcess.Start()
     at Emby.Server.Implementations.LiveTv.EmbyTV.EmbyTV.PostProcessRecording(TimerInfo timer, String path)
    System.ComponentModel.Win32Exception
     at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd)
     at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
     at System.Diagnostics.Process.Start()
     at Emby.Server.Implementations.Diagnostics.CommonProcess.Start()
     at Emby.Server.Implementations.LiveTv.EmbyTV.EmbyTV.PostProcessRecording(TimerInfo timer, String path)
    
2017-11-22 23:38:09.069 Info HttpServer: HTTP Response 204 to ::1. Time: 36ms. 



#8 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 124473 posts
  • Local time: 06:18 AM

Posted 23 November 2017 - 12:59 AM

yea that would suggest it does not have permission to execute.  the server is running under a user called emby so you will want to make sure it is able to execute that script.



#9 dcrdev OFFLINE  

dcrdev

    Advanced Member

  • Members
  • 846 posts
  • Local time: 11:18 AM
  • LocationUK

Posted 23 November 2017 - 05:05 AM

The default acls will prevent other users (besides root) from accessing anything in someone else's home directory, regardless of whether the containing folder/file is owned by someone else.

 

You need to do two things:

  • Move the script somewhere outside of your home directory like /usr/local/bin
  • Make the script executable i.e. chmod +x filename.ext


#10 mwongjay OFFLINE  

mwongjay

    Advanced Member

  • Members
  • 247 posts
  • Local time: 06:18 AM

Posted 15 January 2019 - 04:19 PM

I've been using this script for about a year and it's solid. Just need to ensure log rotation is set up correctly as this causes the syslog and daemon.log files to grow rapidly. In addition to rotating daily I also set them to rotate with a maxsize of 5GB.

 

Below is a partial logrotate config

/var/log/syslog
/var/log/daemon.log
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        maxsize 5G
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

I also modified the original script to add timestamps to the log file and support hardware acceleration using my Nvidia gpu

#!/bin/csh
set path = ($path /usr/local/bin)
set lockFile = '/tmp/dvrProcessing.lock'
set origFile = "$1"
set tmpFile = "$1.tmp"
set tmpEncode = "$1.mkv"
set tmpEncode2 = "$1.2.mkv"
set tmpSrt = "$1.srt"
set dvrPostLog = '/opt/scripts/commercial-cutting/dvrProcessing.log'
set dvrLockLog = '/opt/scripts/commercial-cutting/dvrLock.log'

#Wait if post processing is already running
while ( -f $lockFile )
    echo "`date '+%Y-%m-%d %H:%M:%S'` '$lockFile' exists, sleeping processing of '$origFile'" | tee $dvrLockLog
    sleep 60
end

#Create lock file to prevent other post-processing from running simultaneously
echo "`date '+%Y-%m-%d %H:%M:%S'` Creating lock file for processing '$origFile'" | tee -a $dvrPostLog
touch $lockFile

#Mark and cut commercials
echo "`date '+%Y-%m-%d %H:%M:%S'` cut from '$origFile'" | tee -a $dvrPostLog
/usr/local/bin/comcut --lockfile=/tmp/comchap.lock --comskip-ini=/opt/scripts/commercial-cutting/comskip.ini "$origFile"

#Pull CC from file to SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Pulling Closed captions from '$origFile' to SRT file" | tee -a $dvrPostLog
/usr/local/bin/ccextractor "$origFile" -o "$tmpSrt"

#Encode file to H.264 with mkv container using ffmpeg and mux in CC srt
echo "`date '+%Y-%m-%d %H:%M:%S'` Re-encoding '$origFile' to MKV file while adding cc data" | tee -a $dvrPostLog
/opt/emby-server/bin/ffmpeg -c:v h264_cuvid -i "$origFile" -i "$tmpSrt" -map 0 -map 1 -acodec copy -scodec copy -c:v h264_nvenc -preset medium -crf 23 -profile:v high -level 4.1 -deinterlace "$tmpEncode"

#Trim off first minute
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove first 60 sec of file" | tee -a $dvrPostLog
/opt/emby-server/bin/ffmpeg -ss 00:01:00 -i "$tmpEncode" -vcodec copy -acodec copy -scodec copy "$tmpEncode2"

#Remove SRT file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove SRT file" | tee -a $dvrPostLog
rm -f "$tmpSrt"

#Remove tmpEncode file
echo "`date '+%Y-%m-%d %H:%M:%S'` Remove tmpEncode file" | tee -a $dvrPostLog
rm -f "$tmpEncode"

#Rename transcoded file to temp file in case no subtitles
echo "`date '+%Y-%m-%d %H:%M:%S'` Rename 2nd transcoded file to tmp file" | tee -a $dvrPostLog
mv -f "$tmpEncode2" "$tmpFile"

#Overwrite original ts file with the transcoded file
echo "`date '+%Y-%m-%d %H:%M:%S'` Removing '$origFile'" | tee -a $dvrPostLog
mv -f "$tmpFile" "$origFile"

#Rename .ts file to .ts.mkv
echo "`date '+%Y-%m-%d %H:%M:%S'` Renaming '$origFile' to .mkv" | tee -a $dvrPostLog
mv -f "$origFile" "$1.mkv"

#Remove lock file
echo "`date '+%Y-%m-%d %H:%M:%S'` Done processing '$origFile' removing lock" | tee -a $dvrPostLog
rm $lockFile

exit 0

Thanks @ShotToTheDome for the hard work. 


  • ShotToTheDome likes this

#11 ShotToTheDome OFFLINE  

ShotToTheDome

    Advanced Member

  • Members
  • 39 posts
  • Local time: 05:18 AM

Posted 20 January 2019 - 05:22 PM

I'm still using the script too. making the updates for the log files. How come you are not using the cuda h265 conversion or is card not that new? That's what i would want to use if i had access



#12 alexpaton OFFLINE  

alexpaton

    Member

  • Members
  • 26 posts
  • Local time: 11:18 AM

Posted 13 March 2019 - 04:31 PM

I've just tried setting this up on Ubuntu 18.04, but got stuck at the comskip stage.  When I try to run autogen.sh, it comes up with the following:

Preparing the Comskip build system...please wait

Found GNU Autoconf version 2.69
Found GNU Automake version 1.15.1
Found GNU Libtool version 2.4.6

Automatically preparing build ... Warning: autoreconf failed
Attempting to run the preparation steps individually

Preparing build ... ERROR: aclocal failed

I would very much appreciate it if anyone could offer any help.







Also tagged with one or more of these keywords: dvr, post-processing, ffmpeg, ccextractor, comskip, comcut, srt, linux

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users