Cardinal Library Sin - Tdarr H.265 Transcoding

[Apotropaic 64]

A bit of a back story first, years ago I started coming across more and more forum folk talking about transcoding media to save space, which I thought was insane due to the time, effort and cost it would require. I just didn't see the value and my thoughts were always - just buy another drive, expand the pool, add another microserver...

I've now realised maybe I was the fool here ;( I'm now worried about the mental energy of maintaining my storage army, my drives failing, my physical media degrading.

So while I've been downsizing my fish tank over the past few months (someone else can save the corals).  I've also decided my leadership of media preservation is over and someone else can deal with it all!

So my 1080p content library is for the chop! I've been playing around with a number of tools and settled on Tdarr. Subjective quality isn't my priority, I've gone with 'half the bitrate of H264' as I'm on a 120 inch screen so a lot of stuff looks naff on it, if it's 5-10^%worse I won't really care and no one else (in my household) would ever notice it on a smaller screen anyway. My favourite films I have in 4k anyway so this is content is more sentimental than anything else.

My only transcoding concerns were to set a limit of a minimal H.264 bitrate which I wouldn't transcode, to keep the audio soundtracks and subs as they are, I don't want to save space by downmixing stuff to Stereo. I also wanted it to be semi-automated to keep some control over what gets transcoded (I want my favourite films to be held back for as long as possible in case I come across some new wisdom.

Finally, I wanted this to be GPU accelerated on my Macbook, it's the only decent option I have. In a perfect world I would have liked to have encoded to AV1or maybe even H.266/AV2 but not a choice I have right now.

So open question, has anyone done something similar, or thinking about it? Oh and in terms of real figures, so far 49 films completed, 213gb saved, took maybe10 hours'ish.

[Eigeplackter 95]

Between years moved from a Synology Rackstation to self-build TrueNas for storage and to Fedora Linux with a ARC GPU.

As I rearranged my drives and moved bit for bit, I decided to encode all my H264 files to AV1-10bit, as prices for new drives where already rising.

 

I tried tdarr years ago and I‘m not a fan of it, so I made decode/encode with ffmpeg. I started around 58TB of media and now I‘m

down to 42TB, As for the time … it took around 4 weeks nonstop. Initially it was really slow as I had to adapt the script several times.

At the end it was just ploughing through.

As this was created by Gemini I‘ll share my script with the community, maybe it can help, not with your specific setup.

#!/bin/bash

# Erzwinge den Intel Media Driver für Systemd-Umgebungen
export LIBVA_DRIVER_NAME=iHD

# --- KONFIGURATION ---
BASE_DIR="/mnt/Serien"
SCRATCH_DIR="/scratch/encode"
LOCK_DIR="/tmp/av1_locks"
SCAN_FILE="/tmp/scan_list.txt"
DEBUG_LOG="/tmp/av1_debug.log"
DASHBOARD_URL="http://localhost:8080/api/update"
NUM_WORKERS=3

mkdir -p "$SCRATCH_DIR" "$LOCK_DIR"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$DEBUG_LOG"; }

# --- AUTOMATISCHER SCANNER ---
if [[ ! -f "$SCAN_FILE" ]] || [[ ! -s "$SCAN_FILE" ]]; then
    log "Scanner: Starte Scan in $BASE_DIR (Suche NUR H264 MKVs)..."
    (
        find -L "$BASE_DIR" -type f -name "*.mkv" -size +50M 2>/dev/null | \
        grep -i "\[h264\]" | \
        grep -ivE "\[h265\]|hevc|x265|\[AV1\]|#recycle|@eaDir|\.error" > "$SCAN_FILE"
        log "Scanner: Scan abgeschlossen ($(wc -l < "$SCAN_FILE") Dateien gefunden)."
    ) &
fi

notify_dashboard() {
    local status="$1" filename="$2" saved_mb="${3:-0}"
    # Dashboard-Update ohne Hintergrundprozess für bessere Zuverlässigkeit
    curl -s -X POST "$DASHBOARD_URL" -H "Content-Type: application/json" \
         -d "{\"filename\": \"$filename\", \"status\": \"$status\", \"saved\": $saved_mb}" > /dev/null
}

get_dynamic_quality() {
    local input="$1"
    local height=$(ffprobe -v error -select_streams v:0 -show_entries stream=height -of csv=p=0 "$input" 2>/dev/null)
    [[ -z "$height" || ! "$height" =~ ^[0-9]+$ ]] && echo "20" && return
    if [ "$height" -le 480 ]; then echo "23"; elif [ "$height" -le 720 ]; then echo "21"; elif [ "$height" -le 1080 ]; then echo "20"; else echo "18"; fi
}
do_work() {
    local worker_id=$1
    sleep $((worker_id * 2))
    while true; do
        for path_file in "$SCRATCH_DIR"/*.path; do
            [[ -e "$path_file" ]] || continue
            
            # Pfade sauber trennen
            local path_file_name=$(basename "$path_file")
            local candidate_name="${path_file_name%.path}"
            local lock="$LOCK_DIR/${candidate_name}.lock"

            if ( set -o noclobber; echo "$$" > "$lock" ) 2>/dev/null; then
                cd "$SCRATCH_DIR"
                local current_in="$candidate_name"
                local current_out="${candidate_name%.mkv}.out.mkv"
                
                # Pfad zum Original aus der .path Datei lesen
                local original_path=$(cat "$path_file_name")
                local orig_filename=$(basename "$original_path")
                local target_quality=$(get_dynamic_quality "$current_in")

                log "W$worker_id: Start $orig_filename (Q$target_quality)..."
                notify_dashboard "encoding" "$orig_filename"

                # --- DER OPTIMIERTE FFMPEG BEFEHL ---
                # Wir nutzen jetzt exakt die Parameter aus deinem erfolgreichen Test
                ffmpeg -hide_banner -loglevel error \
                    -hwaccel qsv -hwaccel_output_format qsv -qsv_device /dev/dri/renderD128 \
                    -i "$current_in" \
                    -vf "vpp_qsv=format=nv12" \
                    -c:v av1_qsv -preset slow -global_quality "$target_quality" \
                    -map 0:v:0 -map 0:a -c:a copy -map 0:s? -c:s copy \
                    -y "$current_out"

                FF_EXIT=$?
                
                # VALIDIERUNG
                if [ -f "$current_out" ]; then
                    NEW_SIZE=$(stat -c%s "$current_out")
                else
                    NEW_SIZE=0
                fi

                # Mindestens 50MB für eine Folge (Sicherheitsgrenze)
                if [[ $FF_EXIT -eq 0 && $NEW_SIZE -gt 52428800 ]]; then
                    local size_orig=$(stat -c%s "$current_in")
                    local saved_mb=$(( (size_orig - NEW_SIZE) / 1024 / 1024 ))
                    local folder=$(dirname "$original_path")
                    local clean_base=$(basename "$original_path" .mkv | sed -E "s/ \[[hHxX]26[45]\]//gI")
                    local target_file="$folder/${clean_base} [AV1].mkv"

                    if mv "$current_out" "$target_file"; then
                        # WICHTIG: Nur löschen, wenn Ziel existiert!
                        if [ -f "$target_file" ]; then
                           rm -f "$original_path"
                           log "W$worker_id: Erfolg $orig_filename (-${saved_mb}MB)"
                           notify_dashboard "done" "$orig_filename" "$saved_mb"
                        fi
                    else
                        log "W$worker_id: MV FEHLER"
                        notify_dashboard "error" "$orig_filename"
                    fi
                else
                    log "W$worker_id: FEHLER (Exit: $FF_EXIT, Size: $NEW_SIZE). Original sicher!"
                    notify_dashboard "error" "$orig_filename"
                    rm -f "$current_out"
                fi
                
                rm -f "$current_in" "$path_file_name" "$lock"
                break
            fi
        done
        sleep 10
    done
}

do_staging() {
    while true; do
        local ready_count=$(ls -1 "$SCRATCH_DIR"/*.path 2>/dev/null | wc -l)
        if [ "$ready_count" -lt 4 ]; then
            local next=$(flock -x "$SCAN_FILE" sh -c "if [ -s \"$SCAN_FILE\" ]; then head -n 1 \"$SCAN_FILE\" && sed -i \"1d\" \"$SCAN_FILE\"; fi")
            if [[ -n "$next" && -f "$next" ]]; then
                local bname=$(basename "$next")
                log "Staging: Kopiere $bname..."
                if cp "$next" "$SCRATCH_DIR/$bname"; then
                    echo "$next" > "$SCRATCH_DIR/$bname.path"
                    log "Staging: $bname bereit."
                fi
            else
                sleep 60
            fi
        fi
        sleep 20
    done
}

log "--- AV1 ENGINE START (Safety Mode) ---"
do_staging &
for ((i=1; i<=NUM_WORKERS; i++)); do do_work "$i" & done
wait

 

 

 

[kikinjo 298]

Using tdarr for years, i have gone easier way, configured % target birate modifer to 50%, i m using Boosh transcode QVS plugin fully automated with my Intel Arc 310 gpu wich

is really a small monster for transcoding. already transcoded like 2-3k files. I would suggest to not go for AV1, many clients / hardware / browsers do not support it, stick to plain

x265 codec. 

[Eigeplackter 95]

3 minutes ago, kikinjo said:

…. I would suggest to not go for AV1, many clients / hardware / browsers do not support it, stick to plain

x265 codec. 

Uh ? Chrome, Firefox support AV1 native. Edge needs a free extension. Fire TV 4k Max supports AV1,  even a NVIDIA Shield (2019) does.

But yes, you might need to transcode more often, if your clients are older, i.e. older FireTV Sticks.

[yocker 1456]

I can recommend Fileflows instead of Tdarr.
Easier to work with and more options.

Or if you want complete easy mode there is always Unmanic.

As Kikinjo said, it's recommended to stay away from AV1.
Most devices do not support it so transcoding will be required on the server and AV1 is still somewhat rare on GPUs unless you have a newer one.

[Neminem 1644]

Also using Tdarr transcoding to H265.

This is the space I have saved until now

That more that 1 20TB HDD

I'm also staying away from AV1 for the moment.

Most of my family is using old clients or old TV's

Edited 5 hours ago by Neminem
Adding

[Apotropaic 64]

I knew I was late to this party but looks like I'm not alone which has given me some confidence I'm doing the right thing

I also should have said I'm encoding to 10-bit H.265.

I tried Unmanic and although I got it to work I wanted the challenge of getting Tdarr to work with my particular setup. 

17 minutes ago, yocker said:

I can recommend Fileflows instead of Tdarr.
Easier to work with and more options.

FileFlows was going to be my next attempt if Tdarr hadn't worked, I still have a few thousand files to go so will spin up FileFlows and see how it looks, thanks.

[yocker 1456]

3 hours ago, Apotropaic said:

I knew I was late to this party but looks like I'm not alone which has given me some confidence I'm doing the right thing

I also should have said I'm encoding to 10-bit H.265.

I tried Unmanic and although I got it to work I wanted the challenge of getting Tdarr to work with my particular setup. 

FileFlows was going to be my next attempt if Tdarr hadn't worked, I still have a few thousand files to go so will spin up FileFlows and see how it looks, thanks.

Remember that converting files might loose you Dolby Vision.
I have personally not seen a method that converts Dolby Vision videos using GPU either, only using CPU.

@Neminem You know of any way to do it?