My home media server project got a little out of hand…

[skidmarks 201]

I’ve been building a pretty ambitious home media/server monitoring and recovery system around Emby, Plex, Cloudflare, and the rest of my home infrastructure.

The goal is simple: if something important goes down while I’m away, the system should be able to recognize what actually failed, avoid making the problem worse, recover the right thing automatically when it’s safe to do so, and leave a clear record of what happened.

Cloudflare is also a significant part of the design. I know there has been a lot of discussion lately about whether Cloudflare is still a practical option for home media environments. Without getting into the nuts and bolts of my implementation, I can say that I’m actively using it as part of a secure remote-access architecture and have been deliberately testing not only normal operation, but failure and recovery behavior as well.

It’s grown into a lot more than a dashboard. I now have health monitoring, service checks, dependency awareness, controlled automatic recovery, startup protection, failure history, and safeguards to keep one bad recovery attempt from taking down everything else.

One of the more interesting recent tests started with Plex acting strangely after a reboot. Plex eventually recovered on its own, but the recovery system exposed a completely different hidden problem: one stuck recovery attempt could block recovery for another service. That led to redesigning the recovery layer so each service is handled independently, with its own limits and safety rules.

I’m also documenting the whole thing as I go so it can be rebuilt from scratch years from now instead of becoming one of those “only the guy who built it understands it” projects.

It’s still very much unfinished, but it’s now at the point where I can deliberately break parts of the system, watch it determine what actually failed, recover what it should, refuse what it shouldn’t, and prove afterward what happened.  I am also in the middle of building the API that will work hand and hand with my control panel to visually inspect and repair IF the automation should fail its checks and balances.

For a home media setup, it has gotten a little ridiculous.

And I’m having way too much fun with it.  

There are a few control panel elements, not the entire panel, that i am willing to share for now.  I now completely understand what the developers go thru to deliver an application with little to no bugs every update that is released.  Months and more months have been put into this and probably many more before everything i Invision is implemented.   

 

See guys i didn't go away, i took a break and have come back with a little project i've been working on.

Edited August 23 by skidmarks

[ebr 16691]

Cool.  Hopefully, it doesn't end up with "Sorry, Dave..."

[skidmarks 201]

We’ve discussed this. HAL is under strict instructions not to lock the pod bay doors.

I didnt just want remote access to HAL, that is way too easy, i needed to create a command-and-control infrastructure that healed itself.  Learned a lot, not so easy...

  

Edited August 23 by skidmarks

[skidmarks 201]

Well, did a bare metal build of my computer.  Complete rebuild of my Autonomous System, went without a hitch!  Other than a installer package my project is finished.  Every service is capable of looking at failure conditions, analyze all its dependencies, determine what caused the failure (within predetermined guidelines) and restart and fix any service autonomously.   

This is HUGE!  Pretty proud of it...  

Can see:

CPU and Memory Utilization

In Bound / Outbound Traffic totals

Top CPU Processes (3) / Top Memory Processes (3)

Storage Health / Storage Detail

All Services Health

All services Status

Number of Devices Connected / Number Applications Seen

PLEX / Emby / PIA Current Traffic

AND YES THE WEATHER from my personal weather station.  WHY?  Because ALL of these items are part of the autonomous recovery scripts.  Weather can tell WHY (or at least give a clue) why the connection (Starlink) to my gateway at a certain time was unavailable.  

Everything is monitored with set guide lines.  Everything is documented.  Everything is visual all on one control panel.  Finished the google API.  Everything is secure everything is available on my cellphone.  If the automation fails, i can over ride the system and restart any service that is showing a fault condition.  

Over kill for a private server, absolutely!  The coolest way to know or repair a service or its processes?  No doubt!  A pain in the ....  well, yes it was.  But well worth every hour put into it. 

No more guessing, no more wondering why something is down, no more being 400 miles away and finding out the service X, Y or Z is down.  It should fix with out intervention.