Moving Emby to New Server

[ebr 14917]

Make sure your initial library scan has finished and then re-evaluate the watched status.

[Rolivaras 2]

13 hours ago, Happy2Play said:

No and normally comes back to permissions assuming specific example is showing images.

If you test on a specific example and Edit image and change the image or even downloading the same image what does the server log show?  You can do the same by Edit metadata and hitting Save.

I checked here that Raidrive was no longer allowing write, read only. I activated the test subscription to see if that was the problem.

[Rolivaras 2]

It was Raidrive that stopped allowing writing anyway.
I'm trying to figure out how to update all the metadata without being on an individual movie basis.

[jriker1 16]

Thanks for the assist.  Note I have this working now in a Hyper-V virtual on Windows 2022 Server and even though I'm not technically passing thru the built in Ryzen 7950x GPU, it shows it doing hardware transcoding.  Everything is faster, knock on wood, thank my physical server was which is odd considering it was a dual Xeon server so was no slouch.  Maybe the NVMe drives are helping.

[Carlo 4330]

NVMe, M.2 & U.2 drives make a huge difference with virtual machines due to their higher IOPs. Just to give you an idea here's a rough idea of IOPs to expect by different drive type.

HDDs
SATA/5400 60 IOPs
SATA/7200 75 IOPs
SAS/10K or FC 10K    or NL/SAS 150 IOPs
SAS/10K Enterprise Drive 250 IOPs
FC/15K 180 IOPs
SAS/15K 200 IOPs
SAS/15K Enterprise Drive 350 IOPs

SATA SSD
SATA 6 Gbps SATA 2.5"SSD (FireCuda 120 1TB)
   Read:100K IOPs, Write 90K IOPs
   Sequential Read:560 MB/s, Write:530MB/s

NVMe PCIe 3.0
PCIe3.0 Gen3x4 SK Hynix P31 2TB
   Read:570K IOPs, Write 600K IOPs
   Sequential Read:3,500 MB/s, Write:3,200 MB/s

NVMe PCIe 4.0
PCIe4.0 Gen4x4 NVMe Samsung 990 Pro SSD 1TB
   Read:1,200K IOPs, Write:1,550K IOPs
   Sequential Read:6,635 MB/s, Write:4,943 MB/s

PCIe4.0 Gen4x4 NVMe SK Hynix Platinum P41
   Read:1,400K IOPs, Write:1,300K IOPs
   Sequential Read:7,371 MB/s, Write: 6,728 MB/s

So yes, as you can see NVMe and even SATA SSDs can provide much faster access to storage which normally is the bottleneck in the system.

For VMs and Databases reading/writing 8K blocks Intel Optane is still the Storage of the Gods (Enterprise). A good way to think about Optane is a cross between RAM and NVMe performance wise. Optane has crazy low latency of 6 to 30 microseconds while NVMe is closer to 90 microseconds. It's endurance is second to none at 100 DWPD for 5 years.  That means you can write 100 times the size of the drive every day for 5 years and it's warrantied to still be 90%.

Each of my Intel 5800X do about 2.5 million IOPs but hit a bottleneck saturating the 4 PCI Lanes of the U.2 interface or it could be even faster. Optane never caught on in the consumer market due to PCI lanes needed and the crazy costs of close to $5K for a 1.6TB drive. Optane drives were available a couple ways like this U.2 drive or 288-pin DDR4-2666 PC4 Persistent Memory which of course requires a server motherboard designed to use Optane this way. You can configure the Persistent Memory to act like memory or SSD this way (or hybrid).  The memory stick style Optane drives have the lowest latency and shortest path directly connected to the CPUs. You can certainly use them as a conventional NVMe drive but they typically are used as part of tiered storage or non-volatile memory.

The reason I mention Optane is because they are no longer produced and not hitting expectations of profit.  You can find both new and used drives on eBay at fire sale prices which makes them worthwhile for high-end consumer use.

[richardvrusso 10]

On 1/12/2023 at 6:06 AM, Carlo said:

I would first configure the machine/environment, so you have the same paths available with proper permissions.

I would copy all the main subdirectories from old to new system. This would include the cache, metadata config, data directories at the least.

What I actually do is a clean install to look at the directory layout. If both locations are the same under EmbyServer I mass copy everything from EmbyServer down to the new server.
I would then check and setup permissions and ownership if needed.
Once that's done I would reinstall the EmbyServer package again to overwrite any executables or DLLs in case we go from Windows to Linux or similar.

If done correctly and you have already setup the environment to look the same path wise you should be able to start the server and it works. If you can't match things exactly doing a full library scan should fix that and make whatever changes are needed.
I'd then go to database menu, set the memory used for caching as well as enable optimization for the next server restart.

It really shouldn't be a hard process if you have access to both the existing and new machine.

Carlo

I am doing a fresh install on unRAID. I've never altered the database menu cache size. I'm guessing it depends on what hardware you have? should it not be set too high? Right now it says 96MB

[Happy2Play 8292]

1 minute ago, richardvrusso said:

I am doing a fresh install on unRAID. I've never altered the database menu cache size. I'm guessing it depends on what hardware you have? should it not be set too high? Right now it says 96MB

Depends on how much media you have as 96MB is default .  If you have the RAM you would adjust the cache size according to your database size.

 

[richardvrusso 10]

This is awesome, thank you! I'm going to check my old emby server too.

[them8os235 16]

On 1/15/2023 at 12:46 AM, Carlo said:

NVMe, M.2 & U.2 drives make a huge difference with virtual machines due to their higher IOPs. Just to give you an idea here's a rough idea of IOPs to expect by different drive type.

...[snip]...

The reason I mention Optane is because they are no longer produced and not hitting expectations of profit.  You can find both new and used drives on eBay at fire sale prices which makes them worthwhile for high-end consumer use.

 

 

Let me jump in on this thread instead of starting a new one as [A] it can't be more perfect to what I'm presently doing -- as now that I've discovered IPTV and OTA recordings and my storage needs have exploded, I'm also setting up Emby, again, now for a 3rd time -- and [B] I just bought a [I hope] new Intel Optane H20 512 Gb version at the fire sale price of $80 off Amazon!  With that I have a question about how to deploy that H20 and am hoping you (or anyone, all should feel free) can offer an informed opinion about it.

I'll keep the component details to a minimum; here's a pertinent sub-set of what I've got arriving today and tomorrow:

• ASRock Z690 Extreme
• Intel i7-12700K (which I intend to not overclock as I do w/my i7-7700K but rather both underclock and undervolt; unless I'm using the system interactively to rip disks or other things where I can use the ASRock A-tuning utility to easily change the system's operation to High Performance).
• Presently have two Seagate SATA3 IronWolf 12Tb disks that I manually use as a selective mirror in a 2-bay drive replicator (which is working quite well performance wise, actually, given its 10Gbps usb-c interface), but I intend to get two more and run all 4 in a RAID 10 via Intel RST.  This part is important and at the heart of my question I'm about to ask.
• Will redeploy 32 Gigs of G.Skill DDR4-3200 under utilized from my daily driver (which presently has 64 Gigs in it, and I never come close to utilizing it all).
• Could re-deploy an 'old' under-utilized 1Tb Samsung 960 EVO M.2 NVME into this new system... or not.  

 

So, to that end my question is how do you think I should best deploy/utilize that H20? 

Here are a few ways I believe I can use it (I may actually be incorrect about how I can even use the H20, and feel free to correct me when I'm wrong).  Also, I'll be using Windows 11 Pro as the OS.

1. Use the 512 Gb Optane H20 as my C:\ Drive for the OS and Emby-Server software and cache location and via Intel RST to use the true 32 Gig Optane memory cache section to accelerate it?
2. Use the Samsung 1Tb M.2 NVME drive I have for the Windows OS C:\ drive and the Emby Server software and cache location, and use the Optane memory section of the H20 as an accelerator for it?
3. Same as #2 for the OS but set the Optane H20 SSD and cache memory up as a D:\ drive and put the Emby Server software and cache on that?
4. Similar as the two immediately above for the Windows OS but use the Optane H20 as an cache accelerator for the Emby media library located on the 4 spinning disk RAID 10 array?

My intent originally with the Optane H20 was to use it as option #4.  But, I didn't really realize what I was buying *lol*.  And learned the H20 is a "hybrid" device, not purely a "Optane Memory" cache accelerator device.  So the more I read about the H20 it seems I should use the H20 as the OS and Emby-Server device, i.e. #1, or alternately as described in #3.  In part because, I believe, caching and accelerating the Emby-Server and cache directory would have a greater effect than the media library, as (correct me if I'm wrong) the huge number of meta files and images are read from the Emby-Server directory not the files located in the media library directories (those are only use when scanning media/metadata, correct?), and caching the multi-gig .ts video files aren't as necessary or useful as caching the meta / .nfo / images files?

Would really love to hear your thoughts about this, Carlo.  But anyone w/an informed opinions feel free to chime in. And much thanx all for your time in this regard.

 

Edited March 26 by them8os235

[richardvrusso 10]

Hey them8os235,

Fantastic post first off! I feel like I wrote it myself.

I'm no emby expert but I've been around building PCs since the 90s. also 20+ years in corporate world designing in the enterprise space. I'm super intrigued how the H20 performs. Seen a few YT vids, most notably Level1Techs, show them off. Others can correct if I'm mistaken but you might not be able to independently utilize each side of the H20. So it's just 1 drive for _______.

Intel seems to think the H20 would make a good OS drive. https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://download.intel.com/newsroom/2021/storage-memory/optane-memory-h20-solid-state-storage-product-brief.pdf&ved=2ahUKEwjkgum795KFAxWIKlkFHVbHC1QQFnoECFQQBQ&usg=AOvVaw2X5osb3r27xEJBFoiRvv3B

What I'm curious to know if this was me, what has more throughput? The H20 or the Samsung NVME? Maybe 1 is faster than the other and you'd be better off keeping it simple and single drive for OS/emby. Another thing came to mind, is the emby cache where transcoding happens? Then you could split out the cache to it's own dedicated drive. 

[them8os235 16]

@richardvrusso I relate to your history.    My first PC upgrade was putting in an I386 daughter board into my i286 based IBM Model 70 in the early 90s. Other than laptops that's the last pre-built machine I purchased, building a new one every 5-10 years. ... Pardon the delay responding.  I was waiting for our resident expert in this realm of Optane given his great post above, Carlo, to chime in on my question above, to combine responses.  But he's no doubt pretty busy with official support or other Emby related things so I won't worry about it now.  But @Carlo(now that I've tagged you) if you do come here and now read my above question and have an opinion, I'd love to hear it.  But I think I have my answer, noted below.

Yep, I too had finally found & read the H20 isn't geared to accelerate slow storage, like stand-alone Optane Memory is w/o the Hybrid SSD section.  A good summary that finally clicked with me in regards to what it [the H20] is intended and good for what, here:  https://www.intel.com/content/www/us/en/support/articles/000058286/memory-and-storage/intel-optane-memory.html

I'm about to embark on using the Optane H20 as my system OS + Emby Server install location (my motherboard has Intel VMD & RST in the bios, and name support for the H20 specifically, which is required to utilize it).  Its random IIOP rating is incredible. So I hope Emby Theater and the Webapp will fly on it when I interact with it remotely.  And given that the H20, with its SSD section being based on 'legacy' Intel SSD tech which has long trailed the likes of Samsung and almost all others in sustained large block read/write performance, I'm going to put the Windows swap file onto the spare Samsung M.2 NVMe drive I have instead of keeping it on the H20.  This review put that question to bed, clearly demonstrating the only place the H20 performance doesn't trail is in random small block IO: https://www.tomshardware.com/reviews/intel-optane-memory-h20-m2-nvme-ssd-review-running-optane-and-flash-in-harmony  ... if I'm left concerned about anything when using it for the core system OS drive it'll be that the SSD section's endurance is so low, which is ironic given how high the endurance is of the Optane memory section.

I'll put all my libraries on a RAID10 without the enhancement of Optane acceleration, consisting of 4x 12Tb Seagate IronWolf drives, which are excellent performers. The two I bought a few months ago used in a non-RAID implementation are giving me 4x the sustained read/write performance over my 2Tb IronWolf drives I've had for years. Far better performance than the 7200rpm in the 12Tb drive vs the 5400rpm in the 2Tb drive would suggest, leaving me scratching my head as to why those are so slow: With cache enabled I get the spec'd 280MB or better write perf of the 12Tb drive, 240-260MB with cache disabled, but only 40-60MB out of the spec'd 180MB with the 2Tb drives (and of course Intel RST in RAID5 is complete sh1t reducing that write performance to about a dozen MB or lower when used in that manner).  But boy are those 12Tb IronWolf drives noise!  And in my last sentence just like the first one above, going back to the wayback machine it reminds me of the noise my full height 5.25" 1Gig drive (Maxtor, I believe?) made, that I bought in the mid 90s and had an MSRP of $2k [per it's 1Gig!].  lol 

[richardvrusso 10]

@them8os235ahh days of x86, DOS only and memmaker, to optimize that whole megabyte of ram.   

Great url too. I wonder if a hybrid would work well in an unRAID box as a cache drive. though not sure if my mobo supports optane. 

So will the hybrid be your OS and emby install? 

[them8os235 16]

2 hours ago, richardvrusso said:

@them8os235ahh days of x86, DOS only and memmaker, to optimize that whole megabyte of ram.   

Great url too. I wonder if a hybrid would work well in an unRAID box as a cache drive. though not sure if my mobo supports optane. 

So will the hybrid be your OS and emby install? 

I was put'n that meg (or few) of RAM to optimal use multi-tasking via Quarterdeck DESQview! 

Yes, I'll be putting the OS and Emby-Server on the Optane H20, and will experiment with putting the swapfile on a Samsung M.2 NVMe drive which has better multi-gig file performance; tho not sure how I can stress-test the speed impact of the swapfile location. I'll post an update here detailing the results of the PassMark, DiskMark and other benchmark tests once I get it all operational.  

[ebr 14917]

12 hours ago, richardvrusso said:

that whole megabyte of ram

Actually only 640k  .  Remember 384 was dedicated to display memory.

[Dickydodah! 128]

43 minutes ago, ebr said:

Actually only 640k  .  Remember 384 was dedicated to display memory.

640k!!! sheer luxury I remember programming in 1k on my first ZX81 in the early 1980's........................................... god I'm getting old 

[jaycedk 382]

Your not getting old, if you where, you would not remember

[richardvrusso 10]

Wait what we're we talking about?

[them8os235 16]

We digress.    Back on-track:

In regards to bringing my new server online @ebror  @Carlo or any others in the know ...

As discussed with others much earlier in this thread.  I too will be restoring my backup of my current server on my new server.  And I'll be copying all the media files over to the new server before that.  However, here's the thing: 

1. On the new server the HDD disk letter mapping (and maybe root path naming) won't be the same as on my old server.  So there will be some time necessary (many hours) after bringing the server up then changing the disk library path locations, before the libraries are 'ready'.
2.  The current [old] server configuration I'll be restoring has dozens of scheduled HDHomeRun tuner recordings.  If they are allowed to immediately activate on the new server, they will duplicate the recordings and quickly exceed the 4 tuner count of my HDHomeRun device.

So, the question:

Is there a way I can restore my Emby-Server backup on the new server, but pause [NOT cancel ] the Live TV series and movie recording schedules I have defined on it?  Until I'm ready to switch that over too to the new server. Implied here is I do not want to cancel any of the series and movies scheduled recording that are restored into the new server's runtime configuration.   I ask, as I don't see this as a Live TV config option.  But maybe I missed it or it can be achieved by modifying a config file directly?

If not, all I can think of to do is to simply not bring the new server online until I'm ready for it to perform all my recordings too and switch immediately at that time over to it, and perform all the Library path alterations after that... Which is the very thing I wanted to avoid as scanning media of dozens of libraries containing thousands of media resources makes Emby interaction pretty unresponsive, and why I wanted to do that before I started using the server for real-time Live TV interactions and recordings. 

Please advise if you have other guidance you can provide that would be better than that route.  Thanx!

Edited April 4 by them8os235

[Luke 37087]

@them8os235you should use the emby backup and restore process:

Configuration Backup

However just be aware there is no way to selectively restore only Live TV settings, so it is a complete restore from the backup.

[them8os235 16]

On 4/16/2024 at 1:19 AM, Luke said:

@them8os235you should use the emby backup and restore process:

Configuration Backup

However just be aware there is no way to selectively restore only Live TV settings, so it is a complete restore from the backup.

On the first sentence I'm pretty clear and what I intend to do. 

If you can please confirm or correct however, I think I realized how I can accomplish "disabling" Live Recordings without losing Channel Mappings, and restoring the scheduled recordings later by doing the following:

1. Install the same Emby-Server (beta) version on the new server as I run on the current server.
2. Copy over the latest backup from the current server.
3. Restore the backup on the new server.
4. Now here is what I'd like confirmation on: Immediately clear the contents of seriestimers.json and timers.json under the path 

   C:\Users\[username]\AppData\Roaming\Emby-Server\programdata\data\livetv

5. Restart Emby-Server.
6. Reconfigure the restored Library's paths as needed specific to the new server's filesystem.
7. When I'm ready to switch-over Live TV recording to the new server, copy over only the files seriestimers.json and timers.json from the previous to the new server, and restart.  

Does that seem like it'll do what I'm hoping?  Much thanx.

 

[Luke 37087]

Hi, yes in theory that sounds ok.