Time shifting recording is messy

[matsonjr 6]

So this has happened a couple times now with football games...

 

The recording starts and I join late.  I try to catch up during commercials and often something happens and the recording stops and resumes.  For example the game I was watching Saturday has 11 different pieces.

 

What would cause that behavior?

 

My setup isnt conventional...

VM1 is truenas.  Passed through are the 12 sas spinner drives.  They are set as a raid z2.  Media and record folder is just an smb share

VM2 is ubuntu server with a 1080ti passed through and that has the shares mounted running emby.  

I'm also recording four network security cameras to the pool.  All I can think of is I cant write fast enough.

 

 

 

[Carlo 4560]

Hi,

You're recording to a network share?
If so, I would change that, so it records locally and doesn't have an IO slowdown.

Only guessing here without server logs, but you might have been transcoding the file during playback.  If so, when you try and jump to catch up the transcoder has to process the file as fast as it can to catch up to where you jumped.  It could be that you're just waiting for it to catch up to where you jumped.

You should be able to see this on the Dashboard for the playback.

[Luke 42077]

@matsonjr has this helped?

[matsonjr 6]

 

On 10/20/2024 at 11:51 PM, Carlo said:

Hi,

You're recording to a network share?
If so, I would change that, so it records locally and doesn't have an IO slowdown.

Only guessing here without server logs, but you might have been transcoding the file during playback.  If so, when you try and jump to catch up the transcoder has to process the file as fast as it can to catch up to where you jumped.  It could be that you're just waiting for it to catch up to where you jumped.

You should be able to see this on the Dashboard for the playback.

It isn't really a traditional network share.  All the drives are housed on the same machine and go through the proxmox internal software network switch, which is something like 100Gbps...So I wouldn't think that is the issue.  And yes, it does transcode.  

On 10/23/2024 at 7:44 AM, Luke said:

@matsonjr has this helped?

I haven't tried much yet.  I'm still not sure what exactly to try.  Add an nvme for recordings?  That requires taking the system down to majorly reconfigure things.  

 

Now I'm suspicious I found the problem...the temp conversion folder was set to default.  It barely has any space remaining on the install disk...hence everything on the big pools.

 

ETA: in a situation where I'm recording a long recording and it does transcode, should I throttle the transcodes?

Edited October 25, 2024 by matsonjr

[matsonjr 6]

2 hours ago, matsonjr said:

 

It isn't really a traditional network share.  All the drives are housed on the same machine and go through the proxmox internal software network switch, which is something like 100Gbps...So I wouldn't think that is the issue.  And yes, it does transcode.  

I haven't tried much yet.  I'm still not sure what exactly to try.  Add an nvme for recordings?  That requires taking the system down to majorly reconfigure things.  

 

Now I'm suspicious I found the problem...the temp conversion folder was set to default.  It barely has any space remaining on the install disk...hence everything on the big pools.

 

ETA: in a situation where I'm recording a long recording and it does transcode, should I throttle the transcodes?

I think I got it figured out...think anyway.

 

Yes, I was running out of space for the default temp files, so I changed it to a temp directory in the smb share.  It didn't like that.  Not sure why, but it was really giving me fits.  So, I made an iSCSI share on the NAS and directed it into emby for the temporary stuff.  Then I have transcoding set to not throttled (although I'm not sure if it should be throttled or not).  That seems to have solved it.   

[Carlo 4560]

2 hours ago, matsonjr said:

It isn't really a traditional network share.  All the drives are housed on the same machine and go through the proxmox internal software network switch, which is something like 100Gbps...So I wouldn't think that is the issue.  And yes, it does transcode.  

I haven't tried much yet.  I'm still not sure what exactly to try.  Add an nvme for recordings?  That requires taking the system down to majorly reconfigure things.  

 

Now I'm suspicious I found the problem...the temp conversion folder was set to default.  It barely has any space remaining on the install disk...hence everything on the big pools.

 

ETA: in a situation where I'm recording a long recording and it does transcode, should I throttle the transcodes?

Are you using ZFS?
Are you running Emby in a VM or LXC container or as a docker container in a VM?

If running in an LXC container you can simplify disk access avoiding the need to use SMB, NFS or iSCSI running the data through the client network stack, the emulated NIC, the emulated network, virtual switch or virtual bridge, host emulated NIC, host network stack, etc avoiding all the checksum calculations, buffer & queue managements associated with any network transfer all being emulated by the host/KVM or Proxmox software stack.

With LCX contains you can create a bind mount point (or device mount point) configured on the host. You can then use this bind point in the container configuration. This gives the LCX container direct access to the host resource providing host like performance and simplifying storage use.  https://pve.proxmox.com/wiki/Linux_Container#_bind_mount_points

VM NETWORKING & VIRTUALIO SETUP/TUNING
If you must use a VM with VirtIO you want to make sure you have an optimal setup (tricky) or the VM with VirtIO (paravirtualized) NIC will likely only get a max throughput of 15~30 Gbit/sec. With all the emulated layers the packets go through, you hit the limit of your memory speed and network queues start building up. Same when doing GPU or any device passthrough to the VM.

The standard VM machine config only supports PCI emulation by default.  To get higher performance you want to make sure IOMMU and SV-OR are enabled in your BIOS. You want to make sure IOMMU is enabled on the host OS as well as IOMMU passthrough mode. Then you need to make sure these modules load in the kernel: vfio, vfio_iommu_type &  vfio_pci.

You need to check/configure, if you have enough VF (Virtual Functions) setup for the device.  This way a GPU or NIC with SR-IOV support can provide multiple VFs per physical port to allow things such as checksum offloading inside a VM, reducing the (host) CPU overhead, latency as well as improve throughput. Not all VM machine types support this. 440fx and q35 machine types support PCI passthrough but PCIe passthrough is only available on q35 machines. Besides using q35, you also want to use OVMF instead of SeaBios, 

That will make a big difference, but switching to virtio-net can improve performance as it moves part of the virtio code from user space to the kernel.  Multi-Queue virtio-net with guest settings can provide the fastest virtual performance as it allows for more than 1 set of RX & TX queues between the host and guest VMs. These are dynamically adjusted based on number of vCPUs in use, so it can scale throughput.  The performance of an LCX container using a bind mount will still be higher, however.

ZFS
If using ZFS it's worth noting, LXCs use datasets and VMs use zvols. The main difference is that a zvol's volblocksize is fixed due to the nature of block storage. Every file on that zvol will use the same block size.

Comparatively, recordsize is dynamic and represents the maximum block size for a single file on a dataset. Meaning, if you write a 4K file to a dataset with a recordsize of 128K, it will only use a 4K block. However, if you do the same from a VM it will require the full block of 128K. Meaning every time it needs to process a single 4K record, it will actually need to process 128K of data which isn't great for performance or storage efficiency.

If you have a specialized workload like a DB with fixed size random IO, you'll get poor performance unless the recordsize (dataset) or block size (zvod) match the workload. 8K for a postgres DB, 16K for an innodb mysql db, 1M for media files, etc. 

The volblocksize can only be set for the whole volume so every VM would use the same block size which isn't optimal.  On the other hand, you can have as many datasets as needed each tuned for what is being stored on it.

 

 

 

 

 

 

[matsonjr 6]

4 hours ago, Carlo said:

Are you using ZFS?
Are you running Emby in a VM or LXC container or as a docker container in a VM?

If running in an LXC container you can simplify disk access avoiding the need to use SMB, NFS or iSCSI running the data through the client network stack, the emulated NIC, the emulated network, virtual switch or virtual bridge, host emulated NIC, host network stack, etc avoiding all the checksum calculations, buffer & queue managements associated with any network transfer all being emulated by the host/KVM or Proxmox software stack.

With LCX contains you can create a bind mount point (or device mount point) configured on the host. You can then use this bind point in the container configuration. This gives the LCX container direct access to the host resource providing host like performance and simplifying storage use.  https://pve.proxmox.com/wiki/Linux_Container#_bind_mount_points

VM NETWORKING & VIRTUALIO SETUP/TUNING
If you must use a VM with VirtIO you want to make sure you have an optimal setup (tricky) or the VM with VirtIO (paravirtualized) NIC will likely only get a max throughput of 15~30 Gbit/sec. With all the emulated layers the packets go through, you hit the limit of your memory speed and network queues start building up. Same when doing GPU or any device passthrough to the VM.

The standard VM machine config only supports PCI emulation by default.  To get higher performance you want to make sure IOMMU and SV-OR are enabled in your BIOS. You want to make sure IOMMU is enabled on the host OS as well as IOMMU passthrough mode. Then you need to make sure these modules load in the kernel: vfio, vfio_iommu_type &  vfio_pci.

You need to check/configure, if you have enough VF (Virtual Functions) setup for the device.  This way a GPU or NIC with SR-IOV support can provide multiple VFs per physical port to allow things such as checksum offloading inside a VM, reducing the (host) CPU overhead, latency as well as improve throughput. Not all VM machine types support this. 440fx and q35 machine types support PCI passthrough but PCIe passthrough is only available on q35 machines. Besides using q35, you also want to use OVMF instead of SeaBios, 

That will make a big difference, but switching to virtio-net can improve performance as it moves part of the virtio code from user space to the kernel.  Multi-Queue virtio-net with guest settings can provide the fastest virtual performance as it allows for more than 1 set of RX & TX queues between the host and guest VMs. These are dynamically adjusted based on number of vCPUs in use, so it can scale throughput.  The performance of an LCX container using a bind mount will still be higher, however.

ZFS
If using ZFS it's worth noting, LXCs use datasets and VMs use zvols. The main difference is that a zvol's volblocksize is fixed due to the nature of block storage. Every file on that zvol will use the same block size.

Comparatively, recordsize is dynamic and represents the maximum block size for a single file on a dataset. Meaning, if you write a 4K file to a dataset with a recordsize of 128K, it will only use a 4K block. However, if you do the same from a VM it will require the full block of 128K. Meaning every time it needs to process a single 4K record, it will actually need to process 128K of data which isn't great for performance or storage efficiency.

If you have a specialized workload like a DB with fixed size random IO, you'll get poor performance unless the recordsize (dataset) or block size (zvod) match the workload. 8K for a postgres DB, 16K for an innodb mysql db, 1M for media files, etc. 

The volblocksize can only be set for the whole volume so every VM would use the same block size which isn't optimal.  On the other hand, you can have as many datasets as needed each tuned for what is being stored on it.

 

 

 

 

 

 

I'll be honest, some of that flew over my head and will probably take me about four more times reading it...

 

BUT

Of what I did understand....

Machine is running proxmox and has a whole bunch of attached hardware.  So much, I'll probably even forget bits.  I'm not doing any LXCs...I haven't figured those out yet in my life.  Getting proxmox with several VMs running was quite the learning curve for a dumb farmer.

Top is a W-2191b with 128GB ECC with a 10gbe nic running proxmox.  

VM100 is truenas running q35.  Passed through are the SAS storage controllers and one small nvme drive.  Inside truenass, there are 12 10tb SAS drives running RaidZ2 with the nvme acting as a slog.  At home, we all have an iSCSI share where we keep our steam library.  Then there is my media library and last use is NVR storage.  My writes are slow (expected behavior) but my reads are very fast.  That is fine considering we really don't do much writing to the pools.  Reads can saturate the 10gbe.    

VM101 is Ubuntu server 24.04 running q35.  Passed through are a GTX 1080ti and a coral TPU.  Inside Ubuntu I'm running I've got the nvidia docker runtime stuff so the GPU can be shared.  The apps I'm running are: immich, emby, frigate NVR and crafty controller.  Three of those use the gpu.  Although, I pretty much moved all my crafty to a 5950x server considering the 2191b single thread performance isn't good enough...

VM102 is Home Assistant OS running q35.  It just has a Zwave dongle passed through.  

I did just run an iperf test truenas to ubuntu and was getting ~30Gbits/sec.  

No that electric eel is about out and it is docker based, I wonder about keeping everything in truenas if it can share the gpu.  But then again, now that it is fixed, if it ain't broke, don't fix it.  

 

The biggest upgrade I need to make is upgrade the boot drive from 500GB.  It sounds like a lot, but really...it isn't.

 

[Carlo 4560]

If things are working, leave it alone for now and maybe try playing with LCX containers. If you don't pass the drives through to TrueNAS, Proxmox would be able to import them directly. Using Bind Mounts would not require user/group permissions inside the contains either. 

I was really excited early on for TrueNas SCALE. It was supposed to scale out, with distributed apps and distributed storage using Kubernetes and GlusterFS under the hood with a GUI to easily manage your server or cluster.  GlusterFS has been sunset. "Kubernetes" is a custom k3s implementation running Docker with Containerd under the hood. Neither Kubernetes or docker is available to use outside of SCALE control as it's just to run their stack. In May they posted in their forums they are removing mention of Kubernetes from the product.

What many people end up using it for is easy GUI management of ZFS, user and group permissions for file sharing. All pretty easy to do from the command line once you make a  point of learning how to do it. Rancher & Harvers using Linbit SDS is what TrueNas SCALE was supposed to be.

This post on their forum, sums it up the current state of TrueNas SCALE.

Spoiler

https://www.truenas.com/community/threads/how-did-we-get-to-such-a-ludicrous-position-with-containerisation.117442/
I mean, we have the 'apps' section which has two catalogs as far as I know, one that has limited apps and one that makes some annoying decisions for you, both of which have at times some pretty serious problems and limitations as documented elsewhere in this forum.

Then we have the custom apps which is where we add something that is not on the list, usually where people refer to the place where they can 'finally get what they want, docker'. But it runs on an engine which is not docker and is encapsulated by Kubernetes causing more problems as documented elsewhere in this forum. Among these problems a particular killer for me is not being able to set a hostname. And I'll just throw out another example of the term ludicrous by mentioning the limitation on ports lower than 9000.

Then we have VM's which seems to be the last ditch effort for people to get sanity for example with docker, which does work well but has other distinct disadvantages, as documented elsewhere on this forum (RAM, Disk, Shares, Permissions, Accounts etc)

Now, excitement abuzz, we have Jailmaker, the latest solution to this containerisation problem being thrown around, IX-Systems have officially added the service built in to TrueNAS and say that it isn't currently supported, but they promise upgrades won't break it. Yay people say, now I can run docker that way!
But it's also fraught with problems which may not be well documented so here are a few:

• It's a container in a container requiring double entry of all shares
• It's a container in a container requiring triple entry of all permissions
• it's a container in a container requiring triple entry of all accounts
• If we create separate jails for all our apps (say we have 20 apps), we now must run 20 simultaneous instances of docker engine, WTF.
• If we don't create separate jails for all our apps, we're basically the same as running a VM with some different trade offs (Ram, Disk, Portability)

So, if this jailmaker script was suddenly unofficially endorsed, I assume to give some relief to people wanting a resolution to this ludicrous containerisation problem, possibly to entice some TrueNAS CORE people to come over to the dark side of linux as well because it 'sounds like' BSD jails, all so that we can run docker, why don't IX Systems just listen to the silent screams everywhere and just add docker engine?

Don't get me wrong, Jailmaker is great for some smaller use cases, but it isn't docker. And it has no docker hub. Sure it has the potential to replace TrueCharts, but it will end up probably exactly the same because IX Systems aren't in the business of ensuring we have a massive App Library.

So here we are again with no REAL solution, just a WORKAROUND and now we just get to juggle one more option with different tradeoffs and we just get to choose which one we will tolerate the most for a particular scenario.

I mean what happened to TrueNAS in the past that made IX-Systems so resistant to docker? Do they just have a bee in their bonnet about it? Or was there some 'event' that happened that made IX-Systems resist docker so much? Perhaps it's a strategy they have we don't know about? Or perhaps it WAS the strategy with clustering etc which is now no longer applicable? Does anyone know?

and has been a pain point. They then added custom apps which is where you find most app which are actually docker apps. But these runs on an engine which is not docker but encapsulated by Kubernetes (running on docker) causing more problems ie can't set hostname or use ports under 9000.

. The foundation is built on top of Rancher 

[matsonjr 6]

Well, that was disappointing.  

 

I just time shifted a football game.  I joined probably half way through to catch up.  The fast forwarding was very smooth and no lags or anything like last week.  However, it was separated into three recordings yet again.  This time, I'm attaching the log in case there is a clue there as to what is going on.  

embyserver.txt

[Carlo 4560]

Logs show a couple errors where Emby was denied access to the disk
Emby probably started fresh because something went wrong ad it wanted a fresh try capturing TV data.

System.UnauthorizedAccessException: System.UnauthorizedAccessException: Access to the path '/mnt/media/PVR-TV/College Football (2010)/College Football 2024_10_27_02_30_00 - Washington State at San Diego State.ts' is denied.

 

System.UnauthorizedAccessException: System.UnauthorizedAccessException: Access to the path '/mnt/media/PVR-TV/College Football (2010)/College Football 2024_10_27_02_30_00 - Washington State at San Diego State - 1.ts' is denied.

 

 

[Luke 42077]

@matsonjrIs it possible the drive holding the recordings is not allowing simultaneous reads and writes?

The server starts writing to the recording file. Then you try to play that recording in progress, so it needs to read the file while it's still being written to. As soon as the server tries to do this, this causes the recording process to fail.