Emby ate all my space! Plus, some weird files that need explaining...

[Bob Schnatterly 0]

I'm running Emby on the WD PR2100 NAS.

 

This morning I had 140GB of space on my NAS. But around 10:30pm or so, I found out I had NO space left on my NAS. I scrambled to free up some space, and then I dug around the "webs" for a way to force Emby to clear up some space. Someone said I could delete all the files in emby/metadata/library, so I did, but that only freed up about 8GB. I dug around some more and found (in the transcoding-temp folder) this rather huge file (see attachment). If I'm reading that right, it's 170GB (for an on-going recording?)!  Emby was trying to record a Lost in Space episode at the time, so I left it alone, but shortly after that, the recording failed, and these 2 files were left behind. I deleted them, but it failed to free up any space, so I'm scratching my head over that.

I dug around some more and found these odd MKV files in the emby-server folder (see other attachment). Obviously, these are Windows pathnames on a Linux server. (I've never used Emby as a client or a server (or on the web) on a Windows machine.)  However, those are shows/movies I've recorded or ripped at various times, so I'm stumped. I suspect they're related to chapter thumbnails?

I turned off chapter thumbnail generation, and at this point I'm going to leave it off.

Is there a way to find out why Emby ate all my data so quickly, what these odd files are, and how to minimize disk space usage in Emby?

Thanks for creating Emby. I'm glad you're so responsive to problems/questions we have. I had to put up with a lot of bizarre behavior by Plex, and this was exacerbated by their poor tech support response time.

[Bob Schnatterly 0]

A follow-up...  When I went to bed, I had about 55+ GB of space, and I was off-loading about 80GB to another hard drive. I woke up to about 32GB of space (and it was shrinking while I was watching, even though I wasn't currently recording anything).

I just shut down Emby and my free space immediately jumped from 27GB to 299GB! Pardon my French, but WTF?

What exactly is Emby doing to suck up all that space JUST RUNNING?

[Deathsquirrel 745]

Do you have a convert job setup by chance?

[Bob Schnatterly 0]

No. I turned off conversions fairly early after I set up Emby 'cause I wasn't sure how it worked (or if it would even work - I didn't know about the in-place conversion feature until later). Still relying on MCE Buddy. I turned off chapter preview fairly early yesterday (or later in the evening on Friday - can't remember)

[sluggo45 47]

I've been running Emby server for years and do all kinds of things with it - chapter generation, Roku thumbnails, DVR conversion, you name it - and I have never seen it eat space out of control like that. Something is seriously weird with your setup. 

 

Recommend you read and follow the instructions in How to Report a Problem thread because it's going to take a lot more detail to figure this one out.

[Bob Schnatterly 0]

It seems to be behaving itself now (while recording 2 shows). Space is diminishing as expected. I turned on debugging. If it happens again, I'll have something to look at. (Glad to hear you've had a good run of it...)

[Bob Schnatterly 0]

Emby played nice yesterday. I'm thinking the problem I had was from a plug-in (or me continually messing with metadata settings or something), but debug logging was turned off, so I'm only shooting in the dark.

This probably isn't the place for this, but I thought I would give my overall impression of Emby:

 

Pros:

1) Expansive media library options (eBooks? Cool)

2) Responsive

3) Ability to selectively exclude a given episode in a season

4) Pretty nice menu layout

5) Nice device support

6) Nice media playback support - it's played back everything I've thrown at it (although I will have to convert a DVD file I have to MP4 at some point)

7) Stable

 

Cons:

1) It's too picky about filenames. I had to rename "Season 01" to "Season 1" (etc) to keep Emby from creating new folders all over the place. (This is disappointing because it makes it harder to sort folder names using a file manager!)  I also had to change the name of my Lost in Space folder to be EXACTLY like the official metadata source (I had to lowercase the word "in") because Emby didn't see the folder at all and created a brand new one, season and all. (Don't ask me what metadata sources I'm using - I turned on most of them )  I would think Emby would internally uppercase everything before matching.

2) The plugins I've played with are poorly documented, even the ones built-in, and have few (or no) settings you can mess with

3) The manager home page could be better organized. Alerts should come before Active Devices, for example

 

There are many things I haven't looked at yet, though, like subtitles and captioning, the built-in transcoding (simply because I can't use it until it lets me filter out MKV files), the recording of all NFL games (or all games for one team), etc.

 

The Pros definitely outweigh the Cons for me. A very impressive package y'all have put together

 

I'm not going to turn on chapter image generation again until I put in much larger hard drives in my NAS. I'm not sure if that was what caused my problem, but I don't want to be overrun again!

[ebr 16184]

1) It's too picky about filenames. I had to rename "Season 01" to "Season 1" (etc) to keep Emby from creating new folders all over the place. (This is disappointing because it makes it harder to sort folder names using a file manager!)  I also had to change the name of my Lost in Space folder to be EXACTLY like the official metadata source (I had to lowercase the word "in") because Emby didn't see the folder at all and created a brand new one, season and all. (Don't ask me what metadata sources I'm using - I turned on most of them )  I would think Emby would internally uppercase everything before matching.

 

Neither one of those names should have been a problem.  Can we look at an example of these?

[Bob Schnatterly 0]

I'm not sure what you're asking for. I had Emby record an episode for a show I had already taped before using Plex (can't remember which one, though). The folder name was "Season 02" but Emby created a new folder called "Season 2" and put the new episode there instead.

And I've already explained what happened with Lost in Space (I had "Lost In Space (1965)", but Emby created a new folder called "Lost in Space (1965)").

I can tell you what all my metadata settings are if you need that. I probably have a log or two by now too (I turned on debugging yesterday) - let me know and I can send you one of those

[ebr 16184]

Okay, a recording is a different situation.  Since Emby didn't record the first part of that series it had no knowledge of it.

 

In regular media, that naming would have been no problem.

[Bob Schnatterly 0]

Movies

=======

Movie metadata downloaders:

TheMovieDB

The Open Movie Database

 

TV Shows:

===========

Series metadata downloaders:

TheTVDB

The Open Movie Database

 

Season metadata downloaders:

<none>

 

Episode metadata downloaders:

TheTVDB

The Open Movie Database

 

Am I having these problems because I don't have anything picked for Season metadata downloaders (and would this explain the show title problem too)?

Edited March 4, 2019 by Bob Schnatterly

[Bob Schnatterly 0]

Unfortunately, I have had a very similar experience tonight. I logged into my NAS and found that only 40GB of space was left, although I had about 240GB as of this morning. (Throughout the day today I set up Emby on my parents' 2 Roku boxes, my mother's and my brother's phones, etc, and we all tested it with several live TV and video samples. Also throughout the day, Emby has been steadily recording shows and movies, and MCE Buddy has been faithfully transcoding these to much smaller MKV files.)

 

I shutdown the Emby server from within the Emby web manager. No change in disk space. I shut down the Emby app on the NAS web manager. No change in disk space. I restarted the Emby app (which I assume automatically restarted the Emby server).  This time the space rebounded back up to 215GB.

 

Now that 2 of my family members are going to be using Emby in the future, I need to know what is causing this, and if it's normal. Do I need to free up a lot of space on my NAS to accommodate this wild variation of free space?

Edited March 17, 2019 by Bob Schnatterly

[Luke 42079]

Did you look at what folders were taking up the space?

[Bob Schnatterly 0]

Not this time. I guess I'll have to wait until the next time it happens. Has anyone else reported this? Do you have any idea what might be causing this?

 

[i have not turned on chapter generation or enabled any other metadata generation since the last time this happened. I did install the IPTV and TuneIn plugins, but I haven't been using them. Maybe a plugin is doing this?]

Edited March 17, 2019 by Bob Schnatterly

[Bob Schnatterly 0]

The problem is definitely in the transcoding-temp folder. I lost significant space again overnight (but not as dramatically), so I ran "du -csh *.ts" and it came out with this (see attached). My mother left ABC streaming overnight (I guess she fell asleep ), but  she wasn't even set up for Emby before yesterday, so that probably didn't have anything to do with this problem. Emby wasn't recording anything a few minutes ago when I checked (but it had been recording shows all night). These files are either not supposed to be created at all, or aren't being purged. There are 11,045 of these files. I have debug logging turned on now, so if you want to see a specific log or logs, let me know...

Edited March 17, 2019 by Bob Schnatterly

[Bob Schnatterly 0]

Ok, I don't know why this attachment isn't working. Let's try that again...

Edited March 17, 2019 by Bob Schnatterly

[Bob Schnatterly 0]

I think I see one potential problem (but I'm still a Linux newbie, so who knows): these files are marked -rw-r--r-- (but are owned by root). Wouldn't the root user need execute permission to delete these files? According to one person on the "webz", it would. (The original question posted was "What are the minimal permissions needed to delete a file in Linux".)

 

The permissions on the folder itself are drwxr-xr-x

Edited March 17, 2019 by Bob Schnatterly

[Bob Schnatterly 0]

Another new development: just moments ago, I checked again and Emby must have finally deleted all those files because I regained all my space! Nice, but can we get Emby to clear them much sooner so it doesn't suck down all this space needlessly?

 

(Confirmed: the folder is empty now)

Edited March 17, 2019 by Bob Schnatterly

[Luke 42079]

These files are normal when live streams are active, but when stopping they will be cleaned up.

[Bob Schnatterly 0]

This is normal?  Whatever you're doing, it's taking up tens of gigabytes of space and creating thousands of files!  The one file alone is 59 gigabytes in size. This means I'm gonna have to make sure to set aside at least 100GB (if not 150GB) of space on my NAS for this. That's space I can't use to record shows or movies. This is very disappointing

[Bob Schnatterly 0]

Please keep in mind that in my original post, I identified a file that was 172GB in size. This is NOT normal. I don't mean to be rude, but you need to find a different way to buffer content. I never had this problem with Plex, so there's apparently a better way of doing this...

[Bob Schnatterly 0]

I think I may have stumbled on a glitch in your Roku app that's causing these huge files: My mother was watching a pre-recorded show and hit the Home button to end it and use a different Roku app.  Since I was planning on making a change to my NAS box (you mean it took me 2 years to figure out I could use that 2nd Ethernet port for something??!!), I wanted her to stop using Emby so I could power down the app, etc.  But even though she said she was now using that other Roku app, my server dashboard showed that the earlier pre-recorded program was *still* being streamed. In fact, at least 2 minutes had passed and it was still streaming even though she wasn't watching anymore.  So I had to manually kill it on my end!

I'm thinking there's a glitch in the Roku app that isn't registering the exiting of the Emby app.  Hope that helps you...

[ebr 16184]

I'm thinking there's a glitch in the Roku app that isn't registering the exiting of the Emby app.  Hope that helps you...

 

The problem here is the Roku system does not tell us the app is being shut down when you press the home button.  So the app cannot stop the stream.

 

However, the server should notice that the app has stopped reporting progress and stop it.  Can you try doing what she did again and waiting about 5 minutes to see if the server stops the stream?

 

Thanks.

[Bob Schnatterly 0]

Ok, I'll try to do that a little later today. Thanks

[Bob Schnatterly 0]

Ok, I had her re-do that scenario. The stream continued on for about 8 minutes after she hit the Home button. Is it possible that at other times, Emby could have run a lot longer than that?  I'm just wondering what caused Emby to acrue a 172GB temporary file...