Working with ftp automatically

[darkassassin07 434]

This isnt strictly emby related, but is a part of my media server setup:

 

I am trying to find a way to have my windows 7 machine connect to an ftp server every 10-30min, download all folders/files it contains, then delete them from the server.

 

My plan is to have my emby server machine sleep as much as possible to save power. The problem is a security camera that connects via ftp and uploads snapshots on motion detection events. This camera connects sporadically throughout the day and keeps the server awake unnecessarily. So I want to host an ftp server via a raspberry pi that is always online and have the camera dump images there. Then whenever the server is awake (both on wakeup and every 30min) have it connect to the pi and offload the new images into an emby library, cleaning out the pi's memory as it does this.

 

 

Could anyone point me in the right direction to accomplish this? I assume a batch file with a task scheduler task could do this, im just not sure where to start with that bat. I have never done anything network related from command line, and im not even very familiar with local command line use.

 

The pi has not arrived yet, so im just trying to get a little foresight

[PenkethBoy 2063]

not 100% sure i understand

 

is the win7 machine the emby server?

 

ftp is used only on the RPi for the camera to use as a target for downloading images?

 

So i think you mean you want something to run on your win7 machine to pull the files from the RPi when it wakes up?

[darkassassin07 434]

Yes, im looking for a solution to have a windows 7 machine automatically connect to a standard ftp server download everything available, then delete everything it downloaded off the ftp server. Effectively moving the files from the pi to windows 7 emptying it as it does so.

 

Preferably a bat file or similar that i can just set to run via task scheduler.

 

 

Im open to a solution that doesn't use ftp as well, I just figured that would be easiest as the pi will be hosting an ftp server regardless.

 

As long as I can move the files from the pi (which is on the same lan) to win7 and clear out the pi.

 

This has to be done from the windows 7 machine though as it needs to be triggered when it wakes up

Edited August 31, 2017 by darkassassin07

[Jdiesel 1114]

Why not just move the files rather than copy then delete. I'm not an expert with Windows but you should be able to do this easily with a script. 

 

Are you able to map a ftp connection to a drive letter in Windows? If so it will make your script even simpler but I do believe CMD supports ftp if you can't.

[darkassassin07 434]

You can map an ftp server as a network location, im just not sure if/how that makes it accessible from command line.

 

I don't know where to start with that script.

[PenkethBoy 2063]

@@darkassassin07 - i thought all assassins had Ninja skills 

 

if you have the Pi data store mapped to the win 7 machine its no different than a local drive as far as the script is concerned

 

As for the script

 

in a command prompt window type "move /?" will give you the syntax of the command

 

you will probably need the /Y command if the new location might have files that could be overwritten otherwise the move will stall

 

e.g. (not tested)

 

move /Y f:\filedump\*.* g:\newfiledump

 

try it out manually before saving it in a bat file

 

as for running it every x minutes/hours - setup a scheduled task and point it at the bat file

 

if you get stuck post back the commands you tried etc

[dcook 266]

Here is what I use to fetch files:

@echo off

"C:\Program Files (x86)\WinSCP\WinSCP.com" ^
  /log="C:\Users\dcook\Documents\CloudDownload.log" /ini=nul ^
  /command ^
    "open scp://dcook@x.x.x.x/ -hostkey=""ecdsa-sha2-nistp256 256 b4:be:a8:c1:c4:2e:04:75:e6:ae:36:69:74:50:6b:bb"" -privatekey=""C:\Users\dcook\Documents\dcook-Private.ppk"" -rawsettings AuthKI=0" ^
    "get -delete ""/emby/cloud/download/*"" ""X:\NEW\""\" ^
    "exit"

set WINSCP_RESULT=%ERRORLEVEL%
if %WINSCP_RESULT% equ 0 (
  echo Success
) else (
  echo Error
)

exit /b %WINSCP_RESULT%

It will download the files, and after successful download remove them from the server, and also write to a log file.

 

I assume you will run Linux on your PI so this would be able to connect to it via SCP instead of FTP

 

FTP is obsolete, must easier to use SCP and more secure

Edited September 1, 2017 by dcook

[PenkethBoy 2063]

depends - it sounds to me the RPi will be inside his network if so WinSCp is not necessary

[dcook 266]

Even inside your own network, its easier to script using SCP then FTP and SCP is more reliable

 

 

depends - it sounds to me the RPi will be inside his network if so WinSCp is not necessary

[Sludge Vohaul 22]

@@dcook

 

Why is ssh more reliable than ftp? 

[Sludge Vohaul 22]

@@darkassassin07

 

You should be aware that you might end up with corrupted files in case the camera has not finished uploading a file yet while you start to download it already.

 

I'd also opt for copying the files with ssh as dcook already suggested. If you have cygwin installed, you could go with

ssh your_pi_box "cd /path/to/the/camera/files && tar cf - . && rm -f *.jpg_what_ever" | tar xf -

This connects to your_pi_box via ssh, changes directory to the camera files directory, tars the contents of the directory to stdout, <copies the stream encrypted to the box where you invoked the ssh connection>, deletes all files with extension 'jpg_what_ever' and extracts the stream from stdin to disk.

 

In case encryption is not relevant, you could speed up the whole thing a bit by using a weak encryption via the -c parameter, e.g.

ssh -c arcfour your_pi_box .....

Sorry, I cannot provide a Windows script.

 

HTH,

sv

Edited September 1, 2017 by Sludge Vohaul

[darkassassin07 434]

Thanks for the suggestions, a couple of things though:

 

Im required to host ftp for the camera to upload to anyway and I'm not particularly worried about encryption.

The camera creates sub directories:

Camera-path/<date>/images/image.jpg

I need to keep this structure as well. If i was to use the 'move' command with camera-path/* would it keep the folder structure?

 

Don't have the ability to test atm

Edited September 1, 2017 by darkassassin07

[Sludge Vohaul 22]

@@darkassassin07

 

If you mean the Windows command line "move command" then you're wrong. You cannot use it at all, as "move" is for  moving files which are accessible via the Windows Explorer (local file systems or network shares).

If you use the ssh + tar way as described in #11, then the existing directory structure will be preserved. You'd still have to create a script which you'd call from Windows' "Scheduled Tasks"...

 

But as I already said you might end up with incomplete files on your windows box.

Maybe you should have a look at rsync which is a rock solid solution for incremental backups. Google says that there are also implementations for Windows (e.g. DeltaCopy). You could set up a rsync scheduled task on your Windows machine, and configure rsync to delete the sync'ed files on the pi-box. Google it, but I think it is possible to configure rsync to skip open files (i.e. files which your camera is currently transferring), so that those files would be transferred during the next invocation of rsync.

You'd have to set up an rsync server on the pi-box and configure the server and your client. This is probably better than the ssh solution.

 

sv

[PenkethBoy 2063]

wow this is getting way too complicated

 

look at SyncBackPro or more probably its free version SyncBackFree(?) it can connect via ftp and you have loads of options on when to run and remove the copied from the host

 

or any ftp client will have all the options you need also

 

no need for ssh/rsync/winscp etc

 

KISS

[dcook 266]

My script is very simple its downloading * so doesnt matter if there are directories.

 

And you dont have to worry about coruption either as it wont delete them until after they are confirmed downloaded.

[darkassassin07 434]

Im not too worried about corrupt files really, it would be verrry rare that the camera would be uploading an image at the same moment the transfer happens, and even if it does, its not a critical situation at all.

 

I was asking about the basic move command as we were talking about mounting the ftp server as a network location, that would make it accessible wouldn't it?

 

"if you have the Pi data store mapped to the win 7 machine its no different than a local drive as far as the script is concerned" - penkethboy

[dcrdev 251]

If it's on your network - isn't there a far more obvious answer here?

 

Don't move the files at all!

 

Create an ftp share on your windows machine, mount that on your raspberry pi natively via fstab or systemd mount and have whatever process is running to create the images write to that mounted folder. That way you don't have to concern yourself with incomplete files and the files never actually sit on the pi.

 

^ Also probably don't use ftp  - use smb as it negates the need to install anything.

Edited September 1, 2017 by dcrdev

[darkassassin07 434]

The whole point was to have the pi recieve files from the camera and have the main server retrieve them when it happens to be awake. That way the camera isn't keeping the main server awake all the time. That is what the current setup is doing. (camera -ftp-> main server)

 

I have no choice but to use ftp between the pi and camera, the camera itself has no other option (and its not getting changed)

[dcook 266]

Simplest way is to have your camera send files to your PI using FTP, and then have your server pick up the files when it wakes, you can have it run a batch script which would grab all the files from the PI

 

The script I posted above will do exactly that.

[darkassassin07 434]

I think I am going to go with @@dcook s script.

 

I just have a couple questions:

Can I ditch -hostkey and -privatekey without issue? I dont really need or want to generate/use my own keys. Im not even all that fussed about encryption at all for this.

Also what's the purpose of -rawsettings authKI=0?

Edited September 2, 2017 by darkassassin07

[darkassassin07 434]

As a bit of a side note, how could I go about having the raspberry pi monitor its own free space and if if starts to fill up its internal storage ping the main server (which would trigger WoL triggering the script to empty it)?

 

 

Though that's not a big deal, the main server is awake often enough, and the camera only uploads ~100kb images

[dcook 266]

This will help you:

 

https://winscp.net/eng/docs/guide_automation

 

You can just run the tool and it will generate the script code for you

[Sludge Vohaul 22]

@@darkassassin07

 

If you want "passwordless" (i.e. no password prompt) encrypted copying from the

pi-box then you need the keys. Otherwise you'll be prompted for the password.

 

The WinSCP website has a FAQ which answers all of your questions. Please read yourself.

 

https://winscp.net/eng/docs/scriptcommand_open

https://winscp.net/eng/docs/ssh_keys

https://winscp.net/eng/docs/rawsettings

 

You won't find a solution matching exactly your needs without investing some

time into reading some docs.

[Sludge Vohaul 22]

@@darkassassin07

 

Post #21:

You'd set up a cron job on the pi-box let's say every 15 minutes, invoke something (df) which tells you the free space, in case it's lower than your defined watermark you'd ping the other server, which would wake up and eventually execute a scheduled task which would download and delete the data from pi-box.

 

You for sure could use some monitoring tools on the pi-box which would execute a preconfigured script pinging the other box.

[darkassassin07 434]

Thanks for all the help guys, my RPI finally made it here so i began messing around with it.

after much trial and error both setting up the RPI's plain ftp server and automatically retrieving the images on it via windows I finally settled on this:

 

Retrieve Remote Files.txt

open ftp://<user>:<pass>@<IP>:<PORT>/files/
option transfer binary
get * c:\ftp\Camera\
rm *
close
exit

then using a task scheduler task to trigger:

"C:\Program Files (x86)\WinSCP\WinSCP.com" /init=nul /script="C:\ftp\Retrieve Remote Files.txt"

on system event 'Microsoft-Windows-Power-Troubleshooter' ID: 1  (fires on system wake)

 

 

 

 

 

 

 

 

 

I do however have one more question: 

 

Can I trigger emby to rescan a specific library from cmd line? or even trigger a whole rescan?

Id like to add that to the end of this script so as soon as the images are transferred emby becomes aware of them.

 

I have file system monitoring enabled, its not fast enough to add them to emby before the user makes it into that library waking the computer from emby remotely.

 

user starts connection, that wakes server, script triggers retrieving files, user makes it to home screen and selects camera library (by now file transfer has finished), new images are not yet available in emby (but are on filesystem).
though if you stick around for 5-20secs then re-enter the camera library the images show up. Id like to remove that delay and have the new images available by the time the user makes it to his home screen/camera library.

 

note: there is quite a bit of time between a user starting the connection to the sleeping server and being at the home screen. a good 30-60sec at least, usually requiring the user to refresh/reconnect once the server is actually awake. where as the script transferring 4-5 days worth of images only takes 3-8sec. there is plenty of time to refresh embys camera library (which only has 14days of images (~200 x 200kb) ever.)

Edited September 14, 2017 by darkassassin07