FTP on Demand

You know it: If you want to load down a big file from a distant server (overseas perhaps), the maximum transfer rate can not be reached. If you are using ISDN it is extreme. But you have no ISDN Port to spend the telecom union your money, and you use the best operating system in the world :-) People who can access the Internet via a university and send some Unix commands to it, can do something.

Theory

But how? The trick is to load the file not directly, but let the provider do it for you. After some time you can load the desired file from the provider without wasting time - the connection to your provider most times is so fast that you reach the maximum transfer rate. Unix knows the command nohup which allows a command execution without being logged in - this is the trick. You can log in at your provider and type in:

  provider:# nohup ftp -i ftp.leo.org &
  provider:# exit
  connection closed by foreign host.
  root:# _

This takes only a few seconds, only one tariff unit is lost for a good purpose.

But you have to do some prework, because ftp is an interactive program, which needs a lot of information for its work (e.g. what to load). You can tell ftp how to be less interactive by giving the option "-i". But you need to tell ftp the information what to do, and this happens if you create a file .netrc at your provider. (The option "-n" tells ftp to ignore this file.)

If you want, for example, load down the Linux kernel 2.0.7 from the LEO.ORG server (more than 5 Megabytes!) the file .netrc could look like:

provider:~/.netrc
  machine ftp.leo.org
  login anonymous
  password <my@email.address>
  macdef init
    lcd /usr/src
    cd /pub/comp/os/linux/Linus/v2.0
    bin
    get linux-2.0.7.tar.gz
    close
    quit

So cou could create an entry for each server you want to connect. This file must be stored in your directory at your provider before you call the nohup ftp command. One hour later you can load down directly your kernel. Great, isn't it?

Automotion

Ok now. Now we want to do it automatically. Of course we need the cron again, which will do the data transfer in the cheap night hours.

... But this is not finished yet. Please wait...

I had no time to do more research - but some people wrote me hints.

Uwe Bonnes <bon@elektron.ikp.physik.th-darmstadt.de> wrote:

Hello,

you should point at "ncftp" for ftp-on-demand. I open the program using
"nohup ncftp", open the server I need, choose the file using the "tab" 
expansion and start the download. A ctrl-Z and "bg" move the download into
background and myself in the commandline to stop the connect.

Gunnar Larisch <Gunnar.Larisch@lpr.e-technik.tu-muenchen.de> wrote:

Some nice scripts:

getme <Server> <Dir> <File>
cronftp must be started using cron. You can fetch your file the next day.

fetfrom also may be useful.

-----------getme--------------
#!/usr/local/bin/bash

# Parameters: Server Path File(s)

FTPFILE="/tmp/.$UID.ftpjobs"
echo Server : $1
echo Dir    : $2
echo "File   : $3"
echo Jobfile: $FTPFILE

touch $FTPFILE

echo lcd ~/tmp/ftpjobs >>$FTPFILE
echo open $1 >>$FTPFILE
echo user anonymous t8231ah@sunmail.lrz-muenchen.de >>$FTPFILE
echo bin >>$FTPFILE
echo "cd $2" >>$FTPFILE
echo "mget $3" >>$FTPFILE
echo close >>$FTPFILE
-------------------------------

---------getfrom---------------
echo Hole $2 von $1...
echo ftp -i $1 >>~/tmp/ftpjob
echo bin >>~/TMP/ftpjob
echo cd $2 >>~/tmp/ftpjob
-------------------------------

---------cronftp--------------
#!/usr/local/bin/bash
FTPFILE="/tmp/.$UID.ftpjobs"
NEWFTPFILE="/tmp/.bgjob_$UID_$RANDOM"
mkdir ~/tmp/ftpjobs
echo Using $FTPFILE... as $NEWFTPFILE
mv $FTPFILE $NEWFTPFILE
nohup ftp -in <$NEWFTPFILE &
-------------------------------------

Ciao Gunnar

Siggi Thoma <sthoma@twilight.rhein-neckar.de> schrieb:

Did you hear about FTPMAIL? Perhaps this matches the purpose and is a bit
simpler.

Just write an e-mail to ftpmail@ftp.uni-stuttgart.de 
subject HELP
body:
help
quit


---------------
This server at the Uni allows to fetch any file from all over the world 
to Stuttgart. It mails this file in uuencoded format to you. So only the
providers transfer data rate is the limiting factor.

You find detailled descriptions in the net. Search for FTPMAIL at Yahoo or 
Altavista.

Tschau
Siggi