bolug bonner linux user group
news about wissen files archive hilfe suchen  

 

archive :: SchAN-User

druckfassung

SchAN-User

Re: schan-user: Automatische ftp-upload

To: schan-user@xxxxxxxxxxxxxxxxx
Subject: Re: schan-user: Automatische ftp-upload
From: Hans-Dietrich Kirmse <hd.kirmse@xxxxxxxxx>
Date: Tue, 18 Apr 2000 21:06:01 +0200
entschuldigt bitte die lange mail - aber vielleicht kommt wirklich 
jemand mit dem Quelltext zurecht.

------------------------------------------

Lieber Gisbert, 

ich habe mal ein Perl-Programm kopiert, welches vor unserer Arktur-Zeit
genau das Problem erledigte. Es hatte ein ehemaliger Schüler für mich 
programmiert. Es lief auf einer SuSE4.4 (muß natürlich angepaßt werden)

Aber erst noch ein paar Bemerkungen zur E-Mail, weil meine Forderungen 
damals andere waren.


Gisbert Friege schrieb:
> 
> Hallo Hubertus,
> 
> > es wäre natürlich noch sehr hilfreich, wenn man ein automatisches
> > Download der gesamten Schulhomepage  (mit Unterverzeichnisse)
> > vom FTP-Server auf den V3.0 /home/www/hompagae machen
> > könnte.

das wird doch sehr selten gebraucht - doch eignetlich nur beim 
erstenmal. Im Normalfall sind es doch nur wenige Seiten die auf den
Server gespielt werden.

Problem ist aber dann ein anderes: oft wird ein Bild oder eine verlinkte
Seite, bei der es nur eine kleine Änderung gab, vergessen.

Meine Forderungen damals waren:  alle GEÄNDERTEN Dateien
(einschließlich neuer Verzeichnisse) mußten gedownloaded werden.

Danach mußte noch abgecheckt werden, ob auf der HP noch Dateien 
gelöscht werden müssen. 

Das Prinzip funktioniert natürlich auch, wenn noch nichts da ist.
Also, wenn alles gedownloaded werden muß.


> Je nach ftp-Server, auf dem die Homepage liegt, funktioniert glaube ich
> auch die Verwendung von tar : das wäre eigentlich am schnellsten und
> praktischsten.

zum "enttaren" braucht man einen telnet-zugriff. Ist auf den Bildungs-
servern aber doch meistens gegeben.

> Vielleicht kann uns jemand da weiterhelfen ?

Weiß ich nicht genau. ich kopiere jedenfalls mal das Prog. Es lief ca.
8 Monate. Dann hatte wir versucht, unsere Server "besser" einzurichten.
Das ging schief -> Arktur. Hatte mit dem Programm NICHTS zu tun.  

Der ehemalige Schüler ist für mich derzeit schlecht erreichbar. 
Sollte er gebraucht werden, bekomme ich seine e-Mail-Adresse aber raus.
(ist noch an der TU Ilmenau)

> 
> ?! Eventuell sollten wir den Homepage-Upload als cron-Job im
> sysadm-Menue anbieten !?

Bei uns lief dieses Programm nachts als Cron-Job.

vielleicht kann das ja jemand als Vorlage nutzen oder gar auf Arktur 
anpassen. ich wäre sehr daran interessiert ;-)



mit freundlichen Grüßen
Hans-Dietrich





-------------- Kopie des Programms -------------------------------

#!/usr/bin/perl
#========================================================================================
# name        : hp-sync.pl
# author      : Marco Lehmann
# version     : 1.08
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : perl hp-sync.pl
#----------------------------------------------------------------------------------------
# description : script to synchronize homepage on 'Bildungsserver' with local 
homepage
#----------------------------------------------------------------------------------------
# weak points : unknown
#----------------------------------------------------------------------------------------
# open tasks  : unknown
#----------------------------------------------------------------------------------------
# created     : 14.01.1999
#========================================================================================


# required modules and functions
use File::Find qw(find);


#========================================================================================
# global variables and constants
#========================================================================================


# name of homepage directory (relative to start directory)
my ( $homepage_dir ) = '/usr/local/httpd/htdocs/homepage/current/';

# directory for miscellaneous files
my ( $lib_dir ) = '/usr/local/lib/hp-sync/';

# name of distribution tar file (relative to start directory)
my ( $homepage_filename ) = 'homepage.tar.gz';

# name of server-files-database (text file, relative to start directory)
my ( $files_server_filename ) = 'files.server';

# hash to hold server files
my ( %files_server ) = ();

# name of transfer-files-database (text file, relative to start directory)
my ( $files_transfer_filename ) = 'files.transfer';

# name of delete-files-database (text file, relative to start directory)
my ( $files_delete_filename ) = 'files.delete';

# hash to hold local files
my ( %files_local ) = ();

# name of log file
my ( $log_filename ) = 'hp-sync.log';

# content of new log file entry (if this variable is set -> an error occured)
my ( $error_msg ) = '';

# an array with possible servers to connect to
my ( @servers ) = (
                   {
                    'name' => 'serv.th.schule.de',
                    'user' => 'lehr11',
                    'pass' => 'das_habe_ich_gekillt'
                   }
                  );

# online start and end time
my ( $online_starttime, $online_endtime ) = (0, 0);

# save size of transferred tar file
my ( $homepage_tar_file_size ) = 0;


#========================================================================================
# main
#========================================================================================

# check if program runs in a web server
if (exists $ENV{'REQUEST_METHOD'}) {

    # convert log content file into html statements and print it to stdout
    log2html();

} else {

    # change to homepage directory
    chdir $homepage_dir or $error_msg = 'Directory ('.$homepage_dir.') doesn\'t 
exist.';

    # read server files from local text file
    read_server_files();

    # read local directory
    read_local_files();

    # check for differences between server files and local files
    check_files();

    # create distribution file
    create_distribution();

    # transfer files
    transfer_files();

    # update all important files
    update_files();
}


#========================================================================================
# name        : log2html
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : log2html()
#
#----------------------------------------------------------------------------------------
# description : converts log file entries into html statements
#
#----------------------------------------------------------------------------------------
# created     : 18.01.1999
#========================================================================================

sub log2html {

    # generate HTML-Header
    print 'Content-type: text/html'."\n\n".
          '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">'."\n".
          '<HTML>'."\n".
          ' <HEAD>'."\n".
          '  <META NAME="author" CONTENT="Marco Lehmann">'."\n".
          '  <TITLE>Homepage upload statistic</TITLE>'."\n".
          ' </HEAD>'."\n".
          ' <BODY BGCOLOR=#FFFFFF TEXT=#000000 LINK=#00A8D0 ALINK=#FF0000 
VLINK=#00FFFF>'."\n".
          '  <DIV ALIGN=CENTER>'."\n".
          '   <H1>Homepage upload statistic</H1>'."\n".
          '   <TABLE WIDTH=95% BORDER=0 CELLPADDING=0 CELLSPACING=1>'."\n".
          '    <TR BGCOLOR=#0078A0>'."\n".
          '     <TD><B>Date</B></TD>'."\n".
          '     <TD><B># of server-files</B></TD>'."\n".
          '     <TD><B># of transferred files</B></TD>'."\n".
          '     <TD><B># of deleted files</B></TD>'."\n".
          '     <TD><B>Bytes transferred</B></TD>'."\n".
          '     <TD><B>Time online</B></TD>'."\n".
          '    </TR>'."\n";

    # check if log file exists and open it
    if ( !open LOG, '<'.$lib_dir.$log_filename ) {

        # print an error message (in html statements)
        print '    <TR BGCOLOR=#FFC0C0>'."\n".
              '     <TD COLSPAN=6>Can\'t open log file 
('.$lib_dir.$log_filename.").</TD>\n".
              '    <TR>'."\n";
    } else {

        # array to store all log file entries
        my ( @log ) = ();

        # read all lines
        while (<LOG>) {

            # clear line from trailing newline
            chomp;

            # constructing array of array (cool)
            push @log, [ split /\0/o ];
        }

        # convert log entries into HTML statements (reverse sorted by date)
        my ( $entry ) = 0;
        foreach $entry ( sort { $b->[0] <=> $a->[0] } @log ) {

            # convert file datetime in a readable form
            my ( $sec, $min, $hour, $mday, $mon, $year) = 
(localtime($entry->[0]))[0..5];
            $sec = '0'.$sec if ($sec <10);
            $min = '0'.$min if ($min <10);
            $hour = '0'.$hour if ($hour <10);
            $mday = '0'.$mday if ($mday <10);
            $mon++;
            $mon = '0'.$mon if ($mon <10);
            $year += 1900;

            # every log entry has its time value -> print it
            print '    <TR BGCOLOR=#00A8D0>'."\n".
                  '     
<TD>'.$mday.'.'.$mon.'.'.$year.'&nbsp;&nbsp;'.$hour.':'.$min.':'.$sec.'</TD>'."\n";

            # check for error messages
            if (defined $entry->[7]) {
                print '     <TD BGCOLOR=#FFC0C0 
COLSPAN=5>'.$entry->[7].'</TD>'."\n".
                      '    </TR>'."\n";

                # start next iteration (do not print rest of current log entry)
                next;
            }

            # calculate online time
            if ($entry->[2] == $entry->[1]) {
                $entry->[1] = '-';
                $entry->[2] = '';
            } else {
                # use integer arithmetik for division and modulo (only in this 
scope)
                use integer;

                # assign those values by themselves without any temporary 
variable
                ($entry->[1], $entry->[2]) = (($entry->[2] - $entry->[1])/60,
                                              ($entry->[2] - $entry->[1])%60);
                $entry->[1] .= ' min ';
                $entry->[2] .= ' sec';
            }

            # print rest of current log entry
            print '     <TD ALIGN=CENTER>'.$entry->[3].'</TD>'."\n".
                  '     <TD ALIGN=CENTER>'.$entry->[4].'</TD>'."\n".
                  '     <TD ALIGN=CENTER>'.$entry->[5].'</TD>'."\n".
                  '     <TD ALIGN=CENTER>'.$entry->[6].'</TD>'."\n".
                  '     <TD ALIGN=CENTER>'.$entry->[1].$entry->[2].'</TD>'."\n".
                  '    </TR>'."\n";
        }

        # check if log file is empty
        if (@log == 0) {
            print '    <TR BGCOLOR=#FFC0C0>'."\n".
                  '     <TD COLSPAN=6>No entries in log file.</TD>'."\n".
                  '    <TR>'."\n";
        }

        # print copyright
        print '    <TR>'."\n".
              '     <TD COLSPAN=6 ALIGN=RIGHT><FONT SIZE=1 COLOR=#E0E0E0>&copy; 
Marco Lehmann 1999</FONT></TD>'."\n".
              '    </TR>'."\n";

        # close log file
        close LOG;
    }

    # generate HTML-Footer
    print '   </TABLE>'."\n".
          '  </DIV>'."\n".
          ' </BODY>'."\n".
          '</HTML>'."\n";
}


#========================================================================================
# name        : read_server_files
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : read_server_files()
#
#----------------------------------------------------------------------------------------
# description : retrives all files that resides on the 'Bildungsserver' from a 
local
#               file
#
#----------------------------------------------------------------------------------------
# created     : 14.01.1999
#========================================================================================

sub read_server_files {

    # an error already occured
    return if ($error_msg ne '');

    # open server-files-database or abort on errors
    unless (open FD, '<'.$lib_dir.$files_server_filename) {
        $error_msg = 'Can\'t open server-files-database 
('.$lib_dir.$files_server_filename.').';
        return;
    };

    # set input-record-separator to 0 (only in this scope)
    local ( $/ ) = "\0";

    # read server-files-database at once
    %files_server = map { chomp; $_; } <FD>;

    # close server-files-database
    close FD;
}


#========================================================================================
# name        : read_local_files
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : read_local_files()
#
#----------------------------------------------------------------------------------------
# description : traverses the local files system (starts at $homepage_dir)
#
#----------------------------------------------------------------------------------------
# created     : 14.01.1999
#========================================================================================

sub read_local_files {

    # an error already occured
    return if ($error_msg ne '');

    # traverse homepage directory and collect filenames
    find \&read_local_files_wanted, '.';
}


sub read_local_files_wanted {
    $files_local{$File::Find::name} = (stat($_))[9];
}


#========================================================================================
# name        : check_files
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : check_files()
#
#----------------------------------------------------------------------------------------
# description : - performs some check to ensure that only new and locally 
changed files
#                 are being transferred
#               - after this operation hash 'files_local' contains all files
#                 which should be transferred and hash 'files_server' contains
#                 all files which should be deleted on 'Bildungsserver'
#
#----------------------------------------------------------------------------------------
# created     : 15.01.1999
#========================================================================================

sub check_files {

    # an error already occured
    return if ($error_msg ne '');

    while ( my ($name, $date) = each %files_local ) {

        # check for new files
        next if (!exists $files_server{$name});

        # check for files that are up to date
        delete $files_local{$name} if ($files_server{$name} >= $date);

        # delete file names from hash that are up to date
        delete $files_server{$name}
    }
}


#========================================================================================
# name        : create_distribution
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : create_distribution()
#
#----------------------------------------------------------------------------------------
# description : - creates from all collected information an tar-file (or not) 
that
#                 contains all files that are locally changed or new and a list
#                 of locally deleted files
#
#----------------------------------------------------------------------------------------
# created     : 15.01.1999
#========================================================================================

sub create_distribution {

    # an error already occured
    return if ($error_msg ne '');

    # create file for tar that contains all files that should be transferred
    unless (open FD, '>'.$lib_dir.$files_transfer_filename) {
        $error_msg = 'Can\'t create transfer-files-database 
('.$lib_dir.$files_transfer_filename.').';
        return;
    }

    # insert delete-files-database in distribution file if necessary
    $files_local{$files_delete_filename} = 0 if (%files_server);

    # fill file for tar that contains all files that should be transferred
    while ( my ($name, $date) = each %files_local ) {

        #skip directories
        if (-d $name) {
            delete $files_local{$name};
            next;
        }

        # write filename to file
        print FD $name, "\n";
    }
    close FD;

    # Are there any files to delete ?
    if (%files_server) {

        # create file that contains all files that should be deleted on server
        unless (open FD,'>'.$files_delete_filename) {
            $error_msg = 'Can\'t create delete-files-database 
('.$homepage_dir.$files_delete_filename.').';
            return;
        }

        # write hash to file
        while ( my ($name, $date) = each %files_server ) {
            print FD $name, "\0";
        }

        # close delete-files-database
        close FD;
    }

    # Are there any files to transfer ?
    if (%files_local) {
        # tar (and zip) selected homepage files
        my ( $cmd ) = 'tar -cf - -T '.$lib_dir.$files_transfer_filename.
                      ' | gzip -c -9 > '.$lib_dir.$homepage_filename;
        system $cmd;

        unless ((0xFFFF & system $cmd) == 0) {
            $error_msg = 'Can\'t execute command ('.$cmd.').';
            unlink $lib_dir.$homepage_filename, $files_delete_filename;
            return;
        }

        # save size of transferred tar file
        $homepage_tar_file_size = -s $lib_dir.$homepage_filename;
    }

    # delete old delete-files-database
    unlink $lib_dir.$files_delete_filename;

    # move delete-files-database to $lib_dir
    rename $files_delete_filename, $lib_dir.$files_delete_filename;
}


#========================================================================================
# name        : transfer_files
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : transfer_files()
#
#----------------------------------------------------------------------------------------
# description :
#
#----------------------------------------------------------------------------------------
# created     : 15.01.1999
#========================================================================================

sub transfer_files {

    # an error already occured
    return if ($error_msg ne '');

    # establish connection to 'Bildungsserver'
    my ( $cmd ) = 'isdn.win-shuttle';
    unless ((0xFFFF & system $cmd) == 0) {
        $error_msg = 'Can\'t execute command ('.$cmd.').';
        return;
    }

    # save time for (online) start
    $online_starttime = time;

    my ( $ftp_error_file ) = 'ftp.error';
    my ( $ftp_success );

    # try several servers for ftp
    my ( $server );
    foreach $server (@servers) {

        print $server->{'name'}, "\n";
        open  FTPSERVER, '| ftp -n 2>'.$lib_dir.$ftp_error_file.' 
>ftp.error.stdout';
        print FTPSERVER 'open '.$server->{'name'}."\n";
        print FTPSERVER 'user '.$server->{'user'}.' '.$server->{'pass'}."\n";
        print FTPSERVER 'bin'."\n";
        print FTPSERVER 'lcd '.$lib_dir."\n";    
        print FTPSERVER 'put '.$homepage_filename."\n";
        print FTPSERVER 'bye'."\n";
        close FTPSERVER;

        # ftp succeed if size == 0 , failed otherwise
        $ftp_success = (-s $lib_dir.$ftp_error_file == 0);

        # try next server if ftp failed
        next if (!$ftp_success);

        # execute program on 'Bildungsserver' to perform all necessary changes
        rexec($server->{'name'}, $server->{'user'}, $server->{'pass'}, 'perl 
bin/hp-sync-srv.pl');

        # if we reached this point, we have a stable connection to a server
        last;
    }

    # first close connection to 'Bildungsserver' (no error processing here)
    system 'isdn.stop';

    # save time for (online) end
    $online_endtime = time;

    # delete tar file and file $ftp_error_file
#    unlink $lib_dir.$homepage_filename; #, $lib_dir.$ftp_error_file;

    # no server available
    if (!$ftp_success) {
        $error_msg = 'FTP failed.';
        return;
    }
}


#========================================================================================
# name        : rexec
#----------------------------------------------------------------------------------------
# input       : $host   - dns-name or ip-address for remote host
#               $user   - name for user on $host
#               $passwd - password for $user on $host
#               $cmd    - command to execute in home directory
#
# output      :  0  - everything went OK
#               -1  - socket() failed
#               -2  - 'execd' couldn't fork or couldn't login
#               -3  - all other failures
#
#----------------------------------------------------------------------------------------
# syntax      : rexec( $host, $user, $passwd, $cmd )
#
#----------------------------------------------------------------------------------------
# description : executes a command on remote host
#
#----------------------------------------------------------------------------------------
# created     : 27.01.1999
#========================================================================================

sub rexec {

    use IO::Socket;

    # assign parameters to local variables
    my ( $host, $user, $passwd, $cmd ) = @_;

    # open socket to host
    my ( $sock ) = IO::Socket::INET->new(PeerAddr => $host,
                                         PeerPort => 'exec(512)',
                                         Proto    => 'tcp');

    # something went wrong with that socket
    return(-1, undef) if (!defined($sock));

    # submit $user, $passwd, $cmd
    $sock->syswrite("0\0", 2);
    $sock->syswrite($user . "\0", length($user) + 1);
    $sock->syswrite($passwd . "\0", length($passwd) + 1);
    $sock->syswrite($cmd . "\0", length($cmd) + 1);

    my ( $status, @retval );

    # retrieve status from 'execd'
    $sock->sysread($status, 1);

    #
    if ($status eq chr(0)) {
        # everything went OK
        @retval = (0, $sock->getlines);
    } elsif ($status eq chr(1)) {
        # 'execd' couldn't fork or couldn't login
        @retval = (-2, $sock->getline);
    } else {
        # unknown problem
        @retval = (-3, $status .= $sock->getline);
    }

    wantarray ? @retval : $retval[0];
}


#========================================================================================
# name        : update_files
#----------------------------------------------------------------------------------------
# input       : -
# output      : -
#----------------------------------------------------------------------------------------
# syntax      : update_files()
#
#----------------------------------------------------------------------------------------
# description : updates server-files-database and log file
#
#----------------------------------------------------------------------------------------
# created     : 15.01.1999
#========================================================================================

sub update_files {

    my ( $files_server_new ) = 0;

    if ( $error_msg eq '' ) {

        # local variables
        my ( %files_server_new ) = ();

        # open server-files-database
        open FD, '<'.$lib_dir.$files_server_filename;

        # save old input-record-separator
        my ( $old_input_sep ) = $/;

        # set input-record-separator to 0
        $/ = "\0";

        # read server-files-database at once
        %files_server_new = map { chomp; $_; } <FD>;

        # set input-record-separator back to old value
        $/ = $old_input_sep;

        # close server-files-database
        close FD;

        # save count of server files
        $files_server_new = scalar keys %files_server_new;

        # add all transferred files to server-files-database
        while ( my ($name, $date) = each %files_local ) {
            $files_server_new{$name} = $date;
        }

        # delete all deleted files from server-files-database
        while ( my ($name, $date) = each %files_server ) {
            delete $files_server_new{$name};
        }

        # open server-files-database
        open FD, '>'.$lib_dir.$files_server_filename;

        # update server-files-database
        while ( my ($name, $date) = each %files_server_new ) {
            print FD $name, "\0", $date, "\0";
        }

        # close server-files-database
        close FD;
    }

    # open log file
    open FD, '>>'.$lib_dir.$log_filename;

    # update log file (append an entry)
    print FD time, "\0",
             $online_starttime, "\0",
             $online_endtime, "\0",
             $files_server_new, "\0",
             scalar keys %files_local, "\0",
             scalar keys %files_server, "\0",
             $homepage_tar_file_size, "\0",
             $error_msg, "\n";

    # close log file
    close FD;
}


#========================================================================================
# end of file
#========================================================================================

 « Vorige im Thread  Dieser Thread  Nächste im Thread » 

 

seitenanfang


 

news about wissen files archive hilfe suchen  
kontakt letzte änderung: 18.04.2000