UNIX 1.15 rsync command part 1

Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

rsync OPTIONS SOURCE DESTINATION
SOURCE/DESTINATION can be local or remote
rsync sample.txt local/sample.txt
rsync sample.txt yjiang@testhost:~/sample.txt
rsync -e “ssh -i fuzicast-home.pem” sample.txt hadoop@testhost:~/sample.txt #use this for password-less access
rsync –progress # shows progress of data transfer
# you can time the following commands to check their execution time
rsync -zv sample.txt yjiang@testhost:~/sample.txt # -z for compressed transfer, -v for verbose
rsync -zvr local yjiang@testhost:~/ # -r for recurively copying
rsync -avz local yjiang@testhost:~/ # -a does many things, recursion and preserves timestamp, permission and links
# alternatively, you can use -l (preserve links), -p (preserve permission), -t (timestamp), -o (ownership), -g (group)
rsync -dv local yjiang@testhost:~/ # only synchronize directory tree
rsync -avz –delete local yjiang@testhost:~/ # in addition to synchronization, it will also delete any unmatched files in remote host
rsync -avz –existing local yjiang@testhost:~/ # synchronize but do not add new files to DESTINATION
rsync -avzi local yjiang@testhost:~/ # tells you which side has differences
f+++++++ local/badfile # f means differences in file, + means its new
f.st…. local/sample.txt # s means size differences, t means timestamp changed
rsync -avz –exclude ‘ex*’ local yjiang@testhost:~/ # exclude certain files
rsync -avz –max-size 200k local yjiang@testhost:~/ # set a max size for files to transfer
rsync -avzW local yjiang@testhost:~/ # force whole file transfer
rsync -avzn local yjiang@testhost:~/ # -n will do a dry run with no changes taken effect, it only tells you what would be changed
rsync -avzu local yjiang@testhost:~/ # update remote files only if local is newer than remote
give credit to http://www.thegeekstuff.com/2010/09/rsync-command-examples/

UNIX-1.14 curl command basics

Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

With curl, you can download or upload through any protocol, http, ftp, sftp, scp, ldap, telnet.
curl http://www.yahoo.com -o output.html
curl -O http://www.yahoo.com/index.html
curl -# -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt -o sample.txt # the -# adds progress bar
curl -r 0-99 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt# get first 100 bytes
curl -r -500 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # get last 500 bytes
echo “Hello World” | curl -T – -u yue:fuzicast ftp://ftphost/fuzicast/sample2.txt
curl -T UNIX-1.14 -u yue:fuzicast ftp://ftphost/fuzicast/unix
curl -T localfile1 servername/remotefile1 -T localfile2 servername/remotefile2
curl -T UNIX-1.14 -u yue:fuzicast -a ftp://ftphost/fuzicast/unix # append to FTP file
curl –ftp-create-dirs -T UNIX-1.14 -u yue:fuzicast ftp://ftphost/fuzicast/unix/test.txt
curl –limit-rate 10240 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # limit number of bytes per second
.curlrc # curl configuration file
curl -u yue:fuzicast -z sample.txt ftp://ftphost/fuzicast/sample.txt # download remotefile only if it’s newer than localfile
curl -z “Jan 12 2012” -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # download remote file only if it’s newer than Jan 12 2012
curl -B -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # enforces ASCII transfer during FTP download
curl -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt –create-dirs -o sampledir/sample.txt# create directory if not exist
curl –key id_rsa # use SSH key
curl -u yue:fuzicast ftp://ftphost -Q ‘RNFR /fuzicast/sample.txt’ -Q ‘RNTO /fuzicast/sampleyue.txt’ # rename a remote file in FTP protocol
curl -u yue:fuzicast ftp://ftphost -Q ‘rename /fuzicast/sample.txt /fuzicast/sampleyue.txt’ # rename in SFTP is different from FTP
curl -R -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt -o output.txt # reserve original file timestamp
curl -l -u yue:fuzicast ftp://ftphost/fuzicast/ # list remote filenames
curl -m 1800 -Y 3000 -y 60 servername/filename # speed must be greater than 3000 bytes per second for a minute and download process must be completed within 1800 seconds, otherwise the download will abort