UNIX 1.15 rsync command part 1


Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

rsync OPTIONS SOURCE DESTINATION
SOURCE/DESTINATION can be local or remote
rsync sample.txt local/sample.txt
rsync sample.txt yjiang@testhost:~/sample.txt
rsync -e “ssh -i fuzicast-home.pem” sample.txt hadoop@testhost:~/sample.txt #use this for password-less access
rsync –progress # shows progress of data transfer
# you can time the following commands to check their execution time
rsync -zv sample.txt yjiang@testhost:~/sample.txt # -z for compressed transfer, -v for verbose
rsync -zvr local yjiang@testhost:~/ # -r for recurively copying
rsync -avz local yjiang@testhost:~/ # -a does many things, recursion and preserves timestamp, permission and links
# alternatively, you can use -l (preserve links), -p (preserve permission), -t (timestamp), -o (ownership), -g (group)
rsync -dv local yjiang@testhost:~/ # only synchronize directory tree
rsync -avz –delete local yjiang@testhost:~/ # in addition to synchronization, it will also delete any unmatched files in remote host
rsync -avz –existing local yjiang@testhost:~/ # synchronize but do not add new files to DESTINATION
rsync -avzi local yjiang@testhost:~/ # tells you which side has differences
f+++++++ local/badfile # f means differences in file, + means its new
f.st…. local/sample.txt # s means size differences, t means timestamp changed
rsync -avz –exclude ‘ex*’ local yjiang@testhost:~/ # exclude certain files
rsync -avz –max-size 200k local yjiang@testhost:~/ # set a max size for files to transfer
rsync -avzW local yjiang@testhost:~/ # force whole file transfer
rsync -avzn local yjiang@testhost:~/ # -n will do a dry run with no changes taken effect, it only tells you what would be changed
rsync -avzu local yjiang@testhost:~/ # update remote files only if local is newer than remote
give credit to http://www.thegeekstuff.com/2010/09/rsync-command-examples/

Advertisements

UNIX-1.14 curl command basics


Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

With curl, you can download or upload through any protocol, http, ftp, sftp, scp, ldap, telnet.
curl http://www.yahoo.com -o output.html
curl -O http://www.yahoo.com/index.html
curl -# -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt -o sample.txt # the -# adds progress bar
curl -r 0-99 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt# get first 100 bytes
curl -r -500 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # get last 500 bytes
echo “Hello World” | curl -T – -u yue:fuzicast ftp://ftphost/fuzicast/sample2.txt
curl -T UNIX-1.14 -u yue:fuzicast ftp://ftphost/fuzicast/unix
curl -T localfile1 servername/remotefile1 -T localfile2 servername/remotefile2
curl -T UNIX-1.14 -u yue:fuzicast -a ftp://ftphost/fuzicast/unix # append to FTP file
curl –ftp-create-dirs -T UNIX-1.14 -u yue:fuzicast ftp://ftphost/fuzicast/unix/test.txt
curl –limit-rate 10240 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # limit number of bytes per second
.curlrc # curl configuration file
curl -u yue:fuzicast -z sample.txt ftp://ftphost/fuzicast/sample.txt # download remotefile only if it’s newer than localfile
curl -z “Jan 12 2012” -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # download remote file only if it’s newer than Jan 12 2012
curl -B -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # enforces ASCII transfer during FTP download
curl -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt –create-dirs -o sampledir/sample.txt# create directory if not exist
curl –key id_rsa # use SSH key
curl -u yue:fuzicast ftp://ftphost -Q ‘RNFR /fuzicast/sample.txt’ -Q ‘RNTO /fuzicast/sampleyue.txt’ # rename a remote file in FTP protocol
curl -u yue:fuzicast ftp://ftphost -Q ‘rename /fuzicast/sample.txt /fuzicast/sampleyue.txt’ # rename in SFTP is different from FTP
curl -R -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt -o output.txt # reserve original file timestamp
curl -l -u yue:fuzicast ftp://ftphost/fuzicast/ # list remote filenames
curl -m 1800 -Y 3000 -y 60 servername/filename # speed must be greater than 3000 bytes per second for a minute and download process must be completed within 1800 seconds, otherwise the download will abort

VIM-1.6 More VIM Shortcuts and Tips


Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

:v/^Binary/d – delete all lines that does not start with Binary
ctrl+w gf with cursor on a filename will open that file in a new tab
gf with cursor on a filename will close current file and open that file
press * with cursor on a word will search for that word in file
vim -p file1 file2 file3 file4 # opens these files in tabs
:tabonly – closes all other tabs except current one
vimdiff – when you press ‘do’, obtain any differences from the other window. ‘dp’ will put any differences to the other window
:sort u – sort unique in vim
:sort – regular sort in vim
vim scp://username@host/PATH – open remote files
ctrl+p – autocomplete
:’t,. !awk ‘{print $3 ” ” $2 ” ” $1}’ – execute awk on current opening file
:Explore – execute, delete, rename file system easily with vim
:earlier 2h – undo up to 2hours
:later 2h – redo up to 2hours
:set spell – turn on spell check for strings within quotes
:set nospell
shift + k – with cursor on a command, press that and see man page
!% – will execute current script

SHELL-1.1 Shell Script Introduction and Basics


Make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.
facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/fuzicast

A shell script is basically a script that runs a bunch of unix commands along with conditional statements, loops statements. When programming shell script, space matters.
#!/bin/bash

# SETTING A VARIABLE
greeting=”Hello World”

# SETTING A GLOBAL/ENVIRONMENT VARIABLE
export blogname=”fuzicast”

# ASSIGN OUTPUT OF SUBSHELL TO VARIABLE
filenames=$(ls -ltr) # note that subshell inherits all environment variables from parent shell
no_of_files=`ls -ltr | wc -l`

# HOW TO CREATE MULTIPLE LINE COMMENT
<<COMMENT
The previous line begins a comment block, this is also known as HERE DOCUMENT
Here's another comment
The following line ends the comment block
COMMENT

# PERFORM ARITHMETIC CALCULATION
var1=45
var2=30
echo $[ $var1 + $var2 ]
(( sum=$var1 + $var2 ))
echo $sum

# IF STATEMENT
if [ $[ $var1 + $var2 ] -gt 70 ];then
echo "sum is greater than 70"
elif [ $[ $var1 + $var2 ] -lt 70 ];then
echo "sum is less than 70"
else
echo "unknown"
fi

# WHILE STATEMENT
var1=30
while [ $var1 -gt 0 ];do
echo "while \$var1 is greater than 0, keep running … "
var1=$[ $var1 – 1 ]
done

# UNTIL STATEMENT
var1=0
until [ $var1 -gt 30 ];do
echo "until \$var1 is greater than 30, keep running …"
var1=$[ $var1 + 1 ]
done

# FOR STATEMENT
for arg in $@;do
echo $arg
done

# CASE STATEMENT
var1=1
case $var1 in
1) echo "hello world"
echo "fuzicast video tutorials";;
2) echo "wrong option";;
3) echo "guess again";;
*) echo "Default option";;
esac

Fuzicast hits 5000+ views; New video on UNIX pipes and direction


Wow, after a little more than 2 months since inception of the blog, fuzicast.com now has over 100+ subscribers (twitter, youtube, facebook, wordpress), 5000+ views on the blog and more than 4000+ views on Youtube videos. Is it worth for a celebration? I say yes! But above all, this couldn’t have been done without all the viewers and subscribers. Thank you for making this happen. More videos will be coming out this weekend to celebrate! =)

PS: If you like the blog, please feel free to share fuzicast.com with your friends and coworkers. Also make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.
facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

UNIX-1.12 SSH Command


Make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.
facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/fuzicast

ssh username@host – remote login to a remote host
ssh -i ssh-key username@hostname – using another key to ssh remote host
ssh-keygen -t rsa/dsa – generate ssh keys using either rsa or dsa encryption
sudo useradd -m username – create user with home directory created automatically
passwd username – change password for that user
sudo usermod username -s /bin/bash – set type of shell for user
~/.ssh/config – create this config file and add Host, User, Hostname and IdentityFile settings
Host alias_name
Hostname localhost
IdentityFile ~/.ssh/obama.key
User username
ssh-copy-id username@hostname – copy ssh public automatically to remote host
ssh username@hostname ‘remote-command’ – execute remote commands
ssh -F another_config_file username@hostname – specify different config file
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no username@hostname – do not check for known hosts and add host key automatically to known_hosts file. since we set known_host file to /dev/null, new host key will be added to garbage file.

UNIX-1.11 zip, basename, mktemp, script and rcs commands


Make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.
facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

zip -r filename.zip file1 file2 file3 – zip and compress files, recursively
unzip filename.zip – unzip a zip file
rmdir dirname – remove empty directory
basename fullpath – returns the file or directory name
dirname fullpath – returns the directory
mktemp – creates a temporary file and outputs temp filename
factor number – list the factors of a number
script -a script.out – starts recording your output screen, ctrl+D to stop recording
kill %% – kills previous command
sudo apt-get install rcs – to install linux revision control system (RCS)
mkdir RCS – create a directory called RCS to store all versions of file
ci script.sh – checks in a file, file is not readable until checked out
ci -u script.sh – checks in a file, file is visible for others to read
co -l script.sh – checks out a file, holds the lock.
co -lr1.1 script.sh – checks out a version of file, holds the lock
rcsdiff script.sh – compares current file with latest checked in version
rcsdiff -r1.1 -r1.2 script.sh – compares version 1.1 and 1.2
which command – tells you where such command resides in
locate keyword – searches for files or paths that contains keyword (that are accessible to you)