Notes on Hadoop


Notes on Hadoop

Some notes on Hadoop, just want to share with you guys if you are interested. As you know, big data is pretty hot right now, so I hope this will help you in anyway.

Advertisements

UNIX 1.15 rsync command part 1


Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

rsync OPTIONS SOURCE DESTINATION
SOURCE/DESTINATION can be local or remote
rsync sample.txt local/sample.txt
rsync sample.txt yjiang@testhost:~/sample.txt
rsync -e “ssh -i fuzicast-home.pem” sample.txt hadoop@testhost:~/sample.txt #use this for password-less access
rsync –progress # shows progress of data transfer
# you can time the following commands to check their execution time
rsync -zv sample.txt yjiang@testhost:~/sample.txt # -z for compressed transfer, -v for verbose
rsync -zvr local yjiang@testhost:~/ # -r for recurively copying
rsync -avz local yjiang@testhost:~/ # -a does many things, recursion and preserves timestamp, permission and links
# alternatively, you can use -l (preserve links), -p (preserve permission), -t (timestamp), -o (ownership), -g (group)
rsync -dv local yjiang@testhost:~/ # only synchronize directory tree
rsync -avz –delete local yjiang@testhost:~/ # in addition to synchronization, it will also delete any unmatched files in remote host
rsync -avz –existing local yjiang@testhost:~/ # synchronize but do not add new files to DESTINATION
rsync -avzi local yjiang@testhost:~/ # tells you which side has differences
f+++++++ local/badfile # f means differences in file, + means its new
f.st…. local/sample.txt # s means size differences, t means timestamp changed
rsync -avz –exclude ‘ex*’ local yjiang@testhost:~/ # exclude certain files
rsync -avz –max-size 200k local yjiang@testhost:~/ # set a max size for files to transfer
rsync -avzW local yjiang@testhost:~/ # force whole file transfer
rsync -avzn local yjiang@testhost:~/ # -n will do a dry run with no changes taken effect, it only tells you what would be changed
rsync -avzu local yjiang@testhost:~/ # update remote files only if local is newer than remote
give credit to http://www.thegeekstuff.com/2010/09/rsync-command-examples/

Fuzicast milestone update


First, thank you all for your support and I appreciate every like, every comment, every sharing from you. As we are starting into 2013, I would like to consider 2013 as “Season 2” of Fuzicast :) Over the past 7 months, Fuzicast has reached the following milestones and nothing could’ve been done with you:

600+ subscribers
500 hours watched per month
5000+ views per month
22 episodes made

And we are about to hit 30k views within a few days. So thank you and keep watching, I will have more videos available for you in 2013. 

UNIX-1.14 curl command basics


Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

With curl, you can download or upload through any protocol, http, ftp, sftp, scp, ldap, telnet.
curl http://www.yahoo.com -o output.html
curl -O http://www.yahoo.com/index.html
curl -# -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt -o sample.txt # the -# adds progress bar
curl -r 0-99 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt# get first 100 bytes
curl -r -500 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # get last 500 bytes
echo “Hello World” | curl -T – -u yue:fuzicast ftp://ftphost/fuzicast/sample2.txt
curl -T UNIX-1.14 -u yue:fuzicast ftp://ftphost/fuzicast/unix
curl -T localfile1 servername/remotefile1 -T localfile2 servername/remotefile2
curl -T UNIX-1.14 -u yue:fuzicast -a ftp://ftphost/fuzicast/unix # append to FTP file
curl –ftp-create-dirs -T UNIX-1.14 -u yue:fuzicast ftp://ftphost/fuzicast/unix/test.txt
curl –limit-rate 10240 -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # limit number of bytes per second
.curlrc # curl configuration file
curl -u yue:fuzicast -z sample.txt ftp://ftphost/fuzicast/sample.txt # download remotefile only if it’s newer than localfile
curl -z “Jan 12 2012” -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # download remote file only if it’s newer than Jan 12 2012
curl -B -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt # enforces ASCII transfer during FTP download
curl -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt –create-dirs -o sampledir/sample.txt# create directory if not exist
curl –key id_rsa # use SSH key
curl -u yue:fuzicast ftp://ftphost -Q ‘RNFR /fuzicast/sample.txt’ -Q ‘RNTO /fuzicast/sampleyue.txt’ # rename a remote file in FTP protocol
curl -u yue:fuzicast ftp://ftphost -Q ‘rename /fuzicast/sample.txt /fuzicast/sampleyue.txt’ # rename in SFTP is different from FTP
curl -R -u yue:fuzicast ftp://ftphost/fuzicast/sample.txt -o output.txt # reserve original file timestamp
curl -l -u yue:fuzicast ftp://ftphost/fuzicast/ # list remote filenames
curl -m 1800 -Y 3000 -y 60 servername/filename # speed must be greater than 3000 bytes per second for a minute and download process must be completed within 1800 seconds, otherwise the download will abort

VIM-1.6 More VIM Shortcuts and Tips


Support fuzicast.com by clicking on youtube ads and make sure to subscribe to my channel for more videos on UNIX, Perl and SQL.

facebook: facebook.com/fuzicast
twitter: @fuzicast
youtube: youtube.com/yuejdesigner85

:v/^Binary/d – delete all lines that does not start with Binary
ctrl+w gf with cursor on a filename will open that file in a new tab
gf with cursor on a filename will close current file and open that file
press * with cursor on a word will search for that word in file
vim -p file1 file2 file3 file4 # opens these files in tabs
:tabonly – closes all other tabs except current one
vimdiff – when you press ‘do’, obtain any differences from the other window. ‘dp’ will put any differences to the other window
:sort u – sort unique in vim
:sort – regular sort in vim
vim scp://username@host/PATH – open remote files
ctrl+p – autocomplete
:’t,. !awk ‘{print $3 ” ” $2 ” ” $1}’ – execute awk on current opening file
:Explore – execute, delete, rename file system easily with vim
:earlier 2h – undo up to 2hours
:later 2h – redo up to 2hours
:set spell – turn on spell check for strings within quotes
:set nospell
shift + k – with cursor on a command, press that and see man page
!% – will execute current script

New Videos on VIM and SHELL


Hi all,

It’s been a while since my last video (almost 2 months). Well, I have been busy with something personal that I have to take care of, but don’t you worry, I will always find time to record new videos. I just have so much to share with you guys. This weekend I will try to find some time to record new videos on VIM (I found more shortcuts) and continue on SHELL videos. So stay tuned and follow my posts :)