Venkat$ echo

"Word Count" I know this is one of the common example you will find when searching for hadoop examples. The code for this comes along with the hadoop installation.  This is a very very simple example which you can use to understand how the hadoop code works. Steps: Make some sample files. I have made 2 files - you can download them from this link - test sample files Now load these files into Hadoop's HDFS ./hadoop-1.0.4/bin/hadoop dfs -copyFromLocal /home/venkat/Documents/*.txt /source_data/ You can also see the uploaded files using the hadoop web portal. As i said earlier hadoop installation should contain an example jar which has got "word count" as one of the example. Here is where you can find that example jar /hadoop-examples-1.0.4.jar Use this command to see the classes related to "word count" jar -tvf ./hadoop-1.0.4/hadoop-examples-1.0.4.jar | grep 'wordcount.class' -i If you would like to see the source...

Hadoop shell commands are very similar to linux shell commands. The Below links gives the complete set of commands Hadoop Shell Commands  - http://hadoop.apache.org/docs/r0.18.3/hdfs_shell.html Below are some of the important ones are ./hadoop-1.0.4/bin/hadoop dfs -ls / ./hadoop-1.0.4/bin/hadoop dfs -mkdir /source_data ./hadoop-1.0.4/bin/hadoop dfs -lsr / ./hadoop-1.0.4/bin/hadoop dfs -mkdir /tmp/tmp1 ./hadoop-1.0.4/bin/hadoop dfs -copyFromLocal /home/venkat/Documents/*.txt /source_data/ ./hadoop-1.0.4/bin/hadoop dfs -rmr /source_data/*.txt

Hadoop 1.0.4 on Ubuntu Linux 12.04 ( Single Node )- Part -3/3 Before starting the hadoop, you need to format the HDFS (i.e the hadoop filesystem) via the namenode. What this does is basically clears the directory values given in { dfs.name.dir}. This is the directory used by name-node to keep details of all the data in the data-node( i.e files), basically every detail required to track the which data sits in which data-node. Please understand that it does not store any data itself, it just stored the details of which data sits on which data-nodes. use the below command to format the name-node. ./hadoop-1.0.4/bin/hadoop namenode -format and the result looks like below. Now you can start your hadoop. ./hadoop-1.0.4/bin/start-all.sh and there result looks like this. Also check if all the required hadoop processes are running by typing jps.   If you face any problems then have a look into the logs file and investigate. and If you want to stop hadoop use teh bel...

Hadoop 1.0.4 on Ubuntu Linux 12.04 ( Single Node )- Part -2/3 Download Hadoop Download a stable hadoop version from  http://hadoop.apache.org/ I downloaded the this version - hadoop 1.0.4 and this is the link to it stable hadoop download page . Download the file (hadoop-1.0.4.tar.gz) and copy it under the home directory of hadoop_usr ( ie. /home/hadoop_usr/) Steps to Install Login to terminal as hadoop_usr and extract the contents of the gz file tar -xvf hadoop-1.0.4.tar.gz We have to edit the following configuration file $HADOOP_HOME/conf/hadoop-env.sh $HADOOP_HOME/conf/hdfs-site.xml $HADOOP_HOME/conf/core-site.xml $HADOOP_HOME/conf/mapred-site.xml $HADOOP_HOME/conf/masters $HADOOP_HOME/conf/slaves $HADOOP_HOME/conf/hadoop-env.sh set the Java home # The java implementation to use.Required. # export JAVA_HOME=/usr/lib/j2sdk1.5-sun export JAVA_HOME=/usr/lib/jvm/java-7-oracle $HADOOP_HOME/conf/hdfs-site.xml before editing this file create the f...

Hadoop 1.0.4 on Ubuntu Linux 12.04 ( Single Node )- Part -1/3 There are few ways to install Hadoop.  Cloudera Distibution Hadoop PPA Stable download from hadoop.org I prefer to use the download from hadoop.org. One of the main reason to select this is whenever there is a new version of hadoop, I need not wait for someone else to release their version including the newer version.  Should we not be installing hadoop on multi nodes? Yes. But my objective is to set-up a hadoop environment on my laptop, so that I can play around and also get a better understanding of Map-Reduce. For that single-node set-up is sufficient. Prerequisites Make sure you have already installed Java 1.6 on my machine. if not please follow this link -  Install Java 1.6 Create a separate user & usergroup for hadoop. I like to keep a dedicated user for hadoop. It is much easier when it comes to giving permissions and for various admin acitivies. It is ...

Hadoop Setup and Architecture Hadoop | Installation ( SIngle-Node ) - 1/3 Hadoop | Installation ( SIngle-Node ) - 2/3 Hadoop | Installation ( SIngle-Node ) - 3/3 Hadoop File System Commands Run a Hadoop Example - Word Count