Hadoop Installation(Single-Node)-1/3

-

Hadoop 1.0.4 on Ubuntu Linux 12.04 ( Single Node )- Part -1/3

There are few ways to install Hadoop. 
  • Cloudera Distibution
  • Hadoop PPA
  • Stable download from hadoop.org
I prefer to use the download from hadoop.org. One of the main reason to select this is whenever there is a new version of hadoop, I need not wait for someone else to release their version including the newer version. 

Should we not be installing hadoop on multi nodes?

Yes. But my objective is to set-up a hadoop environment on my laptop, so that I can play around and also get a better understanding of Map-Reduce. For that single-node set-up is sufficient.

Prerequisites

Make sure you have already installed Java 1.6 on my machine. if not please follow this link - 
Install Java 1.6

Create a separate user & usergroup for hadoop.


I like to keep a dedicated user for hadoop. It is much easier when it comes to giving permissions and for various admin acitivies. It is not necessary tho. Create a separate Linux user and usergroup for installing hadoop
sudo addgroup hadoop_group
sudo adduser --ingroup hadoop_group hadoop_usr

Add the hadoop user to SUDOers list.

The best practice is to give only haddop related permission to the hadoop user. But at this stage we are just trying to understand how hadoop works, so let give ALL permissions.
open terminal and connect as admin
sudo visudo
and then add the below line as show in the below screenshot 
hadoop_usr ALL=(ALL:ALL) ALL

Configuring SSH


Hadoop internally uses SSH to manage it nods, So we need to configure SSH access to localhost for our hadoop_usr. So let do that

Before configuring SSH for hadoop make sure you have installed SSH-server on your machine. In most machine it is likely that the ssh server is not installed as only the client is installed by default.
sudo apt-get install openssh-server
now configure SSH for hadoop
su - hadoop_usr
ssh-keygen -t rsa
enter a file name - ( give nothing, just press enter) 
for passphrase - (give click="" enter.="" just="" no="" passphrase="") 
cat /home/hadoop_usr/.ssh/id_rsa.pub >> /home/hadoop_usr/.ssh/authorized_keys
ssh localhost ( first time it will ask a question "do you want to continue..." give "yes")
ssh localhost ( second time it should not ask any question)


Disable IP v6

cat /proc/sys/net/ipv6/conf/all/disable_ipv6
(if the above return 0, then IP v6 is enabled, so we have to disable it)
sudo vi /etc/sysctl.conf
add the below lines to that file and restart the machine and check again
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

contd...

Comments

Post a Comment

Popular posts from this blog

Tableau - Accessing Tableau's DB

-
Image
Click here for other Tableau related Blogs Please note that TABLEAU does not advice anyone to access their DB. So if you are doing it you are on your own. Tableau has 2 DB Postgresql for managing all its reporting repositories FireBird for mapping repositories Tools Required pgAdmin 3 for PostgreSql Flamerobin for FireDB Accessing Tableau's PostgreSQL DB Tableau has 2 buitin userid tableau = with read-only access to some of Tableau's view tblwgadmin = admin user will all privileges Please note tblwgadmin is like a admin user, so if you are using this you have to be very careful. Or if you dont admin privs but just need enough access to read the tables then use just "tableau" user. Get access for "tableau" user.  First set a password for "tableau" userid open command prompt go to <Tableau_Home>\Tableau Server\7.0\bin run -  tabadmin dbpass <give a password> Restart the server Now open pgAdmin3 and from...
Read more

Tableau : Convert ESRI shapes into Tableau Format

-
Image
Click here for other Tableau related Blogs Requirement is to convert data in ESRI Map shapes into Tableau format. Tools Required Quantum GIS -  http://www.qgis.org/en/site/ Python  Download the below Python Code -  https://github.com/venkat-vs-id/python import csv import re import sys #def convertESRI2TableauFormat( p_inputfile, p_outputfile): l_outfileName = 'c:/ESRI_suburb_tab_format.csv' # make changes here.. l_ESRIFileName ='c:/ESRI_suburb.csv' # make changes here.. rdx=0 csv.field_size_limit(sys.maxsize) outfile = open( l_outfileName, 'w', newline='') csvWriter = csv.writer( outfile, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) with open( l_ESRIFileName, 'r') as csvfile: csvR = csv.reader( csvfile, delimiter=',', quotechar='"') for row in csvR: rdx += 1 ci=0 rowValues = [] rowValues.append(row[1]);# ssc_code rowValues.append(...
Read more

Tableau: Convert Oracle Spatial Data into Tableau Format

-
Click here for other Tableau related Blogs I had requirement to shapes we had in Oracle Spatial data( which we used for MapInfo) in Tableau. Since tableau doesnt support Oracle spatial type columns, I had to extract that data in Tableau readable format. Here are the steps to convert oracle spatial data into format useable by Tableau. Run the Oracle package ven_OraSpatial_2_TabData in your oracle environment. Now run the below "ConvertDataFormat" code to extract the data into Tableau format Now use extracted data in tableau ( refer to my blog -  Tableau Maps - Custom shapes / polygon - Basics ) You can also download the code from this link -   VEN_ORASPATIAL_2_TABDATA.zip Oracle package ven_OraSpatial_2_TabData SPEC CREATE OR REPLACE PACKAGE VSUBR.ven_OraSpatial_2_TabData as ----- TYPE TYPE rec_LatLong IS RECORD( Polygon_ID NUMBER, LAT NUMBER, LON NUMBER ); TYPE tab...
Read more