基于ubuntu的hadoop安装和配置

所用版本：2.7.7

说明：在配置安装之前需要先安装java

解压

sudo tar -zxvf hadoop-2.7.7.tar.gz -C /usr/local/

设置权限

cd /usr/local

sudo chown -R erballoon hadoop-2.7.7/

配置环境变量

sudo vim ~/.bashrc

在文件末尾添加以下内容

export HADOOP_HOME=/usr/local/hadoop-2.7.7
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH

source ~/.bashrc

设置hadoop属性

cd /usr/local/hadoop-2.7.7/etc/hadoop/

vi core-site.xml

添加以下内容

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop-2.7.7/hadoopdata</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>

vi hadoop-env.sh

修改java路径为

export JAVA_HOME=/usr/local/jdk1.8.0_291

vi hdfs-site.xml

添加以下内容

    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/hadoop-2.7.7/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/usr/local/hadoop-2.7.7/hdfs/data</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>master:9001</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>

vi yarn-site.xml

添加以下内容

    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:18040</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:18030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:18088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:18025</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:18141</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>

vi mapred-site.xml.template

添加以下内容

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

将mapred-site.xml.template重命名成mapred-site.xml

vi yarn-env.sh

修改java路径为

export JAVA_HOME=/usr/local/jdk1.8.0_291

vi slaves

修改为

slave1
slave2