Linux搭建Hadoop开发环境

 Linux搭建Hadoop开发环境
Hadoop环境搭建安装配置:
[1].官网下载Hadoop-2.7.5安装包: hadoop-2.7.5/hadoop-2.7.5.tar.gz
Linux搭建Hadoop开发环境
Linux搭建Hadoop开发环境
[2].把Hadoop-2.7.5安装包利用Xftp5工具上传到:/usr/local/hadoop
Linux搭建Hadoop开发环境
[3].登录Liunx服务器,利用Xhell5进入:cd /usr/local/hadoop:
[root@marklin hadoop]# cd /usr/local/hadoop
       [root@marklin hadoop]#
       并使用tar -xvf 解压:tar -xvf hadoop-2.7.5.tar.gz,
[root@marklin hadoop]# tar -xvf hadoop-2.7.5.tar.gz
Linux搭建Hadoop开发环境
[4].配置Hadoop环境变量,输入:vim /etc/profile
     #Setting HADOOP_HOME PATH
    export HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.5
    export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin::${HADOOP_HOME}/lib
    export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
    export HADOOP_MAPARED_HOME=${HADOOP_HOME}
    export HADOOP_COMMON_HOME=${HADOOP_HOME}
    export HADOOP_HDFS_HOME=${HADOOP_HOME}
    export YARN_HOME=${HADOOP_HOME}
    export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
    export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
    export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
Linux搭建Hadoop开发环境
保存配置,输入:source /etc/profile
[root@marklin ~]# source /etc/profile
       [root@marklin ~]#
Linux搭建Hadoop开发环境
 PS:最总要的2个点:
【1】修改主机名称:vim  /etc/hostname 
【2】修改配置主机与Ip地址的映射:vim  /etc/hosts 
[5].Hadoop修改配置文件:
core-site.xml:Hadoop的核心配置,包含tmp临时配置文件和访问地址,默认端口9000
mapred-site.xml:Hadoop中相关数据处理模型的配置处理
yarn-site.xml:Hadoop中相关Job的配置处理
hdfs-site.xml:Hadoop配置的文件备份个数以及数据文件夹的配置
 
(1). 配置core-site.xml,在Hadoop安装目录[/usr/local/hadoop/hadoop-2.7.5/etc/hadoop]下 输入: vim core-site.xml
[root@marklin ~]# cd /usr/local/hadoop/hadoop-2.7.5/etc/hadoop
 
[root@marklin hadoop]#
Linux搭建Hadoop开发环境
输入:vim core-site.xml
Linux搭建Hadoop开发环境
并配置:
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://marklin.com:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/repository/hdfs/tmp</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
    </property>
    <property>  
       <name>hadoop.proxyuser.hadoop.hosts</name>  
       <value>*</value>  
    </property>  
    <property>  
       <name>hadoop.proxyuser.hadoop.groups</name>  
       <value>*</value>  
    </property>  
</configuration>
 
Linux搭建Hadoop开发环境
同时在文件路径:/usr/local/hadoop/repository/hdfs,创建tmp目录: mkdir tmp
(2) 修改 hdfs-site.xml,并配置:vim hdfs-site.xml
[root@marklin hadoop]# vim hdfs-site.xml
[root@marklin hadoop]# 
Linux搭建Hadoop开发环境
<configuration>
    <property>
        <name>dfs.namenode.name.dir</name> --dfs.namenode.name.dir定义名称节点路径
        <value>/usr/local/hadoop/repository/hdfs/name</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name> --dfs.datanode.data.dir定义数据节点路径
        <value>/usr/local/hadoop/repository/hdfs/data</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.permissions</name> --dfs.permissions定义权限认证
        <value>false</value>
    </property>
    <property>
        <name>dfs.replication</name>--dfs.replication定义文件副本数
        <value>1</value>
    </property>
   <property>
        <name>dfs.namenode.http-address</name>--dfs.namenode.http-address定义服务http的访问
        <value>marklin.com:50070</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>--dfs.namenode.secondary.http-address定义服务http的访问
        <value>marklin.com:50090</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>
<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/hadoop/repository/hdfs/name</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/usr/local/hadoop/repository/hdfs/data</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>marklin.com:50070</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>marklin.com:50090</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
</configuration>
Linux搭建Hadoop开发环境
同时在文件路径:/usr/local/hadoop/repository/hdfs,创建name和data目录: mkdir  name 和mkdir  data
(3) 创建mapred-site.xml文件,输入:cp mapred-site.xml.template mapred-site.xml
[root@marklin hadoop]# cp mapred-site.xml.template mapred-site.xml
Linux搭建Hadoop开发环境
编辑mapred-site.xml文件,并配置:
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>hdfs://marklin.com:8021/</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>marklin.com:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>marklin.com:19888</value>
    </property>
    <property>
        <name>mapreduce.reduce.java.opts</name>
        <value>-Xms2000m -Xmx4600m</value>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>5120</value>
    </property>
    <property>
        <name>mapreduce.reduce.input.buffer.percent</name>
        <value>0.5</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>mapred.tasktracker.reduce.tasks.maximum</name>
        <value>2</value>
    </property>
    <property>
        <name>mapred.system.dir</name>
        <value>/usr/local/hadoop/repository/mapreduce/system</value>
        <final>true</final>
    </property>
    <property>
        <name>mapred.local.dir</name>
        <value>/usr/local/hadoop/repository/mapreduce/local</value>
        <final>true</final>
    </property>
</configuration>
 
Linux搭建Hadoop开发环境
(4) 修改 yarn-site.xml,并输入::vim yarn-site.xml
[root@marklin hadoop]# vim yarn-site.xml
[root@marklin hadoop]#
Linux搭建Hadoop开发环境
并配置:
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>marklin.com</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.staging-dir</name>
        <value>/usr/local/hadoop/repository/mapreduce/staging</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>
    </property>
</configuration>
 
 
Linux搭建Hadoop开发环境
【6】在Hadoop文件目录[/usr/local/hadoop/hadoop-2.7.5/etc/hadoop]下,
对应的 hadoop-env.sh,mapred-env.sh以及yarn-env.sh文件配置JAVA_HOME:export JAVA_HOME=/usr/local/java/jdk1.8.0_162
输入:vim hadoop-env.sh :
[root@marklin hadoop]# vim hadoop-env.sh
[root@marklin hadoop]#
Linux搭建Hadoop开发环境
export JAVA_HOME=/usr/local/java/jdk1.8.0_162
Linux搭建Hadoop开发环境
输入:vim mapred-env.sh:
export JAVA_HOME=/usr/local/java/jdk1.8.0_162
Linux搭建Hadoop开发环境
[root@marklin hadoop]# vim mapred-env.sh
[root@marklin hadoop]#
Linux搭建Hadoop开发环境
输入:vim yarn-env.sh
export JAVA_HOME=/usr/local/java/jdk1.8.0_162
Linux搭建Hadoop开发环境
[root@marklin hadoop]# vim yarn-env.sh
[root@marklin hadoop]#
Linux搭建Hadoop开发环境
 
【6】开放端口:50070
(1)启动防火墙:systemctl start firewalld.service
[root@marklin ~]# systemctl start firewalld.service
[root@marklin ~]#
Linux搭建Hadoop开发环境
(2)启动防火墙:firewall-cmd --zone=public --add-port=50070/tcp --permanent
[root@marklin ~]# firewall-cmd --zone=public --add-port=50070/tcp --permanent
[root@marklin ~]#
Linux搭建Hadoop开发环境
(3)启动:firewall-cmd --reload
[root@marklin ~]# firewall-cmd --reload
[root@marklin ~]# 
Linux搭建Hadoop开发环境
(4)格式化:hdfs namenode -format
[root@marklin ~]# hdfs namenode -format
[root@marklin ~]#
Linux搭建Hadoop开发环境
(5)启动脚本:start-all.sh
[root@marklin ~]# start-all.sh
[root@marklin ~]#
Linux搭建Hadoop开发环境
 
[root@marklin ~]# start-dfs.sh
Starting namenodes on [marklin.com]
marklin.com: starting namenode, logging to /usr/local/hadoop/hadoop-2.7.5/logs/hadoop-root-namenode-marklin.com.out
marklin.com: starting datanode, logging to /usr/local/hadoop/hadoop-2.7.5/logs/hadoop-root-datanode-marklin.com.out
Starting secondary namenodes [marklin.com]
marklin.com: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-2.7.5/logs/hadoop-root-secondarynamenode-marklin.com.out
 
Linux搭建Hadoop开发环境
 
[root@marklin ~]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/hadoop-2.7.5/logs/yarn-root-resourcemanager-marklin.com.out
marklin.com: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.7.5/logs/yarn-root-nodemanager-marklin.com.out
 
Linux搭建Hadoop开发环境
 
[root@marklin ~]# jps
1122 QuorumPeerMain
6034 Jps
1043 QuorumPeerMain
5413 SecondaryNameNode
5580 ResourceManager
5085 NameNode
5709 NodeManager
5230 DataNode
1119 QuorumPeerMain
 
Linux搭建Hadoop开发环境
【7】输入测试地址:
【1】浏览器输入:http://192.168.3.4:50070/dfshealth.html#tab-overview
Linux搭建Hadoop开发环境
【2】浏览器输入:http://192.168.3.4:8088/cluster
Linux搭建Hadoop开发环境

相关推荐