1 安裝JDK
2 設定 SSH 無密碼登入
3 下載安裝Hadoop
4 設定Hadoop環境變數
5 Hadoop組態檔設定
6 建立與格式化HDFS目錄
7 啟動Hadoop
8 開啟Hadoop Web介面
1.安裝JDK
java -version sudo apt-get update sudo apt-get install default-jdk java -version update-alternatives --display java
2.設定 SSH 無密碼登入
sudo apt-get install ssh sudo apt-get install rsync ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa ll /home/hduser/.ssh ll ~/.ssh cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys3.下載安裝Hadoop
wget http://apache.stu.edu.tw/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz sudo tar -zxvf hadoop-2.6.4.tar.gz sudo mv hadoop-2.6.4 /usr/local/hadoop ll /usr/local/hadoop4.設定Hadoop環境變數
修改~/.bashrc
sudo gedit ~/.bashrc輸入下列內容
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export HADOOP_HOME=/usr/local/hadoop export PATH=$PATH:$HADOOP_HOME/bin export PATH=$PATH:$HADOOP_HOME/sbin export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib" export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH讓~/.bashrc修改生效
source ~/.bashrc5.修改Hadoop組態設定檔
sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh輸入下列內容:
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64Step2 修改core-site.xml
sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml在<configuration></configuration>之間,輸入下列內容:
<property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property>
sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml在<configuration></configuration>之間,輸入下列內容:
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property>
sudo cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml在<configuration></configuration>之間,輸入下列內容:
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>Step5 修改hdfs-site.xml
sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml在<configuration></configuration>之間,輸入下列內容:
<property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value> file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value> file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value> </property>6.建立與格式化HDFS 目錄
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode sudo chown hduser:hduser -R /usr/local/hadoop hadoop namenode -format7.啟動Hadoop
啟動start-dfs.sh,再啟動 start-yarn.sh
start-dfs.sh
start-yarn.sh或
啟動全部
start-all.sh查看目前所執行的行程
jps8.開啟Hadoop ResourceManager Web介面
Hadoop ResourceManager Web介面網址
http://localhost:8088/9.NameNode HDFS Web介面
開啟HDFS Web UI網址
http://localhost:50070/
以上內容節錄自這本書有詳細介紹:
Python+Spark 2.0+Hadoop機器學習與大數據分析實戰 http://www.tenlong.com.tw/items
我照著書盎的步驟操作,但在6.建立與格式化HDFS 目錄時
回覆刪除sudo chown hduser:hduser -R /usr/local/hadoop後得到chown: invalid user: ‘hduser:hduser’
請問該如何解決呢?謝謝
建立一個使用者名稱叫hduser
刪除adduser hduser
您好:我要下在hadoop 2.6.4 但是在只有2.6.5,
回覆刪除安裝hadoop 2.6.5 後,執行hadoop namenode - format 出現錯誤...
wget http://archive.apache.org/dist/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz
刪除我用這個成功了
您好:
回覆刪除無法啟動,
請教解決之道,
謝謝!
hduser@hadoop:~$ start-dfs.sh
Starting namenodes on [localhost]
localhost: Permission denied (publickey,password).
Starting datanodes
localhost: Permission denied (publickey,password).
Starting secondary namenodes [hadoop]
hadoop: Permission denied (publickey,password).
2018-04-14 23:03:41,794 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
可能可以看一下 java 的版本有沒有改
刪除您好:
回覆刪除我在執行start-all.sh時出現以下狀況:
localhost: ssh_exchange_identification: read: Connection reset by peer
查了一下應該是金龥出問題,但目前還是找不出解決辦法,請教您解決之道,感謝!
請問我在執行hadoop namenode -format 他出現 hadoop command not found
回覆刪除我要怎麼解決