HDFS 安装和部署
-
一 HDFS 架构简介
HDFS has a master/slave architecture. An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on. HDFS exposes a file system namespace and allows user data to be stored in files. Internally, a file is split into one or more blocks and these blocks are stored in a set of DataNodes. The NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to DataNodes. The DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction from the NameNode.二，安装和配置
2.1 准备账号和网络环境
我们的网络环境和资源如下图
集群组网：
服务器A(计算节点): 192.168.1.2
服务器B(存储节点): 192.168.1.3
两个服务器都创建anneng 账户 作为hadoop的主帐号 并使用该帐号在两个机器之间建立免密登录
http://an.forum.genostack.com/topic/3/ubuntu-18-04-ssh-key的配置以及免密登录?_=1616574188290
在master和slave节点 都配置好hosts域名解析
192.168.1.2 master
192.168.1.3 slave12.2 安装软件
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
Apache Hadoop 3.3 编译时支持Java8 运行时支持Java8和Java11
我们选择OpenJdk8 来部署后面的应用
sudo apt-get update
sudo apt-get install openjdk-8-jdk
记住javac的实际安装路径
which javac
readlink -f /usr/bin/javac
或者dpkg - l |grep openjdk下载hadoop软件
https://mirrors.bfsu.edu.cn/apache/hadoop/common/hadoop-3.2.2/hadoop-3.2.2.tar.gz配置环境变量
#Hadoop Related Options
export HADOOP_HOME=/home/hdoop/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"2.3 配置hadoop
配置env
sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64配置主节点：
sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/ceph_disk2/hadoop_temp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.1.2:7000</value>
</property>
</configuration>//配置hdfs
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/data_raid1/hdfs_namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/ceph_disk3/hdfs_datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>配置从节点
core-size.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/ceph_disk1/hadoop_temp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.1.2:7000</value>
</property>
</configuration>hdfs-site.xml
<configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>/ceph_disk1/hdfs_datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>三 初始化 启动
初始化：
hdfs namenode -format
./sbin/start-dfs.sh可以使用sudo jps查看各个服务器的进程
主节点
17329 DataNode
47634 Jps
17818 SecondaryNameNode从节点
55904 DataNode -
我们系统当前已经有了anneng账号 如果是新机器 可以用下面的方法创建账号：
创建hadoop普通账户
sudo adduser hadoop
su - hadoop (-表示切换到hadoop的新会话 su hadoop会使用当前用户的上下文)安装openssh 服务器之间配置免密登录
sudo apt install openssh-server openssh-client -y
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
ssh localhost -
此回复已被删除！ -
此回复已被删除！ -
此回复已被删除！ -
此回复已被删除！ -
HDFS读取的过程
A client initiates read request by calling 'open()' method of FileSystem object; it is an object of type DistributedFileSystem.
This object connects to namenode using RPC and gets metadata information such as the locations of the blocks of the file. Please note that these addresses are of first few blocks of a file.
In response to this metadata request, addresses of the DataNodes having a copy of that block is returned back.
Once addresses of DataNodes are received, an object of type FSDataInputStream is returned to the client. FSDataInputStream contains DFSInputStream which takes care of interactions with DataNode and NameNode. In step 4 shown in the above diagram, a client invokes 'read()' method which causes DFSInputStream to establish a connection with the first DataNode with the first block of a file.
Data is read in the form of streams wherein client invokes 'read()' method repeatedly. This process of read() operation continues till it reaches the end of block.
Once the end of a block is reached, DFSInputStream closes the connection and moves on to locate the next DataNode for the next block
Once a client has done with the reading, it calls a close() method.写入的过程
A client initiates write operation by calling 'create()' method of DistributedFileSystem object which creates a new file - Step no. 1 in the above diagram.
DistributedFileSystem object connects to the NameNode using RPC call and initiates new file creation. However, this file creates operation does not associate any blocks with the file. It is the responsibility of NameNode to verify that the file (which is being created) does not exist already and a client has correct permissions to create a new file. If a file already exists or client does not have sufficient permission to create a new file, then IOException is thrown to the client. Otherwise, the operation succeeds and a new record for the file is created by the NameNode.
Once a new record in NameNode is created, an object of type FSDataOutputStream is returned to the client. A client uses it to write data into the HDFS. Data write method is invoked (step 3 in the diagram).
FSDataOutputStream contains DFSOutputStream object which looks after communication with DataNodes and NameNode. While the client continues writing data, DFSOutputStream continues creating packets with this data. These packets are enqueued into a queue which is called as DataQueue.
There is one more component called DataStreamer which consumes this DataQueue. DataStreamer also asks NameNode for allocation of new blocks thereby picking desirable DataNodes to be used for replication.
Now, the process of replication starts by creating a pipeline using DataNodes. In our case, we have chosen a replication level of 3 and hence there are 3 DataNodes in the pipeline.
The DataStreamer pours packets into the first DataNode in the pipeline.
Every DataNode in a pipeline stores packet received by it and forwards the same to the second DataNode in a pipeline.
Another queue, 'Ack Queue' is maintained by DFSOutputStream to store packets which are waiting for acknowledgment from DataNodes.
Once acknowledgment for a packet in the queue is received from all DataNodes in the pipeline, it is removed from the 'Ack Queue'. In the event of any DataNode failure, packets from this queue are used to reinitiate the operation.
After a client is done with the writing data, it calls a close() method (Step 9 in the diagram) Call to close(), results into flushing remaining data packets to the pipeline followed by waiting for acknowledgment.
Once a final acknowledgment is received, NameNode is contacted to tell it that the file write operation is complete.基本操作
将本地文件保存到HDFS
$HADOOP_HOME/bin/hdfs dfs -copyFromLocal temp.txt /
查看系统内容
$HADOOP_HOME/bin/hdfs dfs -ls /
将HDFS文件保存到本地
$HADOOP_HOME/bin/hdfs dfs -copyToLocal /temp.txt
创建新的目录
$HADOOP_HOME/bin/hdfs dfs -mkdir /mydirectory -
执行hdfs ls报错 发现 端口9000被docker占用了 重新分配为7000端口
ls: DestHost:destPort master:9000 , LocalHost:localPort anneng01/103.114.101.5:0. Failed on local exception: java.io.IOException: Connection reset by peer -
-
-
