Hive 组件需要基于之前已部署完毕的 Hadoop 全分布系统，在 master 节点上实现 Hive 组件安装。
1、Hadoop-全分布式配置（全部配置）
2、Hadoop-集群运行（部分配置）
# 查看之前部署的环境[root@master ~]# su - hadoop[hadoop@master ~]$ jps47967 Jps[hadoop@master hadoop]$ start-all.shThis script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [master]master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.outstarting yarn daemonsstarting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out[hadoop@master hadoop]$ jps44678 NameNode45030 ResourceManager45289 Jps44874 SecondaryNameNode[hadoop@master hadoop]$ exitlogout要确保 Hadoop 系统能够正常运行
注意：使用 root 用户登录
[root@master ~]# cd /opt/software/[root@master software]# lsapache-hive-2.0.0-bin.tar.gz jdk-8u152-linux-x64.tar.gz mysql-connector-java-5.1.46.jarhadoop-2.7.1.tar.gz mysql-5.7.18.zip[root@master software]# rpm -qa | grep mariadbmariadb-libs-5.5.52-1.el7.x86_64[root@master software]# rpm -e --nodeps mariadb-libs-5.5.52-1.el7.x86_64[root@master software]# lsapache-hive-2.0.0-bin.tar.gz jdk-8u152-linux-x64.tar.gz mysql-connector-java-5.1.46.jarhadoop-2.7.1.tar.gz mysql-5.7.18.zip[root@master software]# unzip mysql-5.7.18.zip Archive: mysql-5.7.18.zip creating: mysql-5.7.18/ inflating: mysql-5.7.18/mysql-community-client-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-common-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-devel-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-libs-5.7.18-1.el7.x86_64.rpm inflating: mysql-5.7.18/mysql-community-server-5.7.18-1.el7.x86_64.rpm [root@master software]# cd mysql-5.7.18/[root@master mysql-5.7.18]# lsmysql-community-client-5.7.18-1.el7.x86_64.rpm mysql-community-libs-5.7.18-1.el7.x86_64.rpmmysql-community-common-5.7.18-1.el7.x86_64.rpm mysql-community-server-5.7.18-1.el7.x86_64.rpmmysql-community-devel-5.7.18-1.el7.x86_64.rpm[root@master mysql-5.7.18]# rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpmwarning: mysql-community-common-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEYPreparing... ################################# [100%]Updating / installing... 1:mysql-community-common-5.7.18-1.e################################# [100%] [root@master mysql-5.7.18]# rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpmwarning: mysql-community-libs-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEYPreparing... ################################# [100%]Updating / installing... 1:mysql-community-libs-5.7.18-1.el7################################# [100%] [root@master mysql-5.7.18]# rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpmwarning: mysql-community-client-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEYPreparing... ################################# [100%]Updating / installing... 1:mysql-community-client-5.7.18-1.e################################# [100%] [root@master mysql-5.7.18]# rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpmwarning: mysql-community-server-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEYPreparing... ################################# [100%]Updating / installing... 1:mysql-community-server-5.7.18-1.e################################# [100%]添加配置信息到/etc/my.cnf 文件 symbolic-links=0 的下方
[root@master mysql-5.7.18]# vi /etc/my.cnfdefault-storage-engine=innodbinnodb_file_per_tablecollation-server=utf8_general_ciinit-connect='SET NAMES utf8'character-set-server=utf8[root@master mysql-5.7.18]# systemctl start mysqld[root@master mysql-5.7.18]# systemctl status mysqld● mysqld.service - MySQL Server Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2022-04-09 10:12:27 CST; 44s ago Docs: man:mysqld(8) http://dev.mysql.com/doc/refman/en/using-systemd.html Process: 45577 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS) Process: 45499 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS) Main PID: 45580 (mysqld) CGroup: /system.slice/mysqld.service └─45580 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pidApr 09 10:12:17 master.example.com systemd[1]: Starting MySQL Server...Apr 09 10:12:27 master.example.com systemd[1]: Started MySQL Server.[root@master ~]# cat /var/log/mysqld.log | grep password2022-04-09T02:12:24.671882Z 1 [Note] A temporary password is generated for root@localhost: /RrHz+J-t1fj默认密码为/RrHz+J-t1fj
是否更改 root 用户密码n
是否使用设定的密码继续y
是否删除匿名用户n
是否拒绝 root 用户远程登录n
是否删除测试数据库n
是否重新加载授权表n
[root@master ~]# mysql_secure_installationSecuring the MySQL server deployment.Enter password for user root: The existing password for the user account root has expired. Please set a new password.New password: Re-enter new password: The 'validate_password' plugin is installed on the server.The subsequent steps will run with the existing configurationof the plugin.Using existing password for root.Estimated strength of the password: 100 Change the password for root ? ((Press y|Y for Yes, any other key for No) : nEstimated strength of the password: 100 Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : yBy default, a MySQL installation has an anonymous user,allowing anyone to log into MySQL without having to havea user account created for them. This is intended only fortesting, and to make the installation go a bit smoother.You should remove them before moving into a productionenvironment.Remove anonymous users? (Press y|Y for Yes, any other key for No) : n ... skipping.Normally, root should only be allowed to connect from'localhost'. This ensures that someone cannot guess atthe root password from the network.Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n ... skipping.By default, MySQL comes with a database named 'test' thatanyone can access. This is also intended only for testing,and should be removed before moving into a productionenvironment.Remove test database and access to it? (Press y|Y for Yes, any other key for No) : n ... skipping.Reloading the privilege tables will ensure that all changesmade so far will take effect immediately.Reload privilege tables now? (Press y|Y for Yes, any other key for No) : n ... skipping.All done! 添加 root 用户从本地和远程访问 MySQL 数据库表单的授权
[root@master ~]# mysql -uroot -pEnter password: Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 12Server version: 5.7.18 MySQL Community Server (GPL)Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> grant all privileges on *.* to root@'localhost' identified by 'Wangzhigang123!';Query OK, 0 rows affected, 1 warning (0.00 sec)mysql> grant all privileges on *.* to root@'%' identified by 'Wangzhigang123!';Query OK, 0 rows affected, 1 warning (0.00 sec)mysql> flush privileges;Query OK, 0 rows affected (0.00 sec)mysql> select user,host from mysql.user where user='root';+------+-----------+| user | host |+------+-----------+| root | % || root | localhost |+------+-----------+2 rows in set (0.00 sec)mysql> exit;Bye[root@master ~]## 解压hive文件[root@master ~]# tar -zxf /opt/software/apache-hive-2.0.0-bin.tar.gz [root@master ~]# mv apache-hive-2.0.0-bin/ /usr/local/src/hive# 修改用户和用户组为 hadoop[root@master ~]# chown -R hadoop:hadoop /usr/local/src/hive# 添加配置信息[root@master ~]# vi /etc/profile# set hive environmentexport HIVE_HOME=/usr/local/src/hiveexport PATH=$PATH:$HIVE_HOME/bin[root@master ~]# source /etc/profile[root@master ~]# su - hadoop[hadoop@master ~]$ cp /usr/local/src/hive/conf/hive-default.xml.template /usr/local/src/hive/conf/hive-site.xml[hadoop@master ~]$ vi /usr/local/src/hive/conf/hive-site.xml# 设置 MySQL 数据库连接 <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=false</value> <description>JDBC connect string for a JDBC metastore</description># 配置 MySQL 数据库 root 的密码 <name>javax.jdo.option.ConnectionPassword</name> <value>Wangzhigang123!</value> <description>password to use against metastore database</description># 验证元数据存储版本一致性,默认 false <name>hive.metastore.schema.verification</name> <value>false</value># 配置数据库驱动 <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> # 配置数据库用户名 <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>Username to use against metastore database</description> # 配置目录信息（共4处） <name>hive.querylog.location</name> <value>/usr/local/src/hive/tmp</value> <description>Location of Hive run time structured log file</description> <name>hive.exec.local.scratchdir</name> <value>/usr/local/src/hive/tmp</value> <name>hive.downloaded.resources.dir</name> <value>/usr/local/src/hive/tmp/resources</value> <name>hive.server2.logging.operation.log.location</name> <value>/usr/local/src/hive/tmp/operation_logs</value>[hadoop@master ~]$ mkdir -p /usr/local/src/hive/tmp{resources,operation_logs}[hadoop@master ~]$ cp /opt/software/mysql-connector-java-5.1.46.jar /usr/local/src/hive/lib/[hadoop@master ~]$ stop-all.shThis script is Deprecated. Instead use stop-dfs.sh and stop-yarn.shStopping namenodes on [master]master: stopping namenode192.168.100.20: stopping datanode192.168.100.30: stopping datanodeStopping secondary namenodes [0.0.0.0]0.0.0.0: stopping secondarynamenodestopping yarn daemonsstopping resourcemanager192.168.100.20: stopping nodemanager192.168.100.30: stopping nodemanagerno proxyserver to stop[hadoop@master ~]$ start-all.shThis script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [master]master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out192.168.100.20: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out192.168.100.30: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.outstarting yarn daemonsstarting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out192.168.100.30: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out192.168.100.20: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out[hadoop@master ~]$ schematool -initSchema -dbType mysqlwhich: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Metastore connection URL: jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&useSSL=falseMetastore Connection Driver : com.mysql.jdbc.DriverMetastore connection User: rootStarting metastore schema initialization to 2.0.0Initialization script hive-schema-2.0.0.mysql.sqlInitialization script completedschemaTool completed[hadoop@master ~]$ hivewhich: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.propertiesHive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.hive> show databases;OKdefaultTime taken: 0.241 seconds, Fetched: 1 row(s)hive> exit;[hadoop@master ~]$ [hadoop@master ~]$ mysql -uroot -pWangzhigang123!mysql: [Warning] Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 24Server version: 5.7.18 MySQL Community Server (GPL)Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || hive || mysql || performance_schema || sys |+--------------------+5 rows in set (0.00 sec)mysql> exit;Bye[hadoop@master ~]$ jps47057 NameNode47411 ResourceManager47254 SecondaryNameNode47967 Jps[hadoop@master ~]$ slave1节点
[root@slave1 ~]# su - hadoop[hadoop@slave1 ~]$ jps43621 Jps43351 DataNode43464 NodeManagerslave2节点
[root@slave2 ~]# su - hadoop[hadoop@slave2 ~]$ jps41607 Jps41337 DataNode41450 NodeManager声明：未经许可，不得转载