CentOS 7 搭建CDH5.12.0及Kafka安装 - 简书

字数 882阅读 263


cdh5.12.0地址


***步骤3~8集群的每个机器都要操作一次
3.部署jdk,家目录设置为$JAVA_HOME=/usr/java/jdk1.8.0_45,永久关闭SELINUX并关闭防火墙

//关闭SELINUX,修改配置文件/etc/selinux/config,将SELINU置为disabled
[root@hadoop001 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 
[root@hadoop001 ~]# sestatus
SELinux status:                 disabled
//关闭防火墙
[root@hadoop001 ~]# systemctl stop firewalld.service
[root@hadoop001 ~]# systemctl status firewalld.service  //验证
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1) 
[root@hadoop001 ~]# systemctl disable firewalld.service
[root@hadoop001 ~]# systemctl list-unit-files | grep firewalld  //验证
firewalld.service                             disabled 

4.检查python版本(系统默认的python版本不要修改,如果生产上跑Python服务需要更高版本,就另外装一个)

[root@hadoop001 ~]# python --version
Python 2.7.5

赢百万彩票注册5.配置NTP网络时间协议,同步时区和时间

[root@hadoop001 ~]# timedatectl set-timezone Asia/Shanghai
[root@hadoop001 ~]# timedatectl status
      Local time: Fri 2019-02-22 22:17:09 CST
  Universal time: Fri 2019-02-22 14:17:09 UTC
        RTC time: Fri 2019-02-22 14:17:09
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: no
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a
[root@hadoop001 ~]# yum install -y chrony
[root@hadoop001 ~]# systemctl start chronyd
[root@hadoop001 ~]# systemctl enable chronyd
[root@hadoop001 ~]# yum install -y ntpdate
[root@hadoop001 ~]# systemctl enable ntpd.service
[root@hadoop001 ~]# timedatectl set-ntp yes
[root@hadoop001 ~]# timedatectl status
      Local time: Wed 2019-03-06 22:59:01 CST
  Universal time: Wed 2019-03-06 14:59:01 UTC
        RTC time: Wed 2019-03-06 14:59:01
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a
//上面的命令要在三台机器都执行,下面的只在hadoop001执行
[root@hadoop001 ~]# cp /etc/ntp.conf /etc/ntp.conf.bak
[root@hadoop001 ~]# cp /etc/sysconfig/ntpd /etc/sysconfig/ntpd.bak
[root@hadoop001 ~]# echo "restrict 192.168.137.0 mask 255.255.255.0 nomodify notrap" >> /etc/ntp.conf
[root@hadoop001 ~]# echo "SYNC_HWCLOCK=yes" >> /etc/sysconfig/ntpd
//上面的命令只在hadoop001执行,下面的要在三台机器都执行
[root@hadoop001 ~]# systemctl restart ntpd
[root@hadoop001 ~]# crontab -e
*/30 * * * * /usr/sbin/ntpdate 192.168.137.2
no crontab for root - using an empty one
crontab: installing new crontab

如果不执行此步骤集群会告警

6.关闭大页面

[root@hadoop001 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop001 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@hadoop001 ~]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'>>  /etc/rc.local
[root@hadoop001 ~]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'>>  /etc/rc.local

7.swap 物理磁盘空间可以内存使用

[root@hadoop001 ~]# echo 'vm.swappiness = 10' >> /etc/sysctl.conf
[root@hadoop001 ~]# sysctl -p
vm.swappiness = 10

swap的取值范围是0~100,0不代表禁用,但是惰性最高;100代表积极性最高
对集群计算的实时性要求高的,swap设置为0,能不使用磁盘当内存就不使用,允许job挂 迅速的加内存或调大参数然后重启job;
对集群计算的实时性要求不高的,swap一般设置为10或30,不允许job挂,让job慢慢的运行
8.安装http和启动http服务

[root@hadoop001 ~]# yum install -y httpd
[root@hadoop001 ~]# rpm -qa|grep httpd
httpd-2.2.15-69.el6.centos.x86_64
httpd-tools-2.2.15-69.el6.centos.x86_64
[root@hadoop001 ~]# systemctl list-unit-files | grep httpd
httpd.service                                 disabled
[root@hadoop001 ~]# systemctl enable httpd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
[root@hadoop001 ~]# systemctl list-unit-files | grep httpd
httpd.service                                 enabled 
[root@hadoop001 ~]# systemctl start httpd.service
[root@hadoop001 ~]# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-03-06 23:18:59 CST; 35s ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 3478 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─3478 /usr/sbin/httpd -DFOREGROUND
           ├─3479 /usr/sbin/httpd -DFOREGROUND
           ├─3480 /usr/sbin/httpd -DFOREGROUND
           ├─3481 /usr/sbin/httpd -DFOREGROUND
           ├─3482 /usr/sbin/httpd -DFOREGROUND
           └─3483 /usr/sbin/httpd -DFOREGROUND

Mar 06 23:18:58 hadoop001 systemd[1]: Starting The Apache HTTP Server...
Mar 06 23:18:58 hadoop001 httpd[3478]: AH00558: httpd: Could not reliably d...ge
Mar 06 23:18:59 hadoop001 systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.

8‘。3台机器配置SSH相互通信信任

// 8.1  3台机器执行 ssh-keygen
[root@hadoop001 ~]# ssh-keygen
// 8.2  选取hadoop001,生成authorized_keys文件
[root@hadoop001 .ssh]# ll
total 8
-rw------- 1 root root 1675 Mar  7 15:37 id_rsa
-rw-r--r-- 1 root root  396 Mar  7 15:37 id_rsa.pub
[root@hadoop001 .ssh]# cat /root/.ssh/id_rsa.pub>> /root/.ssh/authorized_keys
[root@hadoop001 .ssh]# ll
total 12
-rw-r--r-- 1 root root  396 Mar  7 15:39 authorized_keys
-rw------- 1 root root 1675 Mar  7 15:37 id_rsa
-rw-r--r-- 1 root root  396 Mar  7 15:37 id_rsa.pub
// 8.3  将其他2台的id_rsa.pub内容,手动copy到第一台的authorized_keys文件
[root@hadoop002 .ssh]# more id_rsa.pub
[root@hadoop003 .ssh]# more id_rsa.pub
//拷贝至authorized_keys文件(注意copy时,最好先放到记事本中,将回车去掉,成为一行,不然很可能配置互相信任失败)
// 8.4  设置权限(每台机器)
[root@hadoop001 .ssh]# chmod 700 -R ~/.ssh
[root@hadoop001 .ssh]# chmod 600 ~/.ssh/authorized_keys //只有第一台有
// 8.5  将第一台的authorized_keys scp 给其他2台(第一次传输,需要输入密码)
[root@hadoop001 .ssh]# scp authorized_keys root@hadoop002:/root/.ssh
[root@hadoop001 .ssh]# scp authorized_keys root@hadoop003:/root/.ssh
//8.6  验证(每台机器上执行下面命令,只输入yes,不输入密码,则这5台互相通信了)
[root@hadoop001 .ssh]# ssh root@hadoop002 date
[root@hadoop001 .ssh]# ssh root@hadoop003 date

三、配置离线数据源
****步骤9~11只在server一台机器上创建即可
9.创建 parcels文件

[root@hadoop001 ~]# cd /var/www/html
[root@hadoop001 html]# mkdir parcels
[root@hadoop001 html]# cd parcels
[root@hadoop001 parcels]# rz  (上传CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel、CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1、manifest.json三个文件,然后将CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1重命名)
[root@hadoop001 parcels]# ll
total 1662604
-rw-r--r-- 1 root root 1702423659 Feb 23 00:38 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel
-rw-r--r-- 1 root root         41 Feb 22 22:50 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1
-rw-r--r-- 1 root root      72612 Feb 23 09:06 manifest.json
[root@hadoop001 parcels]# mv CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha
[root@hadoop001 parcels]# ll
total 1662604
-rw-r--r-- 1 root root 1702423659 Feb 23 00:38 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel
-rw-r--r-- 1 root root         41 Feb 22 22:50 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha
-rw-r--r-- 1 root root      72612 Feb 23 09:06 manifest.json

10.校验parcels文件下载未损坏(生产上因为没校验吃过亏)

[root@hadoop001 parcels]# sha1sum CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel
fa704f42b8da8916409c3f52f189629152ba2839  CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel
[root@hadoop001 parcels]# cat CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha
fa704f42b8da8916409c3f52f189629152ba2839

11.下载、上传、解压cloudera-manager-centos7-cm5.12.0_x86_64.tar.gz,然后移动到和官网一样的目录路径cm5/redhat/6/x86_64/下

[root@hadoop001 parcels]# mkdir -p /opt/rpminstall
[root@hadoop001 parcels]# cd /opt/rpminstall
[root@hadoop001 rpminstall]# ll
-rw-r--r-- 1 root root 952281419 Mar  7 14:18 cm5.12.0-centos7.tar.gz
[root@hadoop001 rpminstall]# tar -zxf cm5.12.0-centos7.tar.gz -C /var/www/html/
[root@hadoop001 rpminstall]# cd /var/www/html/
[root@hadoop001 html]# ll
drwxrwxr-x 3 1106  592 4096 Jul  7  2017 cm
drwxr-xr-x 2 root root 4096 Feb 23 09:17 parcels
[root@hadoop001 html]# mkdir -p cm5/redhat/7/x86_64/
[root@hadoop001 html]# mv cm cm5/redhat/7/x86_64/
[root@hadoop001 html]# ll
total 8
drwxr-xr-x 3 root root 4096 Mar  7 00:01 cm5
drwxr-xr-x 2 root root 4096 Feb 23 09:17 parcels

***步骤12每个机器都要配置cloudera-manager.repo文件
12.配置本地的yum源,cdh集群在安装时会就从本地down包,不会从官网了

[root@hadoop001 html]# vi /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
name = Cloudera Manager, Version 5.12.0
baseurl = http://192.168.137.2/cm5/redhat/7/x86_64/cm/5/
gpgcheck = 0

浏览器查看下面两个网址是否出来,假如有,就配置成功


(192.168.137.2要替换为外网IP)

四、使用RPM包安装并启动CM服务
13.在cm实例中安装服务器rpm包

[root@hadoop001 ~]# cd /var/www/html/cm5/redhat/7/x86_64/cm/5/RPMS/x86_64
[root@hadoop001 x86_64]# ll
total 932400
-rw-rw-r-- 1 1106 592   9776948 Jul  7  2017 cloudera-manager-agent-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
-rw-rw-r-- 1 1106 592 701119748 Jul  7  2017 cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
-rw-rw-r-- 1 1106 592      8720 Jul  7  2017 cloudera-manager-server-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
-rw-rw-r-- 1 1106 592     10620 Jul  7  2017 cloudera-manager-server-db-2-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
-rw-rw-r-- 1 1106 592  30604352 Jul  7  2017 enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
-rw-rw-r-- 1 1106 592  71204325 Jul  7  2017 jdk-6u31-linux-amd64.rpm
-rw-rw-r-- 1 1106 592 142039186 Jul  7  2017 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
[root@hadoop001 x86_64]# yum install -y cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
[root@hadoop001 x86_64]# yum install -y cloudera-manager-server-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm

14。配置mysql-connector-java。jar

[root@hadoop001 x86_64]# mkdir -p /usr/share/java
[root@hadoop001 x86_64]# cd /usr/share/java
[root@hadoop001 x86_64]# rz
[root@hadoop001 java]# ll
total 968
-rw-r--r-- 1 root root 989495 Sep 27 07:05 mysql-connector-java.jar

注意一定要把版本号去掉,名字改成'mysql-connector-java.jar'
15.MySQL配置cmf 用户 and 数据库

mysql> create database cmf DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'cmf_password';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

16。设置cloudera-scm-server 连接到 MySQL

[root@hadoop001 java]# cd /etc/cloudera-scm-server/
[root@hadoop001 cloudera-scm-server]# vi db.properties
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#

# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=192.168.137.2:3306

# The database name
com.cloudera.cmf.db.name=cmf

# The database user
com.cloudera.cmf.db.user=cmf

# The database user's password
com.cloudera.cmf.db.password=cmf_password

# The db setup type
# By default, it is set to INIT
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
com.cloudera.cmf.db.setupType=EXTERNAL

17.启动cm server

[root@hadoop001 cloudera-scm-server]# service cloudera-scm-server start
Starting cloudera-scm-server (via systemctl):              [  OK  ]
//可以查看下方日志来实时观察cm server启动是否有问题
[root@hadoop001 cloudera-scm-server]# cd /var/log/cloudera-scm-server/
[root@hadoop001 cloudera-scm-server]# tail -f cloudera-scm-server.log
2018-10-13 13:13:57,831 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180
2018-10-13 13:13:57,831 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

****此处遇见一个坑,因为用的云主机,一定要设置内外网端口转发和放开防火墙7180端口,不然死活都打不开ip:7180
赢百万彩票注册 18.打开web界面,安装CM server 和 agent



19.集群一系列设置


mysql> create database amon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.01 sec)
mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)


20.kafka部署

[root@hadoop001 ~]# cd /var/www/html
[root@hadoop001 html]# mkdir kafka_parcels
[root@hadoop001 html]# cd kafka_parcels
[root@hadoop001 html]# rz
[root@hadoop001 kafka_parcels]# ll
total 66536
-rw-r--r-- 1 root root 68116503 Oct 13 17:05 KAFKA-2.1.1-1.2.1.1.p0.18-el7.parcel
-rw-r--r-- 1 root root       41 Oct 13 17:04 KAFKA-2.1.1-1.2.1.1.p0.18-el7.parcel.sha1
-rw-r--r-- 1 root root     5252 Feb  6  2018 manifest.json
[root@hadoop001 kafka_parcels]# mv KAFKA-2.1.1-1.2.1.1.p0.18-el7.parcel.sha1 KAFKA-2.1.1-1.2.1.1.p0.18-el7.parcel.sha
[root@hadoop001 kafka_parcels]# ll
total 66536
-rw-r--r-- 1 root root 68116503 Oct 13 17:05 KAFKA-2.1.1-1.2.1.1.p0.18-el7.parcel
-rw-r--r-- 1 root root       41 Oct 13 17:04 KAFKA-2.1.1-1.2.1.1.p0.18-el7.parcel.sha
-rw-r--r-- 1 root root     5252 Feb  6  2018 manifest.json

如果安装过程中出了问题,在机器上[root@hadoop001 ~]# cd /var/log/目录下找到的都是role log(角色日志)很多情况下找不到问题出在哪里,这种情况下应该去看stdout,stderr日志,通过web界面找



五、如何停止集群,启动集群
[root@hadoop001 ~]# service cloudera-scm-agent stop
Stopping cloudera-scm-agent:                               [  OK  ]

//上面的命令要在三台机器都执行,下面的只在hadoop001执行

[root@hadoop001 ~]# service cloudera-scm-server stop
Stopping cloudera-scm-server:                              [  OK  ]
[root@hadoop001 ~]# service mysql stop
Shutting down MySQL..                                      [  OK  ]

(启动时反过来就行了,注意mysql是部署在mysqladmin用户下,service mysql start要在mysqladmin用户)

[root@hadoop001 ~]# su - mysqladmin
[mysqladmin@hadoop001 ~]$ service mysql start
Starting MySQL                                             [  OK  ]
[root@hadoop001 ~]# service cloudera-scm-server start
//上面的命令只在hadoop001执行,下面的要在三台机器都执行
[root@hadoop001 ~]# service cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]

推荐阅读更多精彩内容