久久久精品一区ed2k-女人被男人叉到高潮的视频-中文字幕乱码一区久久麻豆樱花-俄罗斯熟妇真实视频

大數(shù)據(jù)之---CDH集群版本部署全網(wǎng)終極篇---更新中-創(chuàng)新互聯(lián)

1、軟件環(huán)境和IP規(guī)劃

創(chuàng)新互聯(lián)建站是一家專注于網(wǎng)站制作、做網(wǎng)站與策劃設(shè)計(jì),潯陽(yáng)網(wǎng)站建設(shè)哪家好?創(chuàng)新互聯(lián)建站做網(wǎng)站,專注于網(wǎng)站建設(shè)十多年,網(wǎng)設(shè)計(jì)領(lǐng)域的專業(yè)建站公司;建站業(yè)務(wù)涵蓋:潯陽(yáng)等地區(qū)。潯陽(yáng)做網(wǎng)站價(jià)格咨詢:028-86922220

RHEL6 角色

jdk-8u45apache-maven-3.3.9

hive-1.1.0-cdh6.7.1-src.tar.gz

hadoop-2.8.1.tar.gz

mysql-connector-java-6.0.6.tar.gz

apache-maven-3.3.9

cloudera-manager-el6-cm5.9.3_x86_64.tar

mysql-5.7

CDH-5.9.3-1.cdh6.9.3.p0.4-el6

172.16.18.133 NN && SN && Jobtrack hadoop01

172.16.18.134 DN && tasktrack hadoop02

172.16.18.136 DN && tasktrack hadoop03

172.16.18.143 DN && tasktrack hadoop04

172.16.18.145 DN && tasktrack hadoop05

NN =namenode   SN=secondarynamenode  DN=datanode

集群介紹：

不收費(fèi)的Hadoop版本主要有三個(gè)（均是國(guó)外廠商），分別是：Apache（最原始的版本，所有發(fā)行版均基于這個(gè)版本進(jìn)行改進(jìn)）、Cloudera版本（Cloudera’s Distribution Including Apache Hadoop，簡(jiǎn)稱CDH）、Hortonworks版本(Hortonworks Data Platform，簡(jiǎn)稱“HDP”），對(duì)于國(guó)內(nèi)而言，絕大多數(shù)選擇CDH版本。

CDH (Cloudera's Distribution, including Apache Hadoop)，是Hadoop眾多分支中的一種，由Cloudera維護(hù)，基于穩(wěn)定版本的Apache Hadoop構(gòu)建，并集成了很多補(bǔ)丁，可直接用于生產(chǎn)環(huán)境。

Cloudera Manager則是為了便于在集群中進(jìn)行Hadoop等大數(shù)據(jù)處理相關(guān)的服務(wù)安裝和監(jiān)控管理的組件，對(duì)集群中主機(jī)、Hadoop、Hive、Spark等服務(wù)的安裝配置管理做了極大簡(jiǎn)化。

集群安裝，本文采用選擇  離線安裝CDH

https://www.cloudera.com/downloads/cdh/5-9-0.html

官網(wǎng)對(duì)CDH的描述，CHD對(duì)system JDK database 等版本支持列表

官網(wǎng)看支持jdk1.8但是部分1.8版本會(huì)報(bào)錯(cuò)，所以我們選擇jdk1.7

2、軟件包裝備

Cloudera Manager軟件包

http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

CDH軟件包  （下載對(duì)應(yīng)Linux版本包）

http://archive.cloudera.com/cdh6/parcels/5.9.3/CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel

http://archive.cloudera.com/cdh6/parcels/5.9.3/CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel.sha1

mysql jdbc驅(qū)動(dòng)版本是：

http://download.softagency.net/MySQL/Downloads/Connector-J/mysql-connector-java-6.0.6.tar.gz

3、系統(tǒng)相關(guān)配置

所有主機(jī)相同  安裝JDK 關(guān)閉selinux  iptables  配置/etc/hosts 配置yum

 [root@hadoop01 ~]# vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_79

export PATH=$JAVA_HOME/bin:$ORACLE_HOME/bin:$R_HOME/bin:$PATH

[root@hadoop01 ~]# getenforce

Disabled

[root@hadoop01 ~]# iptables -L

Chain INPUT (policy ACCEPT)

target   prot opt source        destination

Chain FORWARD (policy ACCEPT)

target   prot opt source        destination

Chain OUTPUT (policy ACCEPT)

target   prot opt source        destination

[root@hadoop01 ~]# cat /etc/hosts

127.0.0.1  localhost

172.16.18.133  hadoop01

172.16.18.134   hadoop02

172.16.18.136   hadoop03

172.16.18.143   hadoop04

172.16.18.145   hadoop05

[root@hadoop01 ~]# hostname

hadoop01

4、配置ssh自動(dòng)登錄互信

參考偽分布式ssh互信配置

每個(gè)節(jié)點(diǎn)驗(yàn)證不需要進(jìn)行交互輸入yes

useradd hadoop  ----建立用戶

ssh hadoop01 date

ssh hadoop02 date

ssh hadoop03 date

ssh hadoop04 date

ssh hadoop05 date

5、修改swap空間的swappiness=0

cat  /proc/sys/vm/swappiness

sysctl vm.swappiness=0

echo 0 > /proc/sys/vm/swappiness

關(guān)閉告警：echo never > /sys/kernel/mm/transparent_hugepage/defrag

6、配置NTP服務(wù)器

先選定主服務(wù)器,其他服務(wù)器都同步這臺(tái)主服務(wù)器的時(shí)間

# hwclock -w

配置開(kāi)機(jī)啟動(dòng)

[root@hadoop01 ~]# chkconfig ntpd on

[root@hadoop01 ~]# chkconfig --list ntpd

[root@hadoop01 ~]#vi /etc/ntp.conf

  (找到這一行,放開(kāi)restrict的注釋,并且修改ip地址)

  # Hosts on local network are less restricted.

  restrict 192.168.128.1 mask 255.255.255.0 nomodify notrap

  (找到這一行,注釋下面的server)

  # Please consider joining the pool (http://www.pool.ntp.org/join.html).

  #server 0.rhel.pool.ntp.org iburst

  #server 1.rhel.pool.ntp.org iburst

  #server 2.rhel.pool.ntp.org iburst

  #server 3.rhel.pool.ntp.org iburst

  添加下面兩行

  server  127.0.1.0   # local clock

  fudge  127.0.1.0 stratum 10

配置其他服務(wù)器

 vi /etc/ntp.conf

  # Hosts on local network are less restricted.

  restrict 192.168.128.1 nomodify notrap noquery

  (注釋下面server)

  # Please consider joining the pool (http://www.pool.ntp.org/join.html).

  #server 0.rhel.pool.ntp.org iburst

  #server 1.rhel.pool.ntp.org iburst

  #server 2.rhel.pool.ntp.org iburst

  #server 3.rhel.pool.ntp.org iburst

  指定時(shí)間服務(wù)

  server 192.168.128.51

所有重啟ntp服務(wù)

[root@hadoop01 ~]# service ntpd restart

[root@hadoop01 ~]# ntpstat

synchronised to NTP server (172.16.18.33) at stratum 12

time correct to within 18 ms

polling server every 64 s

[root@hadoop01 ~]# date

7、禁用ipv6和“透明大頁(yè)面”

[root@hadoop01 ~]# echo "alias ipv6 off" >> /etc/modprobe.d/dist.conf

[root@hadoop01 ~]# echo "alias net-pf-10 off" >> /etc/modprobe.d/dist.conf

[root@hadoop01 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag

[root@hadoop01 ~]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local

[root@hadoop01 ~]#

8、準(zhǔn)備好mysql數(shù)據(jù)庫(kù)

修改 mysql 權(quán)限：

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123' WITH GRANT OPTION;

flush privileges;

delete from user where host !='%';

[root@hadoop01 software]# mysql -h 172.16.18.133  -uroot -p

###############################準(zhǔn)備工作########################################

hadoop01 Server || Agent

hadoop02 Agent

hadoop03 Agent

CDH采用3臺(tái)服務(wù)器，剩下2臺(tái)做集群添加節(jié)點(diǎn)使用

########################################################################

10.CM安裝

安裝cloudera Manager Server、Agent

cdh集群節(jié)點(diǎn)都要安裝 軟件準(zhǔn)備 賬號(hào)建立

主節(jié)點(diǎn)：

[root@hadoop01 software]# ls cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

[root@hadoop01 software]# pwd

/opt/software

[root@hadoop01 software]# ls cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

cloudera-manager-el6-cm5.9.3_x86_64.tar.gz

[root@hadoop01 software]# mkdir /opt/cloudera-manager

[root@hadoop01 software]# tar zxvf cloudera-manager-el6-cm5.9.3_x86_64.tar.gz -C /opt/cloudera-manager/

客戶端配置

/opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-agent/config.ini

server_host=hadoop01  ---在cm server主機(jī)名

[root@hadoop01 software]# useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

[root@hadoop01 software]# id cloudera-scm

uid=495(cloudera-scm) gid=492(cloudera-scm) 組=492(cloudera-scm)

haoop02 hadoop03所有從節(jié)點(diǎn)

useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

mkdir /opt/cloudera-manager

[root@hadoop01 opt]# scp -r /opt/cloudera-manager/cm-5.9.3  hadoop02:/opt/cloudera-manager/

[root@hadoop01 opt]# scp -r /opt/cloudera-manager/cm-5.9.3  hadoop03:/opt/cloudera-manager/

11、配置CM Server數(shù)據(jù)庫(kù)

我們開(kāi)始準(zhǔn)備mysql數(shù)據(jù)庫(kù)建立

[root@hadoop01 ~]# mysql -h272.16.18.133 -uroot -p

mysql>

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123' WITH GRANT OPTION;

flush privileges;

mysql> flush privileges;

[root@hadoop01 schema]# pwd

/opt/cloudera-manager/cm-5.9.3/share/cmf/schema

[root@hadoop01 schema]# ./scm_prepare_database.sh  mysql -hhadoop01 -uroot -p123 --scm-host hadoop01 cmdb root 123

JAVA_HOME=/usr/java/jdk1.7.0_79

Verifying that we can write to /opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server

Sat Apr 28 14:20:38 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

Creating SCM configuration file in /opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server

Executing:  /usr/java/jdk1.7.0_79/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.9.3/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.

Sat Apr 28 14:20:39 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

[              main] DbCommandExecutor        INFO  Successfully connected to database.

All done, your SCM database is configured correctly!

說(shuō)明：這個(gè)腳本就是用來(lái)創(chuàng)建和配置CMS需要的數(shù)據(jù)庫(kù)的腳本。各參數(shù)是指：

mysql：數(shù)據(jù)庫(kù)用的是mysql，如果安裝過(guò)程中用的oracle，那么該參數(shù)就應(yīng)該改為oracle。

-hadoop01：數(shù)據(jù)庫(kù)建立在hadoop01主機(jī)上面,也就是主節(jié)點(diǎn)上面。

-uroot：root身份運(yùn)行mysql。-123：mysql的root密碼是

--scm-host hadoop01 ：CMS的主機(jī)，一般是和mysql安裝的主機(jī)是在同一個(gè)主機(jī)上,

最后三個(gè)參數(shù)是：數(shù)據(jù)庫(kù)名，數(shù)據(jù)庫(kù)用戶名，數(shù)據(jù)庫(kù)密碼。

12、制作CDH本地源

Server節(jié)點(diǎn)

mkdir -p /opt/cloudera/parcel-repo

chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

Agent節(jié)點(diǎn)

mkdir -p /opt/cloudera/parcels

chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

上傳到CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel  manifest.json主節(jié)點(diǎn)/opt/cloudera/parcel-repo/路徑

[root@hadoop01 CDH]# cd /opt/cloudera/parcel-repo/

[root@hadoop01 parcel-repo]# ls

CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel  manifest.json

[root@hadoop01 parcel-repo]# ls

CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel  manifest.json

[root@hadoop01 parcel-repo]# mv manifest.json CDH-5.9.3-1.cdh6.9.3.p0.4-el6.parcel.sha

manifest.json改名文件名與你的 parel包名一致，并加上.sha后綴

13、啟動(dòng)

保障mysql先啟動(dòng)

server：hadoop01

[root@hadoop01 init.d]# pwd

/opt/cloudera-manager/cm-5.9.3/etc/init.d

[root@hadoop01 init.d]# ./cloudera-scm-server start

Starting cloudera-scm-server:

agent:hadoop01 hadoop02 hadoop02

/opt/cloudera-manager/cm-5.9.3/etc/init.d

./cloudera-scm-agent  start

正在啟動(dòng) cloudera-scm-agent：                [確定]

2018-04-28 14:43:37,022 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4

2018-04-28 14:43:37,024 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180

2018-04-28 14:43:37,024 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

出現(xiàn)下面內(nèi)容表示啟動(dòng)成功

14、圖形訪問(wèn)

錯(cuò)誤大全：

問(wèn)題1：JDBC driver驅(qū)動(dòng)

[root@hadoop01 schema]# ./scm_prepare_database.sh mysql cmdb -h hadoop01 -uroot -p123456 --scm-host hadoop01 scm scm scm

JAVA_HOME=/usr/java/jdk1.7.0_79

Verifying that we can write to /opt/cloudera-manager/cm-5.9.3/etc/cloudera-scm-server

[              main] DbProvisioner          ERROR Unable to find the MySQL JDBC driver. Please make sure that you have installed it as per instruction in the installation guide.

[              main] DbProvisioner          ERROR Stack Trace:

java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)[:1.7.0_79]

  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)[:1.7.0_79]

  at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_79]

  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)[:1.7.0_79]

解決方法

[root@hadoop01 software]# ls mysql-connector-java-5.1.46.zip

mysql-connector-java-5.1.46.zip

[root@hadoop01 software]# unzip  mysql-connector-java-5.1.46.zip ^C

[root@hadoop01 software]# cp mysql-connector-java-5.1.46/

build.xml               mysql-connector-java-5.1.46-bin.jar  README.txt

CHANGES                mysql-connector-java-5.1.46.jar    src/

COPYING                README

[root@hadoop01 software]# cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar  /usr/share/java/

[root@hadoop01 software]# mv /usr/share/java/mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar

問(wèn)題2：

dbc url 'jdbc:mysql://hadoop01/?useUnicode=true&characterEncoding=UTF-8'

java.sql.SQLException: Access denied for user 'root'@'hadoop01' (using password: YES)

  at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)[mysql-connector-java.jar:5.1.46]

另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務(wù)器15元起步，三天無(wú)理由+7*72小時(shí)售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國(guó)服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡(jiǎn)單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢(shì)，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場(chǎng)景需求。

新聞名稱：大數(shù)據(jù)之---CDH集群版本部署全網(wǎng)終極篇---更新中-創(chuàng)新互聯(lián)
網(wǎng)站鏈接：http://sd-ha.com/article42/jjihc.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供企業(yè)網(wǎng)站制作、網(wǎng)站設(shè)計(jì)公司、面包屑導(dǎo)航、商城網(wǎng)站、網(wǎng)站改版、外貿(mào)建站

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源： 創(chuàng)新互聯(lián)

成都網(wǎng)站建設(shè)公司