Guadazi
0%

mac搭建hadoop、hive、spark、flink环境

Posted on Edited on In

mysql

启动mysql

brew services start mysql

abc.ABC.123

集群搭建

  1. mac osx 搭建hadoop开发环境
  2. spark on yarn
  3. hive环境搭建

按照上面文章的方式搭建，启动hdfs、yarn

必须使用这个命令

1
2
3
4
5
6
## 启动yarn
hadoop/sbin/./yarn-daemon.sh start resourcemanager
hadoop/sbin./yarn-daemon.sh start nodemanager
## 启动hdfs
hadoop/sbin/start-hdfs.sh

hdfs-web
yarn-web

hive

create user ‘hadoop‘@’%’ identified by ‘hadoop%HADOOP%123’;

grant all privileges on . to ‘hadoop‘@’%’ with grant option;

flush privileges;

flink on yarn部署

在flink lib下增加flink-hadoop兼容包Pre-bundled Hadoop 2.7.5

1
2
# 启动
bin/start-cluster.sh

搭建AthenaX

Building AthenaX and Flink

mac install kafka

1
2
brew install zookeeper
brew install kafka

修改 /usr/local/etc/kafka/server.properties, 找到 listeners=PLAINTEXT://:9092 那一行，把注释取消掉。

1
2
$ brew services start zookeeper
$ brew services start kafka

创建topic

1
/usr/local/bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic order-streaming-test

清理topic

cd /usr/local/var/lib/kafka-logs

install kafka on mac