Guadazi
0%

Flink 本地环境搭建

Posted on Edited on In

Flink打包

Flink 1.9打包，需要maven 3.2.5

1
2
mvn install -U -Dfast -nsu -Dflink.forkCount=4 -Dflink.forkCountTestPackage=4 -Dskip-webui-build -B -Dmaven.test.skip=true

1
mvn install -U -Dfast -Dmaven.test.skip=true -T 8 -nsu -Dflink.forkCount=8 -Dflink.forkCountTestPackage=8 -Dskip-webui-build -B
1
mvn install -U -Dfast -Dmaven.test.skip=true  -nsu -Dskip-webui-build -B -am -pl flink-dist

flink-dist模块的target目录下会打出flink-1.9-tq-SNAPSHOT-bin，该目录可以直接运行

开启远程调试

开启jre远程调试配置

修改flink客户端的conf/flink-conf.yaml配置文件，添加remote debug调试命令

1
2
3
4
5
6
7
8
9
10
# jobmanager debug端口
env.java.opts.jobmanager: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5006"
#  taskmanager debug端口
env.java.opts.taskmanager: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
# 设置cliFrontend 客户端的debug端口
env.java.opts.client: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5008"

# 设置超时 时间 . 单位 毫秒.
rest.connection-timeout: 360000000
rest.idleness-timeout: 360000000

IDEA配置

添加bundled包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/yarn/exceptions/YarnException
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 7 more

异常原因：

Flink没有找到lib目录的pre-bundled Hadoop或者没有找到Hadoop classpath环境变量

Pre-bundled Hadoop 2.7.5 下载地址

![[bigdata/Flink/企业微信截图_f53d3d3a-f9de-4f8f-899c-e438332893d5.png]]