Guadazi
0%

Flink savepoint恢复过程解析

Posted on Edited on In

Flink savepoint恢复

运行日志分析

正常的恢复

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.jobmaster.JobMaster                 :236 - Initializing job t_org_trade_day_dal_statistics_job (000000000000eace000000000000001c).
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.jobmaster.JobMaster                 :164 - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=10000, delayBetweenRestartAttempts=20000) for t_org_trade_day_dal_statistics_job (000000000000eace000000000000001c).
[flink-akka.actor.default-dispatcher-4] org.apache.flink.yarn.YarnResourceManager                    :238 - Recovered 0 containers from previous attempts ([]).
[flink-akka.actor.default-dispatcher-4] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy :81 - yarn.client.max-cached-nodemanagers-proxies : 0
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.executiongraph.ExecutionGraph       :528 - Job recovers via failover strategy: full graph restart
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.jobmaster.JobMaster                 :203 - Running initialization on master for job t_org_trade_day_dal_statistics_job (000000000000eace000000000000001c).
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.jobmaster.JobMaster                 :221 - Successfully ran initialization on master in 0 ms.
[flink-akka.actor.default-dispatcher-4] org.apache.flink.yarn.YarnResourceManager                    :1029 - ResourceManager akka.tcp://flink@9.44.10.157:35473/user/resourcemanager was granted leadership with fencing token 92d59c139e046177dbed78af07b744e8
[main-EventThread] org.apache.flink.runtime.leaderservice.zookeeper.ZooKeeperLeaderElectionService :262 - org.apache.flink.yarn.YarnResourceManager@83ab9bf has been granted leadership.
[flink-akka.actor.default-dispatcher-4] org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl :215 - Starting the SlotManager.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.util.ZooKeeperUtils                 :301 - Initialized ZooKeeperCompletedCheckpointStore in '/checkpoints/000000000000eace000000000000001c'.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.jobmaster.JobMaster                 :200 - Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'hdfs://qy-flink-1-v3/user/u_teg_tdbank/oceanus2/lj_cft_flink/fit-oceanus-prod/project-10001/job-60110/snapshots', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=UNDEFINED, numberOfTransferingThreads=-1}
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.jobmaster.JobMaster                 :207 - Configuring application-defined state backend with job/cluster config
[flink-akka.actor.default-dispatcher-2] org.apache.flink.contrib.streaming.state.RocksDBStateBackend :348 - Using predefined options: DEFAULT.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.contrib.streaming.state.RocksDBStateBackend :563 - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :128 - Recovering checkpoints from ZooKeeper.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :146 - Found 0 checkpoints in ZooKeeper.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :163 - Trying to fetch 0 checkpoints from storage.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :1124 - Starting job 000000000000eace000000000000001c from savepoint hdfs://qy-flink-1-v3/user/u_teg_tdbank/oceanus2/lj_cft_flink/fit-oceanus-prod/project-10001/job-60110/snapshots/savepoint-000000-a5c820c4c53a (allowing non restored state)
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :1139 - Reset the checkpoint ID of job 000000000000eace000000000000001c to 370.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :128 - Recovering checkpoints from ZooKeeper.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :146 - Found 1 checkpoints in ZooKeeper.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :163 - Trying to fetch 1 checkpoints from storage.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :336 - Trying to retrieve checkpoint 369.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :1079 - Restoring job 000000000000eace000000000000001c from latest valid checkpoint: Checkpoint 369 @ 0 for 000000000000eace000000000000001c.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :234 - No master state to restore
[main-EventThread] org.apache.flink.runtime.jobmaster.JobManagerRunner          :298 - JobManager runner for job t_org_trade_day_dal_statistics_job (000000000000eace000000000000001c) was granted leadership with session id 3beb8d55-f7c2-4175-9c7e-ac256f03e8bb at akka.tcp://flink@9.44.10.157:35473/user/jobmanager_0.
[main-EventThread] org.apache.flink.runtime.leaderservice.zookeeper.ZooKeeperLeaderElectionService :262 - org.apache.flink.runtime.jobmaster.JobManagerRunner@548d10c7 has been granted leadership.

JobGraph修改后恢复失败

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.jobmaster.JobMaster                 :236 - Initializing job flake-sql-job-runner (000000000000eafb0000000000000011).
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.jobmaster.JobMaster                 :164 - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=10000, delayBetweenRestartAttempts=20000) for flake-sql-job-runner (000000000000eafb0000000000000011).
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.executiongraph.ExecutionGraph       :528 - Job recovers via failover strategy: full graph restart
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.jobmaster.JobMaster                 :203 - Running initialization on master for job flake-sql-job-runner (000000000000eafb0000000000000011).
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.jobmaster.JobMaster                 :221 - Successfully ran initialization on master in 0 ms.
[flink-akka.actor.default-dispatcher-4] org.apache.flink.yarn.YarnResourceManager                    :238 - Recovered 0 containers from previous attempts ([]).
[flink-akka.actor.default-dispatcher-4] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy :81 - yarn.client.max-cached-nodemanagers-proxies : 0
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.util.ZooKeeperUtils                 :301 - Initialized ZooKeeperCompletedCheckpointStore in '/checkpoints/000000000000eafb0000000000000011'.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.jobmaster.JobMaster                 :200 - Using application-defined state backend: 
RocksDBStateBackend{
checkpointStreamBackend=
   File State Backend (
      checkpoints: 'hdfs://qy-flink-1-v3/user/u_teg_tdbank/oceanus2/lj_cft_flink/fit-oceanus-prod/project-10001/job-60155/snapshots', 
      savepoints: 'null', 
      asynchronous: UNDEFINED, 
      fileStateThreshold: -1
     ), 
localRocksDbDirectories=null, 
enableIncrementalCheckpointing=UNDEFINED, 
numberOfTransferingThreads=-1
}
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.jobmaster.JobMaster                 :207 - Configuring application-defined state backend with job/cluster config
[flink-akka.actor.default-dispatcher-3] org.apache.flink.contrib.streaming.state.RocksDBStateBackend :348 - Using predefined options: DEFAULT.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.contrib.streaming.state.RocksDBStateBackend :563 - Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[flink-akka.actor.default-dispatcher-4] org.apache.flink.yarn.YarnResourceManager                    :1029 - ResourceManager akka.tcp://flink@9.44.34.108:40622/user/resourcemanager was granted leadership with fencing token 9e26d17ff1120b75114052d13be242e0
[main-EventThread] org.apache.flink.runtime.leaderservice.zookeeper.ZooKeeperLeaderElectionService :262 - org.apache.flink.yarn.YarnResourceManager@1a74bd90 has been granted leadership.
[flink-akka.actor.default-dispatcher-4] org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl :215 - Starting the SlotManager.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :128 - Recovering checkpoints from ZooKeeper.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :146 - Found 0 checkpoints in ZooKeeper.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :163 - Trying to fetch 0 checkpoints from storage.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :1124 - Starting job 000000000000eafb0000000000000011 from savepoint hdfs://qy-flink-1-v3/user/u_teg_tdbank/oceanus2/lj_cft_flink/fit-oceanus-prod/project-10001/job-60155/snapshots/savepoint-000000-630ca6fb8b9c (allowing non restored state)
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.Checkpoints              :172 - Could not find ExecutionJobVertex. Including user-defined OperatorIDs in search.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.Checkpoints              :194 - Skipping savepoint state for operator 749b2cad71aaee5be25025871420d4b2.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.Checkpoints              :194 - Skipping savepoint state for operator f7ad6dac6ff3d8ada68fde148d943256.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.Checkpoints              :194 - Skipping savepoint state for operator 5c72ff74f7b814cc2b15db0311cc0aaf.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.Checkpoints              :194 - Skipping savepoint state for operator d6883e647f8d8dd07dbe048d4037223c.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.Checkpoints              :194 - Skipping savepoint state for operator bb6ff7b6d8a06dd49e1764b59667cf62.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :1139 - Reset the checkpoint ID of job 000000000000eafb0000000000000011 to 5.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :128 - Recovering checkpoints from ZooKeeper.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :146 - Found 1 checkpoints in ZooKeeper.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :163 - Trying to fetch 1 checkpoints from storage.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore :336 - Trying to retrieve checkpoint 4.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :1079 - Restoring job 000000000000eafb0000000000000011 from latest valid checkpoint: Checkpoint 4 @ 0 for 000000000000eafb0000000000000011.
[flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    :234 - No master state to restore
[main-EventThread] org.apache.flink.runtime.jobmaster.JobManagerRunner          :298 - JobManager runner for job flake-sql-job-runner (000000000000eafb0000000000000011) was granted leadership with session id 9ecd11e9-590a-4186-8270-edff06b7a810 at akka.tcp://flink@9.44.34.108:40622/user/jobmanager_0.
[main-EventThread] org.apache.flink.runtime.leaderservice.zookeeper.ZooKeeperLeaderElectionService :262 - org.apache.flink.runtime.jobmaster.JobManagerRunner@3da1e67a has been granted leadership.
[flink-akka.actor.default-dispatcher-2] org.apache.flink.runtime.jobmaster.JobMaster                 :691 - Starting execution of job flake-sql-job-runner (000000000000eafb0000000000000011) under job master id 8270edff06b7a8109ecd11e9590a4186.