大数据 jvm 【Flink 核心篇】Flink 内存管理（四）：TaskManager 内存分配（实战篇）

《Flink 内存管理》系列（已完结），共包含以下 4 篇文章：

Flink 内存管理（一）：设置 Flink 进程内存Flink 内存管理（二）：JobManager 内存分配（含实际计算案例）Flink 内存管理（三）：TaskManager 内存分配（理论篇）Flink 内存管理（四）：TaskManager 内存分配（实战篇）

 如果您觉得这篇文章有用 ✔️ 的话，请给博主一个一键三连  吧 （点赞 李、关注 、收藏 ）！！！您的支持  将激励  博主输出更多优质内容！！！

Flink 内存管理（四）：TaskManager 内存分配（实战篇）

1.单独分配 Total Process Size2.单独分配 Total Flink Size3.单独分配 Heap Size && Managed Memory4.分配 Total Process Size 和 Heap Size && Managed Memory5.分配 Total Flink Size 和 Heap Size && Managed Memory6.内存分配小结

在 《Flink 内存管理（一）：设置 Flink 进程内存》中我们提到，必须使用下述三种方法之一配置 Flink 的内存（本地执行除外），否则 Flink 启动将失败。这意味着必须明确配置以下选项子集之一，这些子集没有默认值。

序号for TaskManagerfor JobManager1️⃣taskmanager.memory.flink.sizejobmanager.memory.flink.size2️⃣taskmanager.memory.process.sizejobmanager.memory.process.size3️⃣taskmanager.memory.task.heap.size 和 taskmanager.memory.managed.sizejobmanager.memory.heap.size

1.单独分配 Total Process Size

单独分配 Total Process Size，其它的组件都会自动分配。

taskmanager.memory.process.size: 2000m

内存分配步骤如下：

首先 Total Process Size

=

2000

M

= 2000M

=2000M因为没有显示分配组件中的任何参数，所以 JVM Overhead

=

2000

M

×

0.1

=

200

M

= 2000M × 0.1 = 200M

=2000M×0.1=200MJVM Metaspace

=

256

M

= 256M

=256M⭐ 所以 Native Memory = JVM Overhead + JVM Metaspace

=

456

M

= 456M

=456MTotal Flink Size

=

2000

M

−

200

M

−

256

M

=

1544

M

B

=

1.508

G

B

= 2000M - 200M - 256M = 1544MB = 1.508GB

=2000M−200M−256M=1544MB=1.508GBNetwork Memory

=

1544

×

0.1

=

154.4

M

= 1544 × 0.1 = 154.4M

=1544×0.1=154.4MTask Off-Heap

=

=

=

0

M

B

0MB

0MB（默认）Framework Off-Heap

=

=

=

128

M

128M

128M（默认）⭐ 所以 Total Direct Memory

=

154.4

M

+

0

+

128

M

=

282.4

M

= 154.4M + 0 + 128M = 282.4M

=154.4M+0+128M=282.4MManaged Memory

=

1544

M

B

×

0.4

=

617.6

M

= 1544MB × 0.4 = 617.6M

=1544MB×0.4=617.6MTotal JVM Heap Memory

=

1544

M

−

282.4

M

−

617.6

M

=

644

M

B

= 1544M - 282.4M - 617.6M = 644MB

=1544M−282.4M−617.6M=644MBFramework Heap

=

128

M

= 128M

=128MTask Heap

=

644

M

−

128

M

=

516

M

= 644M - 128M = 516M

=644M−128M=516M

可以与以下的日志进行对比，完全能对上，！

2.单独分配 Total Flink Size

taskmanager.memory.flink.size: 2000m

假如直接只分配 taskmanager.memory.flink.size: 2000m

Total Flink Size

=

2000

M

= 2000M

=2000MManaged Memory

=

2000

M

×

0.4

=

800

M

= 2000M × 0.4 = 800M

=2000M×0.4=800MNetWork Memory

=

2000

M

×

0.1

=

200

M

= 2000M × 0.1 = 200M

=2000M×0.1=200MFramework Off-Heap

=

128

M

= 128M

=128MTask Off-Heap

=

0

B

y

t

e

=

0

M

= 0Byte = 0M

=0Byte=0M⭐ 所以 Total Direct Memory

=

200

M

+

128

M

+

0

M

=

328

M

= 200M + 128M + 0M= 328M

=200M+128M+0M=328MTotal Off-Heap Memory

=

800

M

+

328

M

=

1128

M

= 800M + 328M = 1128M

=800M+328M=1128MTotal JVM Heap

=

2000

M

−

800

M

−

328

M

=

872

M

= 2000M - 800M - 328M = 872M

=2000M−800M−328M=872MFramework Heap

=

128

M

= 128M

=128MTask Heap

=

872

M

−

128

M

=

744

M

= 872M - 128M = 744M

=872M−128M=744MJVM MetaSpace

=

256

M

= 256M

=256M（默认）JVM Overhead

=

(

= (

=(JVM Overhead

+

256

M

+\ 256M

+ 256M Metaspace

+

2000

M

+\ 2000M

+ 2000M Total Flink Size

)

×

0.1

) × 0.1

)×0.1，求解 JVM Overhead

=

250.667

M

= 250.667M

=250.667M 在

192

M

B

～

1

G

B

192MB ～ 1GB

192MB～1GB，生效Total Process Size

=

2000

M

+

256

M

+

250.667

M

=

2506.667

M

=

2.448

G

B

= 2000M + 256M + 250.667M = 2506.667M = 2.448GB

=2000M+256M+250.667M=2506.667M=2.448GB

3.单独分配 Heap Size && Managed Memory

taskmanager.memory.task.heap.size: 1000m

taskmanager.memory.managed.size: 1000m

Framework Heap

=

128

M

= 128M

=128M（默认）Task Heap

=

1000

M

= 1000M

=1000M（配置）Total JVM Heap

=

1000

M

+

128

M

=

1128

M

=

1.102

G

B

= 1000M + 128M = 1128M = 1.102GB

=1000M+128M=1128M=1.102GBManaged Memory

=

1000

M

= 1000M

=1000M（配置）Framework Off-Heap

=

128

M

= 128M

=128MTask Off-Heap

=

0

M

= 0M

=0MNetWork

=

=

= Total Flink Size

×

0.1

×\ 0.1

× 0.1

=

(

= (

=(NetWork

+

1128

M

+

1000

M

+

128

M

+

0

M

)

×

0.1

+\ 1128M + 1000M + 128M + 0M) × 0.1

+ 1128M+1000M+128M+0M)×0.1，计算得到 Network

=

250.667

M

B

= 250.667MB

=250.667MB，处于

64

M

B

～

1

G

B

64MB ～ 1GB

64MB～1GB，有效⭐ 所以 Total Direct Memory

=

128

M

+

250.667

M

=

378.667

M

= 128M + 250.667M = 378.667M

=128M+250.667M=378.667MTotal Flink Size

=

1128

M

+

1378.667

M

=

2506.667

M

=

2.448

G

B

= 1128M + 1378.667M = 2506.667M = 2.448GB

=1128M+1378.667M=2506.667M=2.448GBJVM Metaspace

=

256

M

= 256M

=256M（默认）JVM Overhead

=

(

= (

=(JVM Overhead

+

1128

M

+

1000

M

+

378.667

M

+

256

M

)

×

0.1

=

306.963

M

+\ 1128M + 1000M + 378.667M + 256M) × 0.1 = 306.963M

+ 1128M+1000M+378.667M+256M)×0.1=306.963M，处于

192

M

～

1

G

B

192M ～ 1GB

192M～1GB，有效Total Process Size

=

2506.667

M

+

256

M

+

306.963

M

=

3069.63

M

=

2.998

G

= 2506.667M + 256M + 306.963M = 3069.63M = 2.998G

=2506.667M+256M+306.963M=3069.63M=2.998G

4.分配 Total Process Size 和 Heap Size && Managed Memory

指定 Total Process Size，同时显式分配组件 JVM Heap 和 Mamaged Memory。

taskmanager.memory.process.size: 3000m

taskmanager.memory.task.heap.size: 1000m

taskmanager.memory.managed.size: 1000m

Total Process Size

=

3000

M

= 3000M

=3000MFramework Heap

=

128

M

= 128M

=128M（默认）Task Heap

=

1000

M

= 1000M

=1000M（配置）Total JVM Heap

=

=

= Framework Heap

+

+

+ Task Heap

=

128

M

+

1000

M

=

1128

M

=

1.102

G

= 128M + 1000M = 1128M = 1.102G

=128M+1000M=1128M=1.102GManaged Memory

=

1000

M

= 1000M

=1000M（配置）Framework Off-Heap

=

128

M

= 128M

=128M（默认）Task Off-Heap

=

0

M

= 0M

=0M（默认）Network Memory

=

(

= (

=(Network Memory

+

1128

M

+

1128

M

)

×

0.1

=

250.667

M

+\ 1128M + 1128M) × 0.1 = 250.667M

+ 1128M+1128M)×0.1=250.667M，在

64

M

～

1

G

B

64M ～ 1GB

64M～1GB 之间，满足要求Total Off-Heap

=

1000

M

+

128

M

+

250.667

M

+

0

M

=

1378.667

M

=

1.346

G

B

= 1000M + 128M + 250.667M + 0M = 1378.667M = 1.346GB

=1000M+128M+250.667M+0M=1378.667M=1.346GBTotal Flink Size

=

1128

M

+

1378.667

M

=

2506.667

M

=

2.448

G

B

= 1128M + 1378.667M = 2506.667M = 2.448GB

=1128M+1378.667M=2506.667M=2.448GBJVM Metaspace

=

256

M

= 256M

=256MJVM Overhead

=

3000

M

−

2506.667

M

−

256

M

=

237.333

M

= 3000M - 2506.667M - 256M = 237.333M

=3000M−2506.667M−256M=237.333M，在

192

M

～

1

G

B

192M ～ 1GB

192M～1GB 之间，满足要求

5.分配 Total Flink Size 和 Heap Size && Managed Memory

指定 Total Flink Size，同时显式分配组件 JVM Heap 和 Mamaged Memory。

taskmanager.memory.flink.size: 3000m

taskmanager.memory.task.heap.size: 1000m

taskmanager.memory.managed.size: 1000m

Total Flink Size

=

3000

M

=

2.93

G

B

= 3000M = 2.93GB

=3000M=2.93GB（配置）Managed Memory

=

1000

M

= 1000M

=1000M（配置）Task Heap

=

1000

M

= 1000M

=1000M（配置）Framework Heap

=

128

M

= 128M

=128M（默认）Total JVM Heap

=

=

= Framework Heap + Task Heap

=

128

M

+

1000

M

=

1128

M

= 128M + 1000M =1128M

=128M+1000M=1128MTotal Off-Heap Memory

=

3000

M

−

1128

M

=

1872

M

=

1.828

G

B

= 3000M - 1128M = 1872M = 1.828GB

=3000M−1128M=1872M=1.828GBDirect Memory

=

=

= Total Off-Heap Memory - Managed Memory

=

1872

M

−

1000

M

=

872

M

= 1872M - 1000M = 872M

=1872M−1000M=872MTask Off-Heap

=

0

M

= 0M

=0M（默认）Framework Off-Heap

=

128

M

= 128M

=128M（默认）Network Memory

=

=

= Direct Memory

−

-

− Task Off-Heap - Framework Off-Heap

=

872

M

−

0

M

−

128

M

=

744

M

= 872M - 0M - 128M = 744M

=872M−0M−128M=744MJVM Metaspace

=

256

M

= 256M

=256M（默认）JVM Overhead

=

(

= (

=(JVM Overhead

+

3000

M

+

256

M

)

×

0.1

+\ 3000M + 256M) × 0.1

+ 3000M+256M)×0.1，计算得到 JVM Overhead

=

361.778

M

= 361.778M

=361.778M，处于

192

M

～

1

G

192M～1G

192M～1G 之间，符合条件Total Process Size

=

3000

M

+

256

M

+

361.778

M

=

3617.778

M

=

3.533

G

B

= 3000M + 256M + 361.778M = 3617.778M = 3.533GB

=3000M+256M+361.778M=3617.778M=3.533GB

6.内存分配小结

在 Flink 的集群内存分配的过程中，我们大致可以通过

3

3

3 种方式进行分配。

指定 Total Process Size 或 Total Flink Size，取决于你用什么方式部署。单独指定某个组件，比如 Task-Heap 的大小，其它的组件都会被推导出来。指定 Total Process / Flink Size && Heap or Off-Heap 其中之一，其它的组件通过默认值进行填充或者进推导，如：

Total Flink Size = Total Heap Size + Total Off-Heap SizeTotal Heap Size = Task Heap + Framework HeapTotal Off-Heap = Task Off-Heap + Framework Off-Heap + Network Memory + Managed MemoryNetwork = Total Flink Size

×

0.1

×\ 0.1

× 0.1（没有指定其它组件情况下）JVM Overhead = Total Process Size

×

0.1

×\ 0.1

× 0.1（没有指定其它组件情况下）… …

文章来源