展开查看详情
1 .WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
2 .Automating performance
tuning for Apache Spark
Jean-Yves Stephan & Julien Dumazert,
Founders of Data Mechanics
#UnifiedDataAnalytics #SparkAISummit
3 .What is performance tuning?
Cluster parameters Spark configurations
● Size ● Parallelism
● Instance type ● Shuffle
● # of processors ● Storage
● # of memory ● JVM tuning
● Disks ● Feature flags
● ... ● ...
3
4 .Why automate performance tuning?
Pick new
params
Run Analyze
the job logs
4
5 .Why automate performance tuning?
Pick new
Hard manual work params
30% of your engineers time
Frequent outages
Pager ringing at 3am
Run Analyze
Slow and expensive the job logs
Missing SLAs every week
5
6 .Agenda
Manual Automated
performance tuning performance tuning
6
7 .Hands-on performance
tuning
8 .Perf tuning is an iterative process
For the first run
There are rules of thumb for some params:
• # of partitions: 3x the number of cores in the cluster
• # of cores per executor: 4-8
• memory per executor: 85% * (node memory / #
executors by node)
For the other params: make an educated guess!
8
9 .Perf tuning is an iterative process
On the first attempt, the job crashes or does not meet the SLA. What to do?
Pick new
params • Ensure stability of the job
• Solve performance issues
Run Analyze
the job logs • Adjust speed-cost trade-off
9
10 . Common issues: lack of parallelism
Configuration:
• 26 instances n1-highmem-16 Only 8 cores used
• spark.executor.cores = 16 on each machine!
10
11 . Common issues: lack of parallelism
Configuration:
• 26 instances n1-highmem-16 Only 8 cores used
• spark.executor.cores = 16 on each machine!
The reason: Fix: Use 400+ partitions (duration and cost / 2)
• 26 executors
• spark.sql.shuffle.partitions = 200 See also Adaptive execution (SPARK-23128) for
→ 200 / 26 ~ 7.7 tasks per executor a way to dynamically and automatically set the
number of partitions.
11
12 . Common issues: shuffle spill
The deserialized data produced by the map Fixes:
stages in a shuffle does not fit in memory. 1. Reduce the input data of each task by
increasing the number of partitions
Spark temporarily writes it to disk, which
degrades performance. 2. Increase the memory available to each task
– by increasing spark executor memory
– by decreasing the number of cores per
executor
12
13 . Common issues: data skew
This issue is not addressable with parameter tuning. A change in code is required!
Change in code:
1. Find a better partition key if possible
2. Use a map-side (broadcast) join
3. Use a salted key
13
14 . Improvements based on node metrics
● Low CPU Usage => Consider oversubscription, ie telling Spark to schedule say 2x more
tasks per executor than the number of cores
● Low Memory Usage => Consider pruning the excess memory and switching to
CPU-intensive instances
● IO bound queries => Consider switching instance type or Spark IO configurations such as
compression or IO buffer sizes
14
15 .Cost-speed trade-off
On the efficient frontier:
cheaper ⇒ longer
Solving performance shorter ⇒ more expensive
issues
Once performance issues are
Efficient frontier
solved, adjust your trade-off
given your needs.
Adjusting
cost-speed trade-off
15
16 .Cost-speed trade-off
Example: impact of # of instances
40 instances
10 instances
4 instances
16
17 .Recap: manual perf tuning
Iterative process: Most of the impact comes from a
few parameters:
Solve performance
issues • # and type of instances for
execs and driver
• executor and driver size
(memory, # of cores)
Adjust cost-speed • # of partitions
tradeoff
17
18 .Open source tuning tools
To detect performance issues: DrElephant (LinkedIn)
To simulate cost-speed trade-off: SparkLens
(only supports adjusting # of executors)
18
19 .Automated performance
tuning
20 .Motivations
Performance tuning can make periodic workloads 2x faster
and more stable.
Pick new
params
But:
• tedious manual process
• requires expertise
Run Analyze
the job logs
→ to scale it to 100+ pipelines, automation is required!
20
21 .Architecture (tech)
Customer Data Mechanics
code Kubernetes cluster
Optimization
engine Job history
2) Optimization engine 4) An agent exports
identifies job from history event logs and system
1) Unoptimized Spark job and returns config metrics during job run
description Spark
listener
Scheduler Gateway Spark job
3) Optimized Spark
job description
21
22 .Architecture (algo)
Jun 2nd Jun 3rd Jun 4th Jun 5th Jun 6th Jun 7th
23
23 .Architecture (algo)
Jun 2nd Jun 3rd Jun 4th Jun 5th Jun 6th Jun 7th
Spark events log and metrics
Heuristic Heuristic
A B
24
24 . Architecture (algo)
Jun 2nd Jun 3rd Jun 4th Jun 5th Jun 6th Jun 7th
Spark events log and metrics
Heuristic Heuristic
A B
Param set B2
Evaluator
Evaluators leverage B
historical data
Param set A1
Param set A3
Evaluator
A
25
25 . Architecture (algo)
Jun 2nd Jun 3rd Jun 4th Jun 5th Jun 6th Jun 7th
Spark events log and metrics
Heuristic Heuristic Experiment
A B manager
Param set B2
Evaluator
Evaluators leverage B
historical data Evaluated Evaluated
Param set A1 param set B2 param set A1
Param set A3
Evaluator
A
26
26 . Architecture (algo)
Jun 2nd Jun 3rd Jun 4th Jun 5th Jun 6th Jun 7th
Spark events log and metrics
Optimistic best
param set
Heuristic Heuristic Experiment
A B manager
Param set B2
Evaluator
Evaluators leverage B
historical data Evaluated Evaluated
Param set A1 param set B2 param set A1
Param set A3
Evaluator
A
27
27 . Heuristics
Heuristics look for performance issues: Or push in a given direction:
• FewerTasksThanTotalCores • IncreaseDefaultNumPartitions
• ShuffleSpill • IncreaseTotalCores
• LongShuffleReadTime • …
• LongGC Jun 6th
• ExecMemoryLargerThanNeeded
• TooShortTasks Spark events log and metrics
• CPUTimeShorterThanComputeTime
• … Heuristic Heuristic
A B
Every heuristic proposes different param sets.
Param sets
28
28 . Heuristics example
FewerTasksThanTotalCores
If a stage has fewer tasks than the total number of cores:
1. Increase the default number of partitions if applicable
2. Decrease the number of instances
Jun 6th
3. Decrease the # of cores per instance (and adjust Spark events log and metrics
memory)
Heuristic Heuristic
A B
Param sets
29
29 .Ranking param sets Unevaluated param sets
Evaluator Evaluator
A B
Param sets cost and duration are evaluated by
Evaluators.
The Experiment manager selects the best solution Experiment
according to customer settings. manager
Optimistic best
param set
30
热门城市
