Partitions
Partitioning controls parallelism and shuffle cost.
What you will be able to do
- Use repartition to increase.
- Use coalesce to reduce.
- Avoid tiny partitions.
In plain English
Partitioning controls parallelism and shuffle cost.
- Use repartition to increase.
- Use coalesce to reduce.
- Avoid tiny partitions.
df = df.repartition(8, 'course')
print(df.rdd.getNumPartitions())Your turn
EasyExplain it back
Answer from memory: Which method reduces partitions without a full shuffle?
TEXT · starter
Your answer…Check your understanding before moving on.
Knowledge checkQuestion 1
Which method reduces partitions without a full shuffle?
Select an answer to continue
Study tools
Make it yours.
Keep a private note for this lesson, then turn the key ideas into flashcards you can revisit.
Sign in to save notes and flashcards.Both are private to your account and stay with your progress.
Sign inFinished reading?
Lock in what you just learned.
0%
Mark this lesson complete to save your place and move one step closer to the certificate.