Jobs
Tune memory, shuffle and file sizes before production.
What you will be able to do
- Cache reused DataFrames.
- Write partitioned parquet.
- Watch the Spark UI.
In plain English
Tune memory, shuffle and file sizes before production.
- Cache reused DataFrames.
- Write partitioned parquet.
- Watch the Spark UI.
df.write.mode('overwrite').partitionBy('course').parquet('out/lessons')Your turn
EasyExplain it back
Answer from memory: Which file format is columnar and common in Spark?
TEXT · starter
Your answer…Check your understanding before moving on.
Knowledge checkQuestion 1
Which file format is columnar and common in Spark?
Select an answer to continue
Study tools
Make it yours.
Keep a private note for this lesson, then turn the key ideas into flashcards you can revisit.
Sign in to save notes and flashcards.Both are private to your account and stay with your progress.
Sign inFinished reading?
Lock in what you just learned.
0%
Mark this lesson complete to save your place and move one step closer to the certificate.