Shuffl Hash Join

The default implementation of join in spark is a shuffle hash join.When the size of the table is relatively small, we choose to broadcast it out to avoid shuffle, improve performance. But because the broadcast table is first to collect to the driver segment, and then distributed to each executor redundant, so when the table is relatively large, the use of broadcast progress will be the driver and executor side caused greater pressure.But because Spark is a distributed computing engine, you can partition the large number of data can be divided into n smaller data sets for parallel computing. This idea is applied to the Join is Shuffle Hash Join.
 
The shuffled hash join ensures that data on each partition will contain the same keys by partitioning the second dataset with the same default partitioner as the first, so that the keys with the same hash value from both datasets are in the same partition.The average size of the partition does not exceed the value configured by spark.sql.autoBroadcastJoinThreshold, the default is 10MB.

It follows the classic map-reduce pattern:

  • First it maps through two tables(dataframes)
  • Uses the join keys as output key
  • Shuffles both dataframes by the output key, So that rows related to same keys from both tables will be moved on to same machine.
  • In reducer phase, join the two datasets.

Shuffle hash join works well-

  • When the dataframe are distributed evenly with the keys you are used to join.
  • When dataframes has enough number of keys for parallelism.

While this approach always works, it can be more expensive than necessary because it requires a shuffle.Only supported for equi-joins, while the join keys do not need to be sortable. Supported for all join types except full outer joins.

No comments:

Post a Comment