RDD In Python

Create RDD with parallelize
a = range(100) data = sc.parallelize(a) data.count()
data.take(10)

Parallelized Collections
data= [('JK', 22), ('V', 24), ('Jimin',24), ('RM', 25), ('J-Hope', 25), ('Suga', 26), ('Jin', 27)]
myRDD = sc.parallelize(data)
myRDD.take(7)

From File
rdd = sc.textFile("/FileStore/tables/test.txt")
rdd.take(1)
rdd.count()

def Func(lines):
  lines = lines.upper()
  lines = lines.split()
  return lines
Split_rdd = rdd.map(Func)
Split_rdd.take(5)

No comments:

Post a Comment