Skip to content

Instantly share code, notes, and snippets.

@kaniska

kaniska/spark-scala-minpricedproductbycategory.scala

Forked from dgadiraju/spark-scala-minpricedproductbycategory.scala
Created July 29, 2018 07:23
Show Gist options
  • Save kaniska/3ca773f2c7f97034b18979995e11ba1a to your computer and use it in GitHub Desktop.
Save kaniska/3ca773f2c7f97034b18979995e11ba1a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark-scala-minpricedproductbycategory.scala
val path = "/public/retail_db"
val products = sc.textFile(path + "/products")
val minPricedProductsByCategory = products.
filter(product => product.split(",")(4) != "").
map(product => {
val p = product.split(",")
(p(1).toInt, product)
}).
reduceByKey((agg, product) => {
if(agg.split(",")(4).toFloat < product.split(",")(4).toFloat)
agg
else
product
}).
map(rec => rec._2)
minPricedProductsByCategory.collect.foreach(println)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment