barrelsofdata/spark-wordcount
1
0
Fork 0
generated from barrelsofdata/spark-boilerplate
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Project detailing how to write a word count program in spark along with unit test cases
1 Commit 1 Branch 0 Tags 165 KiB
Scala 100%
f60539d7c4
Go to file
karthik f60539d7c4 Initial commit
2020-06-21 14:03:19 +00:00
gradle/wrapper Initial commit 2020-06-21 14:03:19 +00:00
src Initial commit 2020-06-21 14:03:19 +00:00
.gitignore Initial commit 2020-06-21 14:03:19 +00:00
build.gradle Initial commit 2020-06-21 14:03:19 +00:00
gradle.properties Initial commit 2020-06-21 14:03:19 +00:00
gradlew Initial commit 2020-06-21 14:03:19 +00:00
gradlew.bat Initial commit 2020-06-21 14:03:19 +00:00
README.md Initial commit 2020-06-21 14:03:19 +00:00
settings.gradle Initial commit 2020-06-21 14:03:19 +00:00

README.md

Spark Boilerplate

This is a boilerplate project for Apache Spark. The related blog post can be found at https://www.barrelsofdata.com/spark-boilerplate-using-scala

Build instructions

From the root of the project execute the below commands

  • To clear all compiled classes, build and log directories
./gradlew clean
  • To run tests
./gradlew test
  • To build jar
./gradlew shadowJar
  • All combined
./gradlew clean test shadowJar

Run

spark-submit --master yarn --deploy-mode cluster build/libs/spark-boilerplate-1.0.jar