barrelsofdata/spark-structured-streaming-wordcount
1
0
Fork 0
generated from barrelsofdata/spark-boilerplate
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Project detailing how to write a word count program using spark structured streaming along with unit test cases
1 Commit 1 Branch 0 Tags 162 KiB
Scala 90.1%
Shell 9.9%
62c65fa963
Go to file
karthik 62c65fa963 Initial commit
2020-07-09 13:26:50 +00:00
gradle/wrapper Initial commit 2020-07-09 13:26:50 +00:00
src Initial commit 2020-07-09 13:26:50 +00:00
.gitignore Initial commit 2020-07-09 13:26:50 +00:00
build.gradle Initial commit 2020-07-09 13:26:50 +00:00
gradle.properties Initial commit 2020-07-09 13:26:50 +00:00
gradlew Initial commit 2020-07-09 13:26:50 +00:00
gradlew.bat Initial commit 2020-07-09 13:26:50 +00:00
README.md Initial commit 2020-07-09 13:26:50 +00:00
settings.gradle Initial commit 2020-07-09 13:26:50 +00:00

README.md

Spark Boilerplate

This is a boilerplate project for Apache Spark. The related blog post can be found at https://www.barrelsofdata.com/spark-boilerplate-using-scala

Build instructions

From the root of the project execute the below commands

  • To clear all compiled classes, build and log directories
./gradlew clean
  • To run tests
./gradlew test
  • To build jar
./gradlew shadowJar
  • All combined
./gradlew clean test shadowJar

Run

spark-submit --master yarn --deploy-mode cluster build/libs/spark-boilerplate-1.0.jar