barrelsofdata/spark-boilerplate
Template
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
A boilerplate template for apache spark projects
4 Commits 1 Branch 0 Tags 229 KiB
Scala 100%
b982d4c30a
Go to file Use this template
karthik b982d4c30a Initial commit
2020-07-04 10:56:43 -04:00
gradle/wrapper Initial commit 2020-07-04 10:56:43 -04:00
src Initial commit 2020-06-21 09:56:24 -04:00
.gitignore Initial commit 2020-06-21 09:55:22 -04:00
build.gradle Initial commit 2020-06-21 09:55:22 -04:00
gradle.properties Initial commit 2020-06-21 09:55:22 -04:00
gradlew Initial commit 2020-06-21 09:55:22 -04:00
gradlew.bat Initial commit 2020-06-21 09:55:22 -04:00
README.md Initial commit 2020-06-21 09:56:24 -04:00
settings.gradle Initial commit 2020-06-21 09:55:22 -04:00

README.md

Spark Boilerplate

This is a boilerplate project for Apache Spark. The related blog post can be found at https://www.barrelsofdata.com/spark-boilerplate-using-scala

Build instructions

From the root of the project execute the below commands

  • To clear all compiled classes, build and log directories
./gradlew clean
  • To run tests
./gradlew test
  • To build jar
./gradlew shadowJar
  • All combined
./gradlew clean test shadowJar

Run

spark-submit --master yarn --deploy-mode cluster build/libs/spark-boilerplate-1.0.jar