barrelsofdata/spark-boilerplate
Template
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
A boilerplate template for apache spark projects
5 Commits 1 Branch 0 Tags 229 KiB
Scala 100%
fac2f6661e
Go to file Use this template
karthik fac2f6661e - Use native gradle build to generate jar with dependencies, removed shadowJar plugin 
- Upgrade gradle to 6.7.1
2020-11-25 02:35:56 -05:00
gradle/wrapper - Use native gradle build to generate jar with dependencies, removed shadowJar plugin 2020-11-25 02:35:56 -05:00
src Initial commit 2020-06-21 09:56:24 -04:00
.gitignore Initial commit 2020-06-21 09:55:22 -04:00
build.gradle - Use native gradle build to generate jar with dependencies, removed shadowJar plugin 2020-11-25 02:35:56 -05:00
gradle.properties - Use native gradle build to generate jar with dependencies, removed shadowJar plugin 2020-11-25 02:35:56 -05:00
gradlew - Use native gradle build to generate jar with dependencies, removed shadowJar plugin 2020-11-25 02:35:56 -05:00
gradlew.bat - Use native gradle build to generate jar with dependencies, removed shadowJar plugin 2020-11-25 02:35:56 -05:00
README.md - Use native gradle build to generate jar with dependencies, removed shadowJar plugin 2020-11-25 02:35:56 -05:00
settings.gradle Initial commit 2020-06-21 09:55:22 -04:00

README.md

Spark Boilerplate

This is a boilerplate project for Apache Spark. The related blog post can be found at https://www.barrelsofdata.com/spark-boilerplate-using-scala

Build instructions

From the root of the project execute the below commands

  • To clear all compiled classes, build and log directories
./gradlew clean
  • To run tests
./gradlew test
  • To build jar
./gradlew build
  • All combined
./gradlew clean test build

Run

spark-submit --master yarn --deploy-mode cluster build/libs/spark-boilerplate-1.0.jar