barrelsofdata/spark-state-store-data-deduplication

generated from barrelsofdata/spark-boilerplate

Project showing how the spark hdfs backed state store can be used to deduplicate streaming data

Go to file

karthik 9f66e1036c Initial commit		2020-07-19 13:30:42 +00:00
gradle/wrapper	Initial commit	2020-07-19 13:30:42 +00:00
src	Initial commit	2020-07-19 13:30:42 +00:00
.gitignore	Initial commit	2020-07-19 13:30:42 +00:00
build.gradle	Initial commit	2020-07-19 13:30:42 +00:00
gradle.properties	Initial commit	2020-07-19 13:30:42 +00:00
gradlew	Initial commit	2020-07-19 13:30:42 +00:00
gradlew.bat	Initial commit	2020-07-19 13:30:42 +00:00
README.md	Initial commit	2020-07-19 13:30:42 +00:00
settings.gradle	Initial commit	2020-07-19 13:30:42 +00:00

README.md

Spark Boilerplate

This is a boilerplate project for Apache Spark. The related blog post can be found at https://www.barrelsofdata.com/spark-boilerplate-using-scala

Build instructions

From the root of the project execute the below commands

To clear all compiled classes, build and log directories

./gradlew clean

To run tests

./gradlew test

To build jar

./gradlew shadowJar

All combined

./gradlew clean test shadowJar

Run

spark-submit --master yarn --deploy-mode cluster build/libs/spark-boilerplate-1.0.jar