barrelsofdata
/
spark-wordcount
generated from barrelsofdata/spark-boilerplate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Project detailing how to write a word count program in spark along with unit test cases
5 Commits 1 Branch 0 Tags 165 KiB
Scala 100%
Go to file
karthik 1d4662b273
Tests / reset-status (push) Successful in 2s Details
Tests / tests (push) Failing after 4m51s Details
Tests / build (push) Has been skipped Details
Use gradle version catalog 		2023-10-07 13:10:15 +02:00
.gitea/workflows Use gradle version catalog 2023-10-07 13:10:15 +02:00
gradle Use gradle version catalog 2023-10-07 13:10:15 +02:00
src Initial commit 2020-07-09 09:21:43 -04:00
.gitignore Use gradle version catalog 2023-10-07 13:10:15 +02:00
README.md Use gradle version catalog 2023-10-07 13:10:15 +02:00
build.gradle.kts Use gradle version catalog 2023-10-07 13:10:15 +02:00
gradle.properties Use gradle version catalog 2023-10-07 13:10:15 +02:00
gradlew Use gradle version catalog 2023-10-07 13:10:15 +02:00
gradlew.bat Use gradle version catalog 2023-10-07 13:10:15 +02:00
settings.gradle.kts Use gradle version catalog 2023-10-07 13:10:15 +02:00

README.md

Tests Build

Spark Word Count with Unit Tests

This is a project detailing how to write word count program in Apache Spark along with unit test cases. The related blog post can be found at https://barrelsofdata.com/spark-word-count-with-unit-tests

Build instructions

From the root of the project execute the below commands

  • To clear all compiled classes, build and log directories
./gradlew clean
  • To run tests
./gradlew test
  • To build jar
./gradlew build

Run

spark-submit --master yarn --deploy-mode cluster build/libs/spark-wordcount-1.0.0.jar hdfs://path/to/input/file.txt hdfs://path/to/output/directory