Skip to content

Big Data

Analytics And More
  • Home
  • Map Reduce
  • Spark
  • Hive
  • Hdfs & Yarn
  • Pig
  • Oozie
  • Hbase
  • Design Patterns
  • streaming

Tag: kafka

submit spark job programmatically using SparkLauncher

March, 2019 adarsh Leave a comment

In this article I will illustrate how to submit a spark job programmatically using SparkLauncher. Let us take a use…

Continue Reading →

Posted in: aws, Spark, stream processing Filed under: kafka, Spark Rdd, spark streaming

kafka example for custom serializer, deserializer and encoder with spark streaming integration

November, 2017 adarsh 1 Comment

Lets say we want to send a custom object as the kafka value type and we need to push this…

Continue Reading →

Posted in: Data Analytics, Spark, stream processing Filed under: kafka, Spark Rdd, spark streaming, streaming

stateful transformation spark streaming example

adarsh Leave a comment

Stateful transformations are operations on DStreams that track data across time that is, some data from previous batches is used…

Continue Reading →

Posted in: Data Analytics, Spark, stream processing Filed under: kafka, Spark Rdd, spark streaming, streaming

stateless transformation spark streaming example

adarsh Leave a comment

Stateless transformations like map(), flatMap(), filter(), repartition(), reduceByKey(), groupByKey() are simple RDD transformations being applied on every batch. Keep in…

Continue Reading →

Posted in: Data Analytics, Spark, stream processing Filed under: kafka, Spark Rdd, spark streaming, streaming

spark streaming example and architecture

adarsh Leave a comment

Spark Streaming provides an abstraction called DStreams, or discretized streams which is build on top of RDD. A DStream is…

Continue Reading →

Posted in: Data Analytics, Spark, stream processing Filed under: kafka, Spark Rdd, spark streaming, streaming

Recent Posts

  • spark sql consecutive sequence example
  • spark sql example to find second highest average
  • spark sql example to find max of average
  • Home
  • Contact Me
  • About Me
Copyright © 2017 Time Pass Techies