Useful git commands

List all branches: git branchList all remote branches: git branch -aaCreate the branch on your local machine and switch to this branch: git checkout -b <branch name>Create a new branch: git branch <branch name>Rename current branch: git branch -m <new branch name>Rename other branch: git branch -m <old branch name> <new branch name>Delete a branch: … Continue reading Useful git commands

Kafka Connect Concepts & Example

Kafka Connect is used to connect Kafka with external systems such as database, storage, and messaging systems for data import and export in a scalable and reliable way. The framework executes so-called "connectors" that implement the actual logic to read/write data from other systems. Example: In an organization, we have a custom application for customer … Continue reading Kafka Connect Concepts & Example

Key Concepts of Stream Processing

Stream processing is similar to any data processing where you read the data ,apply some transformation and then push it somewhere. However there are 3 key concepts that are unique to stream processing Time Time is the most important concept in stream processing and therefore it is important to have a common understanding of time … Continue reading Key Concepts of Stream Processing

Kafka Streams

Kafka Streams is a client library (API) to build applications that analyze and process data stored in Kafka in real-time. Streams application takes input from a Kafka topic and stores the output also in a Kafka topic and a stream is an ordered, replay-able, and fault-tolerant sequence of immutable data records, where a data record is defined … Continue reading Kafka Streams

SQL Vs No-SQL

Relational databases store data in table - rows and columns. Each row contains all the information about one entity and each column contains all the separate data points. Some of the most popular relational databases are MySQL, Oracle, MS SQL Server, SQLite, Postgres, and MariaDB. Non-relational databases are unstructured and have a dynamic schema. Example … Continue reading SQL Vs No-SQL

Kafka & Reliability

Apache Kafka guarantees the following with respect to Reliability Kafka does not guarantee ordering of messages between partitions. It does provide ordering within a partition.  Order of messages in a partition will be ordered. If message B was written after message A, using the same producer and to the same partition, then Kafka guarantees that … Continue reading Kafka & Reliability

Kafka Producer API Internals

The Kafka Producer API allows applications to send messages to the Kafka cluster. Producer APIs are simple to use, but this post will talk about what goes on under the hood of the producer when we send data Refer this code for a working example of sending messages to a Kafka topic named "time-series". Example … Continue reading Kafka Producer API Internals

How to consume custom objects from Kafka Topic ?

In the last post i shared an example on how to send custom java object to Kafka. This blog will share details on how to consume the data from Kafka Topic. We will consume the customer object that we sent to Kafka topic in the last post.Below is a depiction of the use case to … Continue reading How to consume custom objects from Kafka Topic ?

How to send custom objects to Kafka

This blog will share an example on how to send custom java object or data to Kafka Producer Use-caseRead a data file with customer detailsMap the data to a customer java objectSend the data to a Kafka Topic To build this use case , we will do following stepsCreate a Kafka topicBuild a json schema … Continue reading How to send custom objects to Kafka

Kubernetes Vs Docker

One of the common question that I encounter in many discussions is - Should I use Docker or Kubernetes? Both of them are leading technology wrt containers but comparing Docker and Kubernetes is wrong, as they are used for different reason and this blog tries to explain what are they are used for and why is it … Continue reading Kubernetes Vs Docker