KAfka In detail

-



I have already explained what is Kafka and how to install Kafka and how to write some code to create producers and consumer for Kafka.


In this article, I will future explain about  Kafka and its functions.

In the era of Big Data, lots and lots of data(volume) are being produced every second(velocity) from various sources like social media, blogs that I am writing currently, e-commerce, etc., which gets stored across different platforms in different schemas(varieties). 
In order to perform any ETL (Extract, Transform, Load) operation, a messaging/streaming system is needed which should be asynchronous and loosely coupled i.e. data from various sources/clients like HDFS, Cassandra, RDBMS, application log file, etc. could be dumped at a single place at the same time without all the clients depending on each other. 
One of the solutions to the problem is Kafka — An open-source distributed streaming platform created by LinkedIn and later donated to Apache. It is written in Scala.

Kafka Broker

Benefits

Following are a few benefits of Kafka −

  • Reliability − Kafka is distributed, partitioned, replicated, and fault tolerance.

  • Scalability − Kafka messaging system scales easily without downtime.

  • Durability − Kafka uses a Distributed commit log which means messages persists on disk as fast as possible, hence it is durable..

  • Performance − Kafka has high throughput for both publishing and subscribing messages. It maintains stable performance even many TB of messages are stored.

Kafka is very fast and guarantees zero downtime and zero data loss.

Use Cases

Kafka can be used in many Use Cases. Some of them are listed below −

  • Metrics − Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.

  • Log Aggregation Solution − Kafka can be used across an organization to collect logs from multiple services and make them available in a standard format to multiple consumers.

  • Stream Processing − Popular frameworks such as Storm and Spark Streaming read data from a topic, processes it, and write processed data to a new topic where it becomes available for users and applications. Kafka’s strong durability is also very useful in the context of stream processing.




Comments

Post a Comment

Popular posts from this blog

Setup source code of Visual studio on Bitbucket

-
Image
Setup source code of Visual studio on Bitbucket  Prerequisites:  Download and install  bitbucket extension for Visual studio  from following URL https://marketplace.visualstudio.com/items?itemName=MistyK.VisualStudioBitbucketExtension Download the following software for command-line access of Bitbucket https://git-scm.com/downloads The following are the steps to set up  VS for Bitbucket. 1.        In the file menu, go to the source control options and clear the bindings. You'll specifically want File - Source Control - Advanced - Change Source Control... 2.        You will see the screen as bellow 3.    Now one by one remove connect checkboxes by clicking the unbind button in the top toolbar and at the end press ok.      4.  To tools--> options--> Source Control change current source control plug-in to as following ...
Read more

.Net Core WebClient and proxy setup

-
Image
In .net core, if you need to call 3rd party API, you may have to create an object of WebClient and do some steps to retrieve and send data to that API. today I am going to explain to you  how you can create WebClient and how you can set up proxy settings in WebClient so you can call API throw proxy and retrieve any data.  First of all, you need to create a dictionary of strings which can contain headers for 3rd party API, that code should be as following, you can add more parameters as per your need, for me, I just added one header. public    Dictionary < string ,  string >  GetHeaderForJson () {      Dictionary < string ,  string >  headers  =  new   Dictionary < string ,  string >();      headers . Add ( "Content-Type" ,  "application/json" );      return   headers ; }   one you created a header dictionary its time to create WebClient...
Read more

Tables with ledger in sql server 2022

-
Image
Ledger  in Sql Server 2022 In SQL Server 2022, you can create a ledger using the built-in functionality of the system-versioned temporal tables. A ledger is a record of all transactions that have affected a particular account or set of accounts, and it can be used to keep track of financial data, inventory data, or any other data that involves transactions. Append-only ledger Are ideal for application patterns that are insert-only, such as security information and event management (SIEM) applications. Append-only ledger tables block updates and deletions at the API level. This blocking provides more tampering protection from privileged users such as system administrators and DBAs. To create a table with append-only control use following is the   query CREATE SCHEMA [AccessControl]; GO CREATE TABLE [AccessControl].[KeyCardEvents] ( [EmployeeID] INT NOT NULL , [AccessOperationDescription] NVARCHAR ( 1024 ) NOT NULL , [ Timestamp ...
Read more