Podcast

In this video I talk to Gunnar Morling from Decodable. Decodable helps you build realtime data-apps and services really fast. It is powered by Apache Flink behind the scenes and in this episode we discuss how it works? and how it can help you if you are dealing with data and real time usecase.

Gunnar shares his experience in building Decodable.

Chapters
00:00 Introduction
04:03 How can Decodable help?
08:11 Why Apache Flink ?
11:29 What Technical knowledge is required to start working with Decodable?
14:59 How does it work behind the scenes?
22:55 Is Idempotency guaranteed by default?
27:00 How does partitioning work?
29:01 How does pricing work?
33:35 What testing strategy should one follow while using Decodable?
36:08 What metrics are available to monitor?
38:07 When Decodable might not be the right fit (yet)?
40:08 Features coming up

Try decodable: https://www.decodable.co/
Connect with Gunnar: https://twitter.com/gunnarmorling and https://www.linkedin.com/in/gunnar-morling-2b44b7229/

I hope you like this episode. If you do, please don't forget to hit the like button, share it with your network and subscribe to the channel.

Cheers,
The GeekNarrator

Hey Everyone, In this video I talk to Gilad Kleinmann, CEO and Co-Founder of epsio.io, about Epsio and how it helps companies to run queries faster and cheaper.

Chapters:
00:00 Introduction
02:09 Defining the problem statement
07:17 What is Epsio ?
09:58 How does Epsio change my architecture?
12:59 Use of CDC
14:05 Where is the query result stored ? (Foreign data wrappers)
15:40 What permissions does Epsio needs?
16:43 How does Epsio parses a query and creates a virtual table?
24:15 Consistency model of Epsio
27:48 How do I know if Epsio is suitable for me?
31:41 How does it compare with Caching?
35:59 What metrics are available with Epsio?
38:32 What other databases does Epsio support? (will support)
40:47 How to know more about Epsio?
41:37 Pricing model of Epsio

Read more about epsio: https://www.epsio.io/
Docs: https://docs.epsio.io/
Foreign data wrappers: https://wiki.postgresql.org/wiki/Foreign_data_wrappers

Other playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

I hope you like this episode, please hit the like button if you did and subscribe to the channel if you haven't.

Cheers,
The GeekNarrator

Hey Everyone,

In this episode I talk to Anjul Sahu, Founder of CloudRaft, about all things Kubernetes. There are many common questions that we try to answer. If you ever wondered if you should adopt Kubernetes as a company? or learn Kubernetes as a Developer? or why people fail adopting Kubernetes? Or even if Kubernetes is worth the hype? Then this episode will bring a ton of clarity.


Chapters
00:00 Introduction
02:37 History and Background
06:06 k8s use cases
08:24 k8s features
09:34 Some basics of k8s
11:54 Common problems people face with k8s
15:48 k8s - challenges
18:59 Should you use k8s?
24:02 Alternatives of k8s
26:54 Top reasons why people fail with k8s
31:38 As a dev - should I learn k8s? Is it good for my career?

Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.

Other playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

Anjul Sahu: https://anjul.dev/
Cloudraft: https://www.cloudraft.io/

Our website: www.geeknarrator.com

References:
Kubernetes course: https://kodekloud.com/courses/kubernetes-for-the-absolute-beginners-hands-on/
Kubernetes Hard Way: https://github.com/kelseyhightower/kubernetes-the-hard-way
Kind: https://kind.sigs.k8s.io/
k3s: https://k3s.io/
k0s: https://k0sproject.io/

Cheers,
The GeekNarrator

In this video I talk to David Jacot who works as a Staff Software Engineer at @Confluent and has been a long time Kafka user, committer and PMC member. We covered how Kafka works internally in great depth.

We use Kafka for various use cases and it works great, but going one level below the abstraction and truly understanding the protocols, techniques and algorithms used is a fun ride.

Chapters:
00:00 Kafka Internals with David Jacot
03:33 Defining Kafka
05:16 Kafka Architecture(s)
11:39 Write Path - Producer sending data
18:35 How does replication work?
25:47 How do we track replication progress?
30:42 Failure Modes: Leader fails
38:18 Consumers: Push vs Pull
40:54 Consumers: How does fetch works?
49:03 Consuming number of bytes vs records
50:50 Optimising consumption
01:00:21 Offset management and choosing partitions
01:09:10 Ending notes

I hope you like this episode and more importantly you learnt some amazing techniques Kafka uses to ensure durability, low latency, simplicity and scalability in its architecture.

Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.

Other playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

Other episodes:
KsqlDB: https://youtu.be/2yE86P6uD_0
Exactly once semantics: https://youtu.be/twgbAL_EaQw

David's Linkedin: https://www.linkedin.com/in/davidjacot/

our website: www.geeknarrator.com

Cheers,
The GeekNarrator

#Shorts Batch vs Realtime Stream Processing (Where is the cost saving coming from?)

Watch the full video here: https://youtu.be/pOqQ-0cRWKU

In this video I talk to @gkcs on the Practical system design for Senior Engineers. Gaurav brings in a ton of experience in Competitive Programming, Teaching System Design and working on real world problems at companies like @Uber and Directi. Currently he is running his startup called @interviewready_ that helps folks perform better at Interviews. Do check him out if you haven't.

I truly believe System Design in the real world requires ton of other skills along with great technical skills. It becomes even more critical at Senior levels. So in this episode we discuss our view and experience on the same while providing some concrete examples from our real world experience.

I hope this episode brings in some clarity and adds some value to folks at all levels. At junior level, you would know what to expect and how to approach, At senior level, you would know what you might be missing or can do more to get to the next level.

Chapters:
00:00 System Design as Senior Engineers - Things that no one talks about about.
04:11 Fundamentals
08:32 Do you get to choose the Tech stack?
16:47 Thinking about the future while being fast
21:18 Real world vs Interviews
23:45 Importance of Communication Skills
30:50 IC does not mean Isolated Contributor
35:20 Senior Engineering is a journey not a Certificate.
39:17 Volatility, Uncertainity, Complexity, Ambiguity
42:18 Importance of Decision Making
47:49 Reducing the blast radius
53:10 Leading by example

Give it a watch and share with your network. Hit the like button and don't forget to subscribe to the channel.

Cheers,
The GeekNarrator

Hey All,

In this episode our guest is Hojjat Jaffarpour, from Deltastream. Hojjat has a ton of experience in Streaming systems. In his previous role at Confluent, he created and started the kSQL project, which is a quite powerful streaming database use widely.

We have discussed "Building Modern Streaming Systems" -

Chapters:
00:00 Introduction
01:14 Introduction to Streaming Systems
09:29 Components of a Streaming system
14:27 What makes these systems fast?
17:12 Push vs Pull Model - making the data available
24:08 Protocols and Techniques used in streaming systems
39:34 A little bit about DeltaStream - Who is it for?
43:06 Future of Stream processing

If you like the episode, please hit the like button, share and subscribe to the channel.

DeltaStream : https://www.deltastream.io/
kSQL : https://ksqldb.io/

Other playlists:
Distributed systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern Databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

Serverless Architecture: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfpX9hT_tJEFb69o0GWlEZS

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

Cheers,
The GeekNarrator

Hey Everyone,

In the 43rd episode I speak with Tim Berglund on Realtime Analytics with Apache Pinot.

Chapters:

00:00 Introduction
01:22 What do we mean by analytics and realtime analytics?
05:35 Can we define realtime in millis, seconds or minutes?
08:54 What is the fundamental difference between traditional analytics systems and Apache Pinot?
12:19 Was Kafka one of the reasons Apache Pinot could reach its full potential?
16:50 E-commerce Application example - How do I get my data in?
20:07 How is data stored (structured) on the disk?
23:31 Are joins available in Apache Pinot?
26:07 Joins vs pre-computing at ingestion
27:15 How is historical data ingested into Apache Pinot?
28:14 Types of indexes available in Apache Pinot
35:42 Do indexes cause write amplification? Is that a problem in Apache Pinot?
40:02 Point lookups in Apache Pinot
42:54 Anamoly Detection
45:51 Coming up in Apache Pinot

Links:
StarTree https://startree.ai/
Apache Pinot: https://pinot.apache.org/
Joins in Pinot: https://startree.ai/blog/apache-pinot-native-join-support
Apache Pinot Indexes: https://docs.pinot.apache.org/basics/indexing

Other playlists:
Distributed systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern Databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

Serverless Architecture: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfpX9hT_tJEFb69o0GWlEZS

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

I hope you like the episode. Like, share and subscribe to the channel.

Cheers,
The GeekNarrator

In this video I talk to Philip Fried from Estuary about Batch vs Realtime Stream Processing.
Philip brings a ton of experience in the world of data processing and has shared some of the best practices in implementing these systems. We dive deep into the world of data processing, covering batch and streaming systems, their challenges, tradeoffs and use cases.

Chapters:
00:00 Batch vs Realtime Stream Processing
03:25 What is Batch and Reatlime processing?
18:29 How does Batch and Realtime compare in terms of Latency and Throughput?
27:24 Where is the cost saving coming from? Compute?Storage? or Network?
31:38 Moving from Batch to Stream processing
37:50 How is Idempotency implemented in Streaming systems?
48:50 How do we approach Schema evolution in Batch and Streaming systems?
57:16 Summary - key points to keep in mind

Do checkout Estuary if you deal with a ton of data, and don't want to deal with the painful operations, infrastructure management, schema migrations etc and only want to focus on building highly scalable and resilient applications.

References:
Estuary: https://estuary.dev/
Flow documentation: https://docs.estuary.dev

If you like this video please hit the like button, share it with your network (whoever works with a ton of data) and subscribe to the channel.

Feel free to watch related episodes in the playlist: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern Databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

Distributed Systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4sd_CAWupznrpBezxT0gEvxB

Cheers,
The GeekNarrator