As systems grow more distributed, choosing the right message broker becomes crucial. Let me share some key insights from my experience working with the major players:
1: No "One Size Fits All"
- Kafka excels at high-throughput event streaming
- RabbitMQ shines for complex routing needs
- ActiveMQ is great for enterprise Java ecosystems
- Pulsar offers impressive geo-replication
- NATS keeps things lightweight and fast
The critical decision factors I've found:
1. Scale Requirements: Consider your growth trajectory
2. Protocol Support: Check compatibility with your stack
3. Operational Complexity: Factor in your team's expertise
4. Performance Needs: Latency vs throughput tradeoffs
5. Message Durability: Data persistence requirements
Start with your specific use case first, then match the technology.
I've seen teams struggle with over-engineered solutions when simpler options would suffice.
𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗡𝗲𝘁𝗳𝗹𝗶𝘅 𝗺𝗮𝗻𝗮𝗴𝗲 𝘁𝗼 𝘀𝗵𝗼𝘄 𝘆𝗼𝘂 𝗮 𝗺𝗼𝘃𝗶𝗲 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗶𝗻𝘁𝗲𝗿𝗿𝘂𝗽𝘁𝗶𝗼𝗻𝘀?
Netflix uses a 𝗺𝗶𝗰𝗿𝗼𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 where different functions are split into small, independent services. When you open Netflix, you connect through an Amazon Elastic Load Balancer (ELB) and API Gateway (Zuul) that directs your requests to the appropriate service. The content is stored in large cloud storage systems (like Amazon S3) and delivered through Content Delivery Networks (CDNs) placed worldwide to ensure fast streaming. This enables you to watch movies without hiccups, as they are streamed to you via the nearest geographical location.
Netflix's core consists of 𝘀𝗲𝘃𝗲𝗿𝗮𝗹 𝗸𝗲𝘆 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀 (𝗔𝗣𝗜𝘀). The Authentication service handles logins and security, while the User Profile service manages your preferences and watching history. The Content service (Play API) deals with all movie and show information, and the Recommendation service suggests content based on your viewing patterns. There's also a dedicated Search service to help users find content quickly.
Behind the scenes, Netflix runs 𝗺𝗮𝘀𝘀𝗶𝘃𝗲 𝗱𝗮𝘁𝗮 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀. They collect viewing data and user behavior to improve their recommendation system through machine learning. All this data is stored in different specialized databases - some for user information, others for content metadata, and separate for analytics. The whole system is designed to be highly available and fault-tolerant, meaning that Netflix can continue working even if some parts fail.
The most impressive part is the scale—Netflix can handle millions of users streaming content simultaneously, thanks to its extensive use of cloud services (primarily Amazon Web Services) and its globally distributed CDN network (Open Connect).
𝗡𝗲𝘁𝗳𝗹𝗶𝘅 𝗧𝗲𝗰𝗵 𝗦𝘁𝗮𝗰𝗸:
⚛️ 𝗙𝗿𝗼𝗻𝘁𝗲𝗻𝗱: React, Node.js, Redux, JavaScript
🖥️ 𝗕𝗮𝗰𝗸𝗲𝗻𝗱: Spring Boot, Apache Kafka, Java, Python
🔄 𝗙𝗿𝗼𝗻𝘁𝗲𝗻𝗱/𝗯𝗮𝗰𝗸𝗲𝗻𝗱 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝗰𝗮𝘁𝗶𝗼𝗻: GraphQL
📱 𝗠𝗼𝗯𝗶𝗹𝗲: Swift for iOS and Kotlin for Android
🗄️ 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀: AWS RDS (MySQL), Cassandra, CockroachDB, EV cache
✉️ 𝗠𝗲𝘀𝘀𝗮𝗴𝗶𝗻𝗴: Apache Kafka and Fink.
📊 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Flink, Apache Spark, Tableau, AWS Glue, Netflix Genie
🔒 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆: AWS IAM, AWS Shield, AWS KMS
📹 𝗩𝗶𝗱𝗲𝗼 𝘀𝘁𝗼𝗿𝗮𝗴𝗲: Amazon S3 and Open Connect
⚙️ 𝗗𝗲𝘃𝗢𝗽𝘀: Jenkins, Spinnaker, Amazon CloudWatch, Grafana, Prometheus, Chaos Monkey, Spinnaker, Atlas and more
🤖 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Amazon SageMaker, TensorFlow
#technology#softwareengineering#programming#techworldwithmilan#softwaredesign
Do you spend time watching live content on #socialmedia? Here's what I learned after hosting 133 live Q&As that accumulated 122.8 million views on the World Health Organization's Facebook, X and LinkedIn accounts:
🎙 It's important to find a way to make viewers your content co-creators by allowing them to ask questions or take part in the conversation directly. It's social media broadcasting, not TV broadcasting.
👩💻 Live features are a useful tool to help an organization bring the human face and personalities behind the brand closer to the followers.
👩⚕️ Live feature is also a great tool when you want to communicate a complex issue that can't fit into 280 characters, to avoid misunderstandings and mitigate misinformation spreading. It allows the subject matter experts to illustrate and humanise complex answers. It's also an opportunity to debunk mis- or disinformation circulating.
🔛 Consistency in the programme, format, speakers is very important to build trust and a regular viewer community.
📈 A success measure should be the quality of engagement, rather than the quantity. We, at WHO, have built a community of regular viewers, which has led to an increased quality of questions that we receive during the live programme.
❗ Don't get into the seat in front of a camera/mic without doing your homework - the prep is key to a successful session. The prep includes:
- social media and news analysis on the subject to make your conversation relevant
- writing a skeleton of possible conversation, so your guests can prepare
- have a prep meeting/chat with your guests. If it's their first time, do a thorough prep session or ran through with question examples and let them practice their answers.
- make sure to test the mics ahead of time
- make sure to test the connection ahead of time, if your live is done remotely
- make sure your guests feel comfortable with the space
- explore possibilities to advertise the session in advance
⌚ Do your best to be punctional, so your viewers aren't waiting for too long, if you advertise the session in advance.
🎦 Technology is amazing when it works. But there's always a possibility that something may fail. Don't be afraid to acknowledge it, apologise if needed and improvise. Last Friday, I hosted a live Q&A on #COVID19 with Dr Maria Van Kerkhove - my mic didn't work well. So we shared Maria's mic back and forth, until our colleagues behind the scenes figured out the problem. It wasn't ideal, but the show must go on! (P.S. We tested everything in advance!)
Below are a few examples of how we run WHO live sessions, thanks to Chris Black, Mark N. and the team. They have significantly advanced our set up and technology - but their work merits a post on its own!
To stream on multiple platforms, we're using Restream - I will also write about that development some time soon. Until then, let me know how you're running social media lives, or give us a feedback on what we can do better.
Here's a detailed blog post from CDN77 that discusses the technical setup, tweaks, and compromises necessary to achieve low latency when streaming over HTTP. It discusses protocols, segment duration, index fetching, and infrastructure compatibility. https://lnkd.in/eyWu4Jyp - #streamingmedia#infrastructure
I'm sure most of the Data Engineers must have worked with structured streaming in Databricks. In that case, Classic Observability (dashboard 1) 👇must be familiar to you.
Until recent months, this was my primary lens into streaming jobs:
📈 Input vs Processing Rate.
🕛 Batch Duration
Useful? Somewhat.
Actionable? Rarely.
In practice, this meant troubleshooting issues blindfolded:
👉 Is the source (Kafka, Autoloader, etc.) lagging? Not much clue.
👉 Are files being picked up at all? Can't tell.
👉 Is there a delay due to network, schema evolution, or bad config? Back to logs and Spark UI you go.
I like the new Streaming Observability in DLT & Workflows (dashboard 2) 👇as the insights to these metrics matter while troubleshooting issues:
✅ Backlog Duration: in plain seconds.
Now you know how far behind you are, not just guess from processing dips.
👉 Just imagine, if a Kafka source starts lagging due to a broken upstream connector, the new UI would show you x seconds of backlog instantly. By adjusting maxOffsetsPerTrigger, you'd clear the backlog.
In the old UI? We would notice 6 hours later via data delay alarms.
✅ Files/Bytes/Records Processed: per trigger.
You would see exactly how many files/records are being picked up across all supported sources.
👉 Example: Someone dropped CSVs to a shared ADLS container/External location but changed the folder structure. In this case, the file count would be 0 which means an immediate red flag.
✅ Alerting on Streaming Metrics
Probably, one of the most desired ones. Setting alerts on backlog thresholds, not just job success/failure (fig 3) 👇was the need of the hour. You could also do anomaly detection (to a certain extent) and get notified.
👉 Let's say your backlog > 30 mins, it's easy to trigger a webhook notification in Teams Channel now.
✅ Per-Trigger insights & real-time feedback
Each trigger gives visibility into its duration, lag, data processed, and even anomalies.
💡 In my opinion, it's literally moving from guesswork to a proper Observability. The UI isn't just prettier but operationally impactful.
While I'm working with streaming dat for so many years, what I'd expect more ⁉️
🚀 Source-level metrics: Backlog per Kafka topic, EventHub or an External location.
🚀 Streaming Lineage: Visual trace of where lag originates, especially useful when working with Live Multi-Task jobs.
🚀 Heatmaps or sparklines for spikes/drops in throughtput.
#Databricks#SparkStructuredStreaming#Lakeflow#Observability#DataEngineering
TVP Sport’s superb multi-channel revolution
It was great to see Poland Euro 2024 host broadcaster TVP Sport attempt a multi-channel approach for the opening game of Germany vs Scotland on Friday.
Of course they were broadcasting via TV and in app, but they also created a dedicated YouTube stream which hosted real-time Hawk-eye graphics integration.
I love to see this type of unique and engaging approach when it comes to live streaming of sports on YouTube - exact lookalikes of standard broadcast just don’t cut it.
It taps into advice I give to anyone when looking at creating live sports broadcasts for YouTube:
1) Different Platforms, Different Content:
YouTube content needs to offer something traditional broadcasting can’t. It’s about creating a unique experience.
2) Embrace Gen Z’s Multitasking:
Gen Z can handle multiple streams of information. Don’t be afraid to go bold with data - TVP Sport pushed this to the max and looking at audience sentiment they will need to find the sweet-spot
3) But Context is Key:
Data needs to tell a story. Without context, it’s just noise - this is where engaging story-driven graphics and personalities can help cut through.
It’s not about favouring broadcast over social streaming.
But about realising different approaches are needed for different audiences.
One size doesn’t fit all.
Congrats to Jakub Kwiatkowski on a bold approach.
H/T Dawid Prokopowicz#sportsbiz#sportsindustry#broadcast
Solving Real Streaming Problems with Elecard's StreamEye Studio and Boro
In a recent interview, I spoke with Alexander Kruglov, Product Manager at Elecard, about practical strategies for identifying and fixing common streaming issues. The discussion focused on two of Elecard’s core tools: StreamEye Studio for file-based analysis and Boro for live stream monitoring.
Alex walked through real-world cases engineers face every day:
• Encoding optimization: Viewers complain about poor quality even with high bitrates. StreamEye Studio helps engineers adjust settings and measure improvements using objective metrics like VMAF, PSNR, and SSIM.
• Live stream troubleshooting: Boro probes placed before and after transcoders help pinpoint exactly where errors occur, avoiding wasted time and guesswork.
• Timestamp misalignment: Even when a stream passes TR 101 290 checks, playback can still fail. StreamEye Analyzer visualizes PTS/DTS drift and audio/video sync issues that standard tools may miss.
• Ad insertion issues: When artifacts appear at ad boundaries, StreamEye and Boro provide clear evidence of problems like open GOP splices, helping teams assign responsibility and take corrective action.
Elecard's tools are used across the industry by encoder developers, broadcasters, IPTV, and OTT operators. Their goal is to provide visibility across the entire delivery chain, enabling engineers to work with data, not assumptions.
Watch the full interview here: https://lnkd.in/eGNhd9sa
Read the full blog summary: https://lnkd.in/eUvfeaUu
Meet Elecard at IBC, Booth 3.B47, or join their workshop on Sept 13 at 11:00 (https://lnkd.in/er6HnVcr)
This content was produced in collaboration with Elecard.
This week's sketch note is about how does throttling works in streaming data.
Throttling is a technique used in the context of streaming data to control the flow of information and prevent overload on systems or networks.
✔️ Throttling is the intentional slowing down or limiting of the rate at which data is sent or processed. It is a proactive approach to manage resources and prevent excessive usage.
✔️ Throttling can be applied at various levels, such as the producer side (source of data), the consumer side (destination of data), or at intermediate stages within the streaming pipeline.
✔️ The primary goal of throttling is to prevent system overload, maintain stability, and ensure that resources are used efficiently.
Assume there are two software systems, A and B, that communicate in a producer-consumer fashion.
- System A, the producer, produces events at a rate of 3 events per second.
- System B, the consumer, processes events at a rate of 1 event per second.
A's production rate is three times higher than what B processes at a time; B will definitely run out of resources and crash. That can be avoided by throttling down A’s events to 1 event per second, a rate at which B can reliably process.
To implement that, we can use a message queue/buffer on either side. For example, B can consume events and buffer them on a queue until B is ready to process them.
Alternatively, we can use Apache Kafka/Redpanda with a stateful stream processor, like Apache Flink, to control throttling in the middle.
For example:
➡️ The source topic in Kafka accepts events from A at the original throughput (3 events/sec)
➡️ A Flink job consumes the source topic and leverages a time/length window function to bucket events to the throttled events topic at a throughput of 1 event/sec. The window function buffers the additional events in its state.
➡️ B consumes from the throttled events topic (1 event/sec) and continues processing.
Throttling in streaming data finds versatile applications across various domains. In real-time analytics, it aids resource management by controlling the rate of data processing, ensuring optimal performance of analytics engines and databases. When integrating with external systems or APIs, throttling aligns the flow of data with the capabilities of receiving systems, preventing overload and maintaining a balanced interaction. Throttling is essential in mitigating Denial-of-Service (DoS) attacks, safeguarding streaming data applications from malicious attempts to overwhelm the system. In IoT applications, it ensures Quality of Service (QoS) by regulating the rate of data transmission, preventing network congestion, and reducing latency. Throttling also plays a crucial role in adhering to rate limits imposed by API providers, promoting fair usage and stable interactions between data producers and consumers.
#streamingdata#sketchnotes#systemdesign#throttling#kafka#redpanda#apacheflink
Choosing the right #DataStreaming technology for a #MicrosoftFabric#Lakehouse is not a trivial decision. The trade offs between Apache Kafka, Azure Event Hubs, and Confluent Cloud directly influence scalability, cost, governance, and long term strategy.
#ApacheKafka is the de facto standard with unmatched flexibility and community support, but self management in the cloud rarely makes sense from a TCO perspective. #Azure#EventHubs integrates tightly with the Azure ecosystem and is simple for plain ingestion into #OneLake and Fabric, but it has clear limits in Kafka compatibility, scalability, and enterprise features. #Confluent#Cloud delivers a full #DataStreamingPlatform including Kafka, Flink, and Iceberg. It goes far beyond ingestion by providing enterprise security, governance, multi cloud flexibility, and disaster recovery.
My blog post breaks this down in detail and helps you understand when to use each option: When to Choose Apache Kafka vs Azure Event Hubs vs Confluent Cloud for a Microsoft Fabric Lakehouse
https://lnkd.in/epNeNddG
For organizations building a data strategy around Microsoft Fabric, the decision is not just about technical fit. It is also about business value, SLAs, and creating a foundation for both analytical and operational workloads.
Which of these three options do you currently see most often in real world Microsoft Fabric projects and why?
Hot take: data streaming is eating analytics. Why? Stale data + AI = 💣
Now with Striim (unified CDC + stream processor) integrating with major cloud data warehouses that natively supporting streaming ingest (Snowflake, BigQuery, Redshift, Databricks), fresh data is attainable for data teams of all sizes.
Here's the simple blueprint on how you can upgrade your current data stack to adopt real-time data, streaming, and AI while simultaneously reducing costs and latency.
1️⃣ Identify your internal streaming data sources. Hint: your database's transaction log is a streaming source.
2️⃣ Connect a streaming tool like Striim (fully managed, free to start) or (Kafka + Debezium) to your database and stream it to your warehouse of choice such as Snowflake. Bonus points for using streaming ingest option such as Snowpipe Streaming.
3️⃣ Turn off your ELT tool's database connectors. You've now decreased your monthly active rows and replaced complicated batch-based merge processes with real-time data flows
Now your data team has...
💰 Lowered internal data infrastructure costs
🐆 Delivered near real-time data freshness SLAs on operational data
🚨 Can alert on data issues (data contract violations, schema changes, broken pipelines) with Streaming SQL and built-in machine learning operators
#dataengineering#analyticsengineering