Data Protection and Backup Solutions

𝐖𝐡𝐚𝐭 𝐢𝐬 𝐂𝐡𝐚𝐧𝐠𝐞 𝐃𝐚𝐭𝐚 𝐂𝐚𝐩𝐭𝐮𝐫𝐞? Being a Data Engineer its not just important to build pipelines to deliver data to the consumers, but synchronising the data to empower organisations for propagating real-time data changes across distributed systems is equally important. Change data capture helps to capture, track and enable teams to replicate data instantly and incrementally by continuously monitoring a source database for changes (inserts, updates, and deletes) and capturing these modifications as they happen. The changes once captured are streamed to target systems such as data warehouses, data lakes, or other databases to keep them in sync with minimal latency. The working of the CDC follows the following steps: 1. Detect 2. Extract 3. Transform 4. Deliver Each of these leverages several approaches to implementing CDC depending on the technical capabilities: -> Log-Based  -> Query-Based -> Trigger-Based As a data engineer, when working with Change Data Capture, do not miss to consider the following: - Schema Evolution Handling - Error Handling and Recovery - Data Synchronization - Monitoring and Observability - Security and Compliance There could be various ways to implement CDC, but with the evolving industry some handy tools and services to leverage include: • AWS: AWS Database Migration Service (DMS) • GCP: Datastream (as shown in our architecture) • Azure: Azure Data Factory • Open Source: Debezium, Maxwell, Airbyte, Kafka Connect Implementing CDC effectively, can help data engineers build more resilient, efficient, and timely data pipelines with real-time data availability to deliver significant business value. 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝗶𝗻𝗰𝗼𝗿𝗽𝗼𝗿𝗮𝘁𝗲 𝗖𝗵𝗮𝗻𝗴𝗲 𝗗𝗮𝘁𝗮 𝗖𝗮𝗽𝘁𝘂𝗿𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀? Also, do let me know in comments if you would like me to share a detailed article on change data capture with some real time use case!! #data #engineering #cloud #azure #gcp #aws #bigdata

Using Veeam Backup & Replication? Great. But it’s not just about having backups, it’s about building a resilient backup strategy that can stand up to ransomware, disasters, and human error. Here’s a proven framework to guide your backup approach: the 3-2-1-1-0 rule: • 3 copies of your data • On 2 different media types • With 1 copy stored offsite • 1 copy kept offline or immutable (e.g., hardened Linux repo) • And 0 errors in backup verification or recovery testing Combine that with security best practices for Veeam: • Use immutability to lock backups from changes, even by admins • Apply RBAC to restrict access to backup infrastructure • Encrypt backup data in transit and at rest • Enable MFA on all backup, related accounts • Use Veeam ONE or a SIEM to detect anomalies and changes Because if your backups are vulnerable, you don’t have a recovery plan, you have a false sense of security. Secure. Test. Verify. Then sleep better at night. #Veeam #BackupStrategy #RansomwareProtection #CyberResilience #DataRecovery #InfrastructureSecurity

"Keeping Multiple #Databases in Sync in Real-Time Using #ApacheKafka Connect and #ChangeDataCapture" #Microservices architectures have now been widely adopted among developers, and with a great degree of success. However, drawbacks do exist. Data silos can arise where information processed by one microservice is not visible to the other microservice. This blog post will review the advantages and disadvantages inherent to moving data from a database using #JDBC and #CDC, and then explores the real use case of how a legacy bank used #KafkaConnect to bridge the silos and keep multiple applications/databases in sync. https://lnkd.in/esv_m5-w

🔒 Understanding AWS: Service Level Backup vs. Service Level Snapshot 🔍 Navigating the AWS landscape can be intricate, especially when juggling terms like "service level backup" and "service level snapshot." 🤔 Let's break down the key differences: 𝐒𝐞𝐫𝐯𝐢𝐜𝐞 𝐋𝐞𝐯𝐞𝐥 𝐁𝐚𝐜𝐤𝐮𝐩: 📌 Definition: A backup is a separate copy of your data at a specific point in time, manually or through services like AWS Backup. 🌐 Scope: Covers various AWS resources – EBS volumes, RDS databases, DynamoDB tables, and more. 🎯 Focus: Primarily safeguards data, aiding recovery from accidental deletion, corruption, or disasters. 🛠 Customization: Highly flexible with scheduling, retention, encryption, and lifecycle management. 🌟 Examples: Automated daily backup of your MySQL RDS database via AWS Backup. Manual full backup of your EBS volume before a risky upgrade. 𝐒𝐞𝐫𝐯𝐢𝐜𝐞 𝐋𝐞𝐯𝐞𝐥 𝐒𝐧𝐚𝐩𝐬𝐡𝐨𝐭: 📌 Definition: A snapshot, tied to a single AWS service (e.g., EBS snapshot), is a point-in-time copy of data. 🌐 Scope: Limited to the associated service – EBS snapshots for EBS volumes, RDS snapshots for RDS databases, etc. 🎯 Focus: Goes beyond recovery; useful for creating new resources, data migration, and testing scenarios. 🛠 Customization: Less flexible than backups; limited scheduling and lifecycle management based on the service. 🌟 Examples: Snapshot of your EBS volume to launch a new EC2 instance with identical data. Taking an RDS database snapshot before a major update. 𝐈𝐧 𝐒𝐮𝐦𝐦𝐚𝐫𝐲: 🔄 Backups: Versatile, customizable data copies for recovery across various services. 📸 Snapshots: Service-specific copies with additional functionalities beyond data recovery. 🤔 𝐂𝐡𝐨𝐨𝐬𝐢𝐧𝐠 𝐁𝐞𝐭𝐰𝐞𝐞𝐧 𝐓𝐡𝐞𝐦: Backup: For straightforward data protection and recovery. Snapshot: When additional features like resource creation or scenario testing are needed. Understanding these distinctions ensures you're equipped to make the right choice based on your specific needs and desired functionality. #learnwithswetha #awsbackup #awssnapshot #learning #career #aws

💭 Ever faced the challenge of keeping your data consistent across regions, clouds, and systems — in real time? A few years ago, I worked on a global rollout where CRM operations spanned three continents, each with its own latency, compliance, and data residency needs. The biggest question: 👉 How do we keep Dataverse and Azure SQL perfectly in sync, without breaking scalability or data integrity? That challenge led us to design a real-time bi-directional synchronization framework between Microsoft Dataverse and Azure SQL — powered by Azure’s event-driven backbone. 🔹 Key ideas that made it work: Event-driven architecture using Event Grid + Service Bus for reliable data delivery. Azure Functions for lightweight transformation and conflict handling. Dataverse Change Tracking to detect incremental updates. Geo-replication in Azure SQL to ensure low latency and disaster recovery. What made this special wasn’t just the technology — it was the mindset: ✨ Think globally, sync intelligently, and architect for resilience, not just performance. This pattern now helps enterprises achieve near real-time visibility across regions — no more stale data, no more integration chaos. 🔧 If you’re designing large-scale systems on the Power Platform + Azure, remember: Integration is not about moving data. It’s about orchestrating trust between systems. #MicrosoftDynamics365 #Dataverse #AzureIntegration #CloudArchitecture #PowerPlatform #AzureSQL #EventDrivenArchitecture #DigitalTransformation #CommonManTips

Monzo Bank built Stand-In, a backup system to ensure resilient banking services to millions of customers. 🚀 🚀 Let's understand in simple words the architecture and trade-offs of the system. Monzo wants customers to continue banking despite disruptions in the software or cloud. What were the goals of the system ? 🎯 High availability - Zero downtime for banking operations. 🎯 Cost effectiveness - Minimize infra/backup costs. 🎯 Avoid single point of failure on a cloud vendor. Monzo operated their primary platform on AWS. And they built Stand-In on Google cloud. What did the architecture of the system look like ? 👉 Critical microservices - Bank transfers, card payments, and balance checks. 👉 Message queues - For data transfer between AWS and GCP. 👉 Managed databases - Syncing state from primary platform. During outages, customers were redirected to the Stand-In platform. This minimized the customer disruption. What trade-offs were made by the architecture ? 1️⃣ Eventual consistency - Was used to ensure high availability. But resulted in data store inconsistencies. 2️⃣ Cost effectiveness - Only a subset of critical services were deployed. This was unlike typical Disaster Recovery solutions. Stand-In only resulted in 1% additional cost over the primary cluster in AWS. What were some other challenges of Stand-In ? 🌐 End-to-end customer testing. 🌐 Reconciliation between AWS and GCP due to inconsistencies. 🌐 Interoperability while dealing with multi-cloud architectures. Monzo's Stand-In platform is a great example of how large companies build resilient distributed systems. 🔥 🔥 One of the key take-aways from the system design is prioritising critical features, minimizing the cost and providing high availability for customers. Have you dealt with multi-cloud deployments in the past ? If yes, share what challenges you faced in the comments below. 👇 #tech #softwareengineering #systemdesign

🔥 A while back, I was given the challenge of designing a Disaster Recovery strategy for a 3-tier architecture. No pressure, right? 😅   Challenge accepted, obstacles overcome, mission accomplished: my e-commerce application is now fully resilient to AWS regional outages.   So, how did I pull this off? Well… let me take you into a world where disasters are inevitable, but strategic planning, resilience and preparedness turn challenges into success—just like in life. ☺️   Firstly, I identified critical data that needed to be replicated/backed up to ensure failover readiness. Based on this, I defined the RPO and RTO and selected the warm standby strategy, which shaped the solution: Route 53 ARC for manual failover, AWS Backup for EBS volume replication, Aurora Global DB for near real-time replication, and S3 Cross-Region Replication.   Next, I built a Terraform stack, and ran a drill to see how it works. Check out the GitHub repo and Medium post for the full story. Links in the comments. 👇   Workflow: ➡️ The primary site is continuously monitored with CloudWatch alarms set at the DB, ASG, and ALB levels. Email notifications are sent via SNS to the monitoring team. ➡️ The monitoring team informs the decision-making committee. If a failover is necessary, the workload will be moved to the secondary site. ➡️ Warm-standby strategy: the recovery infra is pre-deployed at a scaled-down capacity until needed. ➡️ EBS volumes: are restored from the AWS Backup vault and attached to EC2 instances, which are then scaled up to handle traffic. ➡️ Aurora Global Database: Two clusters are configured across regions. Failover promotes the secondary to primary within a minute, with near-zero RPO (117ms lag). ➡️ S3 CRR: Data is asynchronously replicated bi-directionally between buckets. ➡️ Route 53: Alias DNS records are configured for each external ALB, mapping them to the same domain. ➡️ ARC: Two routing controls manage traffic failover manually. Routing control health checks connect routing controls to the corresponding DNS records, making possible switching between sites. ➡️ Failover Execution: After validation, a script triggers the routing controls, redirecting traffic from the primary to the secondary region.   👉 Lessons learned: ⚠️ The first time I attempted to manually switch sites, it happened automatically due to a misconfigured Route Control Health Check. This could have led to unintended failover—not exactly the kind of "automation" I was aiming for.   Grateful beyond words for your wisdom and support Vlad, Călin Damian Tănase, Anda-Catalina Giraud ☁️, Mark Bennett, Julia Khakimzyanova, Daniel. Thank you, your guidance means a lot to me!   💡Thinking about using ARC? Be aware that it's billed hourly. To make the most of it, I documented every step in the article. Or, you can use the TF code to deploy it. ;)   💬Would love to hear your thoughts—how do you approach DR in your Amazon Web Services (AWS) architecture?

You likely know having a backup is essential—but it’s not enough. A backup that hasn't been validated could leave your business vulnerable when disaster strikes. Whether it's a ransomware attack, hardware failure, or accidental deletion, relying on untested backups can lead to incomplete or corrupted data recovery. Periodically restore data from backups to verify their integrity. Don’t assume they work—test them! Implement the 3-2-1 rule: 3 copies of your data, on 2 different media types, with 1 stored off-site. Use automated tools to monitor your backup processes and receive alerts for any failed jobs or inconsistencies. Ensure backups are encrypted, both in transit and at rest, to protect against unauthorized access. A validated backup system ensures you're not just backing up data, but backing up reliably. Thus, giving you peace of mind when you need it the most. If the backup does not have validated recovery, it is not a backup – it is, at best, a hope! - Keith Palmgren Don’t wait for a crisis to find out your backup plan wasn’t enough!

Different Backup Options and their uses in SQL server: 1. WITH INIT - Purpose: Overwrites the existing backup file. - Use case: Clears the old backup before writing a new one. 2. WITH NOINIT - Purpose: Appends the backup to an existing backup file. - Use case: Useful for incremental backups. 3. WITH SKIP - Purpose: Skips backup if the same backup set exists on the media. - Use case: Avoids duplication on backup media. 4. WITH COMPRESSION - Purpose: Compresses the backup to save space. - Use case: Ideal for reducing backup file size. 5. WITH STATS - Purpose: Displays progress during the backup. - Use case: Helps monitor the backup progress in large databases. 6. WITH DESCRIPTION - Purpose: Adds a description to the backup file. - Use case: Provides more context or metadata for the backup file. 7. WITH COPY_ONLY - Purpose: Creates a backup without affecting the log chain. - Use case: Used for making backups that don’t interfere with regular log backups. 8. WITH DIFFERENTIAL - Purpose: Backs up changes since the last full backup. - Use case: Reduces backup time and storage compared to full backups. 9. WITH PASSWORD - Purpose: Encrypts the backup with a password. - Use case: Provides added security for sensitive backup data. 10. WITH NORECOVERY - Leaves the database in a restoring state for further restores. - Use: Part of restore sequence (e.g., log shipping). 11. WITH RECOVERY - Brings the database back online after restore. - Use: Final step in a restore sequence. 12. WITH CONTINUE_AFTER_ERROR - Continues the backup even if errors occur. - Use: When minor errors shouldn't stop the backup. 13. WITH CHECKSUM - Verifies backup integrity during the process. - Use: Ensures backup can be restored without corruption. 14. WITH NOWAIT - Completes backup without waiting for status feedback. - Use: For faster backups without waiting for details. 15. WITH MEDIA_ERROR - Continues backup even with media errors. - Use: When media errors shouldn't interrupt the process. 16. WITH SNAPSHOT - Backs up from a database snapshot. - Use: Backup without impacting database performance. 17. WITH FILEGROUP - Backs up specific filegroups, not the whole database. - Use: For large databases with multiple filegroups. 18. WITH TRUNCATEONLY - Truncates transaction logs without backing them up. - Use: To clear log files without backing up the log.

Day 26 - RMAN Configuration Parameters SHOW ALL; Here’s a breakdown of key RMAN configuration parameters: 1. RETENTION POLICY - Determines how long backups are retained before they are considered obsolete. - Options: - `REDUNDANCY <n>` → Retains the last `n` backups. - `RECOVERY WINDOW <n> DAYS` → Retains backups needed for point-in-time recovery within `n` days. - Example: CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS; 2.BACKUP OPTIMIZATION - Prevents RMAN from backing up files that are already backed up. - Options: `ON` / `OFF` - Example: CONFIGURE BACKUP OPTIMIZATION ON; ``` 3. DEFAULT DEVICE TYPE - Defines the default storage location for backups. - Options: `DISK` or `SBT_TAPE` (tape storage). - Example: CONFIGURE DEFAULT DEVICE TYPE TO DISK; 4. CONTROLFILE AUTOBACKUP - Automatically backs up the control file and the server parameter file (`spfile`) after structural changes. - Options: `ON` / `OFF` - Example: CONFIGURE CONTROLFILE AUTOBACKUP ON; 5.PARALLELISM - Specifies the number of channels (parallel backup streams) for backup and restore operations. - Example: CONFIGURE DEVICE TYPE DISK PARALLELISM 4; ``` 6. CHANNEL CONFIGURATION - Specifies parameters for backup channels, such as disk/tape, format, and maximum files per backup set. - Example for disk: CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/backup/%U'; ``` - Example for tape: CONFIGURE CHANNEL DEVICE TYPE SBT_TAPE FORMAT 'SBT_TAPE'; ``` 7. MAXSETSIZE - Limits the maximum size of a backup set. - Example: CONFIGURE MAXSETSIZE TO 10G; 8.COMPRESSION ALGORITHM - Enables backup compression to save storage space. - Options: `BASIC`, `LOW`, `MEDIUM`, `HIGH`, `NONE`. - Example: CONFIGURE COMPRESSION ALGORITHM 'HIGH'; CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET; 9.ARCHIVELOG DELETION POLICY - Controls when archived redo logs are deleted. - Example (delete logs after backup): CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 2 TIMES TO DISK; ``` 10.ENCRYPTION - Enables encryption for secure backups. - Options: `ALGORITHM 'AES128'`, `AES192`, `AES256`, `NONE`. - Example: ```sql CONFIGURE ENCRYPTION ALGORITHM 'AES256'; ``` 11.SNAPSHOT CONTROLFILE - Specifies the location of the snapshot control file used by RMAN. - Example: CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/backup/snapcf.f'; 12.RMAN DEFAULT BACKUP TYPE - Specifies whether backups should be image copies or backup sets. - Example (for compressed backup set): CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET; #Oracle #DatabaseAdministrator #OracleDBA #SystemsEngineer #SoftwareEngineer