Coder Legion - Recent posts tagged high-availability

Building database synchronization the boring way

Tue, 08 Sep 2026 21:13:56 +0000

TL;DR: iamroot.tech uses a custom-built, asynchronous MySQL database synchronization system handling ~10 million daily changes across several distributed nodes without consensus algorithms or distributed coordinators.
Key Architecture & Trade-offs:

Primary Keys: UUIDs to allow conflict-free row generation on any node.
Change Tracking: Database triggers log INSERT, UPDATE, and DELETE operations into a simplified change log.
Replication Mechanism: Asynchronous background workers process log entries.
Conflict Avoidance: Application-level single-writer authority per dataset
Fault Tolerance: Nodes catch up automatically after offline periods by replaying the simple change log backlog.

The setup - servers will disappear

iamroot.tech runs across numerous servers in different geographical locations. They are deliberately inexpensive machines, and an important assumption behind the entire architecture is that individual servers will - at some point in time - disappear, underperform or otherwise misbehave.

The goal has never been to make every server perfectly reliable. The goal is to make each server failure completely uninteresting.

That creates an interesting problem for the databases. A server needs access to shared data, but I don't want a write on server A to depend on server B being available. In fact, I don't really want server A to care about server B at all while serving a request.

So I ended up building my own synchronization system. It currently keeps roughly 50–60 tables synchronized across numerous servers and handles around 10 million changes on a normal day.

It isn't particularly clever. In fact, it's about as simple as it gets.

And that is rather the point.

A failing server should create a backlog, not an outage

The most important principle is that database writes are local operations. If the application running on server A needs to update something, it writes to the local MySQL database and continues.

Synchronization happens afterwards.

If server B happens to be slow, offline or has been gone for three days, that doesn't change what happens on A. It simply means B has more catching up to do when it eventually returns.

In practice, this gives iamroot.tech a distributed data architecture without trying to turn MySQL itself into a distributed database. Each server owns its local database and can operate independently, while relevant data is eventually synchronized between them.

Temporary differences between nodes are therefore expected. Eventual consistency isn't something the architecture reluctantly tolerates; it is part of the design.

That trade-off works because the application itself understands how the data is used.

The implementation is almost embarrassingly simple

Tables that need to be shared use UUIDs as their primary keys. That allows records to be created independently on different servers without coordinating ID ranges or worrying about primary key collisions later.

Each server then maintains its own local change log. Synchronized tables have triggers for INSERT, UPDATE and DELETE, and whenever something changes, a small entry containing the table, record ID, operation and timestamp is written to the log.

A background service processes those entries and sends the changes to whichever nodes should receive them. There is no central replication queue and no synchronization master.

For inserts and updates, I use MySQL's INSERT ... ON DUPLICATE KEY UPDATE. If a server somehow missed the original INSERT but later receives an UPDATE, the record simply gets created anyway.

Replicated writes bypass the change-log triggers for that database connection, while normal application writes still trigger them. Otherwise A would update B, B would send the same change back to A, and my wonderfully simple synchronization system would become a wonderfully effective infinite loop.

DELETEs work through the same mechanism. The DELETE entry stays in the change log until the relevant nodes have processed it, effectively making it a temporary replication tombstone. A server can therefore disappear for days and still learn which records were deleted while it was gone.

That's basically it.

Avoiding problems instead of solving them

This is not conflict-safe multi-master replication.

If two servers independently modify the same record at exactly the wrong moment, there is no clever distributed algorithm deciding which version represents universal truth.

Instead, I largely avoid the problem at the application level.

Individual records normally have one logical writer - one server or subsystem that is effectively king for that data. The code working with those records knows about that architecture.

That is an important distinction. I could add version vectors, distributed locking, automatic conflict resolution and considerably more machinery. Those are perfectly valid solutions to problems other systems have.

But if I can structure the application so the problem doesn't normally exist, I would rather do that.

The same philosophy applies elsewhere. Synchronization is at-least-once rather than exactly-once. Duplicate work happens, but the operations are idempotent, so that is mostly harmless. The change log isn't an event store either. If a record changes ten times, I don't need to reconstruct all ten historical versions. I care about getting the current state where it needs to go.

Recovery is deliberately boring

For every table and destination node, the synchronizer remembers how far through the local change log that node has progressed.

Important data is typically synchronized every 10 – 20 seconds, while less important tables may only be processed every ten minutes. If a destination disappears, its position simply stops advancing while everybody else continues.

When it returns, there is normally nothing for me to do. The synchronizer knows where it stopped and continues from there. Once all relevant nodes have moved beyond old change-log entries, those entries can be removed.

This isn't just how I hope it works. Servers have disappeared and returned plenty of times in production.

The recovery procedure is dead simple: sit back and wait.

I like recovery procedures like that.

Why not just use MySQL replication?

MySQL already provides very capable replication, and this isn't an attempt to build a better replacement.

I wanted slightly different properties.

Not every table needs to exist everywhere. Different data can have different synchronization intervals and destinations. More importantly, replication isn't part of the availability model for an individual application server. A slow destination creates work for the synchronizer, not latency for the application serving a user.

There are obvious trade-offs. The synchronizer doesn't automatically understand referential integrity between tables, although I can control their processing order. It doesn't negotiate schemas either; keeping compatible schemas across the relevant servers is my responsibility.

Those are not necessarily features waiting to be implemented. They are boundaries I know about and can design around.

Boring as a feature

Around 10 million changes pass through this setup every day, with the ASN database alone sometimes generating several million updates during a processing run.

Could I make the synchronization system considerably smarter? Absolutely.

But I didn't set out to build a distributed database. I wanted a multitude of servers that could operate independently, share the data they needed, tolerate each other disappearing, and recover without me getting involved.

The synchronization system is simply what fell out of those requirements.

There are plenty of situations where this design would be completely inappropriate. There are also plenty where sophisticated distributed database technology solves very real problems.

But complexity has a real cost.

For iamroot.tech, I'd rather have infrastructure I can explain on a single piece of paper, predict when something breaks, and largely leave alone when a server disappears.

Extremely simple, asynchronous and a tad inefficient in the actual operations, yes - but very efficient at solving the problem, very predictable in the way it works, and - most importantly - it does not require any human intervention at any point.

That is all I need.

Zero Downtime: Designing an Azure Multi-Region Architecture

Mon, 01 Dec 2025 05:00:00 +0000

Zero Downtime: Designing an Azure Multi-Region Architecture

Introduction

High availability isn’t just a buzzword to throw into a system design interview; it’s the difference between a minor hiccup and a career-defining disaster. We often build for the "happy path," assuming Azure regions are invincible fortresses. But cables get cut, weather happens, and yes, sometimes entire regions go dark.

If your application lives in a single region, your uptime is at the mercy of that specific geography. Today, we’re going to walk through moving from a fragile, single-region setup to a robust Active-Active Multi-Region Architecture that can survive a total regional outage without your users ever noticing.

The Story: The 3 AM Wake-Up Call

It was 3:14 AM on a Tuesday—it’s always a Tuesday. My phone buzzed off the nightstand, vibrating with the distinctive, heart-stopping pattern of PagerDuty.

"East US is down," the message read.

I groggily opened my laptop, hoping it was just a blip. It wasn't. An underlying storage outage in the primary region had cascaded, taking our entire e-commerce platform offline. We had a "Disaster Recovery" plan, but it was a cold backup in West Europe. Restoring it meant updating DNS records manually, warming up cold caches, and praying the database backups were consistent.

It took us four hours to get back online. In internet time, that’s an eternity. We lost revenue, but worse, we lost trust.

That morning, fueled by too much coffee and regret, we made a pact: Never again. We were going to build a system that could lose an entire region and keep humming along. Here is exactly how we did it.

Core Concepts: Active-Passive vs. Active-Active

Before we write a single line of code, we need to agree on the strategy.

Active-Passive: You have a primary region handling traffic and a secondary region sitting idle (or receiving replication data). Failover usually requires manual intervention or a delay.

Active-Active: Both regions are live. They both handle traffic simultaneously. If one dies, the global load balancer simply stops sending users there. This is the holy grail for zero downtime.

To achieve this, we need three layers of abstraction:

The Brain (Traffic Routing): Something global to direct users.
The Muscle (Compute): Stateless application servers.
The Heart (Data): A database that accepts writes in multiple places.

Step-by-Step Guide

We are going to build this using Azure Front Door, Azure App Service, and Cosmos DB.

1. The Entry Point: Azure Front Door

Old school setups used Azure Traffic Manager (DNS-based). Modern web apps should use Azure Front Door. It operates at Layer 7 (HTTP/S), uses the Microsoft global edge network to speed up content, and fails over almost instantly because it uses active health probes.

Why Front Door? It terminates SSL at the edge and routes traffic to the closest available backend.
The trick: We set up an "Origin Group" containing our East US and West Europe endpoints.

2. The Compute Layer: Go Stateless

This is where most developers get stuck. You cannot store session state (like a shopping cart or user login) in the memory of your web server. If a user hits East US for Request A and West Europe for Request B, the session will be lost.

Solution: Externalize state. Use Azure Redis Cache (Enterprise tier allows active-geo replication) or just keep state in the database.
The Code: Ensure your application connects to the local region's database endpoint to minimize latency.

3. The Data Layer: Multi-Region Writes

This is the hardest part. In a traditional SQL setup, you usually have one Writer and multiple Readers. If the Writer region goes down, you have to promote a Reader, which takes time.

Enter Azure Cosmos DB. It supports Multi-Region Writes. You can write to East US and West Europe simultaneously, and Cosmos DB handles the replication magic in the background.
Consistency: For zero downtime, we usually accept "Session" consistency. It guarantees that a user sees their own writes immediately, which is enough for 99% of apps.

Infrastructure as Code (Terraform)

Let's look at how to define the critical Front Door configuration in Terraform. This snippet creates a backend pool that checks both regions.

```terraform
resource "azurerm_cdn_frontdoor_profile" "main" {
  name                = "example-frontdoor"
  resource_group_name = azurerm_resource_group.main.name
  sku_name            = "Premium_AzureFrontDoor" # Required for some advanced features
}

resource "azurerm_cdn_frontdoor_origin_group" "main" {
  name                       = "example-origin-group"
  cdn_frontdoor_profile_id = azurerm_cdn_frontdoor_profile.main.id

  load_balancing {
    sample_size                     = 4
    successful_samples_required     = 3
    additional_latency_in_milliseconds = 50
  }

  health_probe {
    path                = "/health"
    protocol            = "Https"
    interval_in_seconds = 100 # Aggressive probing for faster failover
    request_type        = "HEAD"
  }
}

# Add East US Origin
resource "azurerm_cdn_frontdoor_origin" "east_us" {
  name                          = "app-eastus"
  cdn_frontdoor_origin_group_id = azurerm_cdn_frontdoor_origin_group.main.id
  enabled                       = true
  host_name                     = azurerm_linux_web_app.east_app.default_hostname
  http_port                     = 80
  https_port                    = 443
  priority                      = 1
  weight                        = 1000
}

# Add West Europe Origin
resource "azurerm_cdn_frontdoor_origin" "west_eu" {
  name                          = "app-westeu"
  cdn_frontdoor_origin_group_id = azurerm_cdn_frontdoor_origin_group.main.id
  enabled                       = true
  host_name                     = azurerm_linux_web_app.west_app.default_hostname
  http_port                     = 80
  https_port                    = 443
  priority                      = 1
  weight                        = 1000
}

Best Practices & "Gotchas"

1. The Cost Factor

Warning: Multi-region is expensive. You are effectively doubling your compute and paying for cross-region data transfer.
Tip: Use auto-scaling rules. In "peace time," run your secondary region with minimal instances (maybe 2). Configure scale-out rules to handle the load only if the primary region fails or traffic spikes.

2. Conflict Resolution

If User A updates a record in the US, and User B updates the same record in Europe at the exact same millisecond, who wins?
Last Write Wins (LWW): The default in Cosmos DB. It uses the system clock. For most apps, this is fine.
Custom Merge: If you need complex logic, you'll need to write a stored procedure to merge the data.

3. Chaos Engineering

You cannot trust this architecture until you break it.
Action: Schedule a "Game Day." Manually turn off the Web App in the primary region during a low-traffic window. Watch your Front Door metrics. Does traffic shift automatically? Did anyone notice?

Community Corner

I’d love to hear how you handle data consistency in distributed systems!

Do you use SQL Failover Groups instead of Cosmos?
Have you ever had a failover that didn't work? What happened?
Drop a comment below. The best war story gets a shoutout in next week's newsletter!

FAQ

Q: Can I use Azure SQL Database for active-active?
A: Generally, no. Azure SQL is typically Active-Passive (one writer). You can use "Failover Groups" to automate the failover, but there is usually a short period (seconds to minutes) of write unavailability.

Q: What is the difference between Availability Zones and Regions?
A: Zones are separate datacenters within the same region (e.g., separate buildings in East US). They protect against fire/power failure in one building. Regions are hundreds of miles apart. They protect against natural disasters affecting a whole city.

Q: How do I handle scheduled deployments?
A: Deploy to one region at a time. Deploy to the "passive" or secondary region first, run smoke tests, and then deploy to the primary. This ensures you don't break the whole world simultaneously.

Q: Does Front Door add latency?
A: Actually, it often reduces it. Front Door uses "Anycast" to onboard user traffic at the nearest Microsoft Edge node (PoP) and rides the dedicated Microsoft backbone network to your app, avoiding the slower public internet.

Q: What is RTO and RPO?
A: RTO (Recovery Time Objective): How long you are down (Goal: Near zero). RPO (Recovery Point Objective): How much data you lose (Goal: Zero).

Conclusion

Building a Zero-Downtime Multi-Region architecture is an investment. It requires more complexity in your deployment pipelines and data handling. But the first time you sleep through a major regional outage because Front Door quietly shifted traffic to Europe, you’ll know it was worth every penny.

Next Step: Go to your Azure Portal right now and check your current application. Is it in a single region? If yes, map out what components (SQL, Storage, Compute) would need to change to support a second region. Start small—maybe just replicate your storage first!

MySQL HeatWave: Read Replicas for Scaling Read-Heavy Workloads

Fri, 28 Nov 2025 08:11:25 +0000

Read replicas are a powerful feature of MySQL HeatWave that enable you to scale read-heavy workloads by distributing queries across multiple database instances. As read-only copies of your source DB system, read replicas handle SELECT queries, analytics, and reporting workloads while the primary instance handles write operations. This architecture improves application performance, enhances availability, and supports disaster recovery strategies.

This guide covers read replica concepts, creation, connectivity, use cases, maintenance, and limitations.

What Are Read Replicas?

Read replicas are read-only copies of a MySQL HeatWave DB system that are automatically updated using asynchronous replication from the source DB system.

Key Characteristics:

Read-only: Cannot execute write operations (INSERT, UPDATE, DELETE)
Asynchronous replication: Data replicated from source with minimal lag
Full data copy: Contains complete copy of source DB system data
Independent endpoints: Each read replica has its own connection endpoint
Automatic lifecycle: Started, stopped, and deleted with source DB system

Benefits of Read Replicas

1. Redirecting Excess Traffic

Distribute read traffic across multiple instances to prevent primary instance overload.

Scenario: E-commerce application during peak shopping season

Primary handles transactions (orders, payments)
Read replicas serve product catalogs, inventory checks, search queries
Reduces primary instance load by 60-80%

2. Business Reporting and Analytics

Offload resource-intensive reporting queries to read replicas.

Scenario: Daily business intelligence reports

Complex JOIN queries with aggregations
Historical data analysis
Real-time dashboards
No impact on transactional performance

3. Geographic Distribution

Place read replicas closer to users in different regions.

Scenario: Global application

Primary in US-East
Read replica in EU-West for European users
Read replica in AP-South for Asian users
Reduced latency for read operations

4. Disaster Recovery

Read replicas serve as standby databases for failover scenarios.

Scenario: Primary DB system failure

Promote read replica to standalone DB system
Minimal data loss (replication lag only)
Faster recovery than backup restoration
Business continuity maintained

5. Development and Testing

Create isolated environments without impacting production.

Scenario: Testing new features

Read replica mirrors production data
Safe environment for experiments
No risk to production workload
Fresh data for realistic testing

Creating Read Replicas

Prerequisites

Before creating read replicas, ensure the following requirements are met:

1. OCI User Accounts with Required Privileges

Your OCI user must have permissions to:

Create and manage read replicas
Access the source DB system
Manage network resources

Required IAM Policy:

Allow group DatabaseAdmins to manage mysql-family in compartment ProductionDatabases

2. Operational DB System

An active MySQL HeatWave DB system must exist and be in ACTIVE state. You can create read replicas on standalone or high availability DB systems.

3. Compatible Shape

Read replicas are not supported on shapes that have less than 8 ECPUs or less than 4 OCPUs.

Supported Shapes:

ECPU: MySQL.8 (8 ECPUs), MySQL.16, MySQL.32, MySQL.64, MySQL.128, MySQL.256
OCPU (deprecated): MySQL.VM.Standard.E3.1.8GB (4 OCPUs) and higher

Note: OCPU shapes are deprecated for new users after September 5, 2024, and will be unavailable after March 13, 2026.

4. Network Configuration

Read replicas inherit the VCN and subnet from the source DB system by default, but you can specify a different subnet in the same VCN or different VCN.

Maximum Read Replicas

You can create a maximum of 18 read replicas per DB system. Each read replica has its own endpoint for connections.

Creating Read Replicas via OCI Console

Navigate to DB System
- Open OCI Console
- Go to Databases → HeatWave MySQL → DB Systems
- Select your source DB system
Access Read Replicas Section
- Under Resources, click Read replicas
- Click Create read replica
Configure Read Replica
- Display name: Descriptive name (e.g., "prod-analytics-replica")
- Description: Purpose description
- Shape: Inherit from source or override with different shape
- Configuration: Inherit from source or select custom configuration
- MySQL version: Inherit or override (must be same or higher)
- Networking: Select VCN and subnet (can differ from source)
- Delete protection: Enable to prevent accidental deletion
Review and Create
- Click Create read replica
- Read replica enters CREATING state
- Typically available in 15-30 minutes

Creating Read Replicas via OCI CLI

Generate configuration template:

oci mysql replica create --generate-full-command-json-input > replica-config.json

Edit configuration:

{
  "dbSystemId": "ocid1.mysqldbsystem.oc1...",
  "displayName": "analytics-read-replica",
  "description": "Read replica for BI workloads",
  "shapeName": "MySQL.16",
  "isDeleteProtected": true,
  "waitForState": ["SUCCEEDED"]
}

Create replica:

oci mysql replica create --from-json file://replica-config.json

Read Replica Properties

Inherited vs. Overridden Properties

By default, a read replica inherits shape, configuration, and version from the DB system. You can override these properties.

When Inherited:

Changing property on DB system also changes it on read replica
Replica follows source configuration automatically

When Overridden:

Changing property on DB system does NOT affect read replica
Replica maintains independent configuration

Overridable Properties:

Shape (e.g., use larger shape for heavy analytics)
Configuration (e.g., tune for read-heavy workload)
MySQL version (must be same or higher than source)

Automatic Read Replica Load Balancer

When you create the first read replica of a DB system, a read replica load balancer is created automatically, which distributes read traffic among the read replicas.

Load Balancer Features:

Distributes queries across all read replicas
Health checks ensure traffic goes to ACTIVE replicas
Automatic failover if replica becomes unavailable
No manual configuration required

Connecting to Read Replicas

Client applications cannot directly access read replicas from the internet. Connection methods mirror those for DB systems.

Connection Methods

1. Load Balancer (Recommended)

Use the automatically created read replica load balancer for balanced traffic distribution.

Load Balancer Endpoint:

<replica-load-balancer-ip>:3306

Benefits:

Automatic distribution across replicas
Health checks
Simplified application configuration
High availability

Important: When connecting via load balancer, user accounts require username only (no hostname restriction like 'user'@'%').

2. Bastion Session

Provides restricted and time-limited secure access.

Use Case: Temporary administrative access, troubleshooting

Setup:

Create bastion session in OCI Console
Session provides temporary SSH access
Connect to read replica from bastion
Session expires after defined period (30 minutes to 3 hours)

Example:

# Create bastion session
oci bastion session create-managed-ssh ...

# Connect through bastion
ssh -i <private_key> -o ProxyCommand="ssh -W %h:%p ..." replica_ip
mysql -h <REPLICA_IP> -u admin -p

3. Compute Instance

For long-term, secure access.

Setup:

Create compute instance in same VCN
Configure security lists to allow MySQL ports
Install MySQL client on compute instance
Connect to read replica endpoint

Example:

mysql -h <REPLICA_IP> -P 3306 -u admin -p

4. VPN Connection

For site-to-site connectivity from on-premises.

Options:

Site-to-Site VPN: IPSec VPN over internet
FastConnect: Dedicated private connection
OpenVPN: Client VPN access

5. Direct Application Connection

Configure application to use read replica endpoints.

Application Configuration:

# Primary for writes
primary_connection = mysql.connector.connect(
    host='primary_db_ip',
    user='app_user',
    password='password',
    database='production'
)

# Read replica for reads
replica_connection = mysql.connector.connect(
    host='replica_ip',
    user='app_user',
    password='password',
    database='production'
)

# Write operation
primary_connection.cursor().execute("INSERT INTO orders ...")

# Read operation
replica_connection.cursor().execute("SELECT * FROM products")

Monitoring Read Replicas

Read Replica Lag

Read replica lag measures how far behind the replica is from the source, measured by the read replica lag metric.

Factors Affecting Lag:

Workload on each read replica
CPU and memory resources
Network latency
Volume of writes on primary

Monitoring Lag:

-- On read replica
SHOW REPLICA STATUS\G

Check Seconds_Behind_Source value.

Creating Alarms

Set up OCI alarms to monitor read replica health:

Navigate to Monitoring service
Create alarm for ReplicaLag metric
Set threshold (e.g., > 60 seconds)
Configure notification topic
Add email or webhook destination

Alarm Query:

ReplicaLag[1m]{resourceDisplayName = "analytics-replica"}.mean() > 60

Automatic Recovery

MySQL HeatWave Service performs automatic recovery on any read replica that has failed.

Recovery Process:

Failure detected
Read replica set to UPDATING state
Cannot accept client connections during recovery
After success, set to ACTIVE state
Becomes operational again

Failed Beyond Recovery:
If replica cannot replicate from DB system anymore, it's set to NEEDS_ATTENTION state. Manual intervention required.

Read Replica Maintenance and Upgrades

Standalone DB System Upgrades

Read replicas are upgraded in parallel with source DB system. Both the source and all read replicas go offline for an update and upgrade at the same time.

Process:

Maintenance window starts
Source and all replicas enter UPDATING state
Upgrades applied simultaneously
All return to ACTIVE state together

Downtime: Brief period (typically 5-15 minutes depending on upgrade type)

High Availability DB System Upgrades

For HA DB systems, read replicas are upgraded one by one using rolling maintenance process. Only one read replica goes offline at a time.

Rolling Upgrade Process:

Replica 1 enters UPDATING state
Replica 1 upgraded, returns to ACTIVE
Replica 2 enters UPDATING state
Replica 2 upgraded, returns to ACTIVE
Continue for all replicas

Benefits:

Minimal impact on read capacity
At least N-1 replicas always available
Reduced performance impact

Duration: Longer total time but better availability

Read Replica Limitations

1. Shape Requirements

Read replicas are not supported on shapes that have less than 8 ECPUs or less than 4 OCPUs.

Unsupported:

MySQL.2 (2 ECPUs)
MySQL.4 (4 ECPUs)
MySQL.VM.Standard.E3.1.8GB (1 OCPU)
MySQL.VM.Standard.E3.1.16GB (2 OCPUs)

2. Load Balancer Account Name Restrictions

When connecting via load balancer, user accounts require username only ('user'@'%'), not hostname-specific restrictions.

Won't Work:

CREATE USER 'app_user'@'10.0.1.%' ...

Must Use:

CREATE USER 'app_user'@'%' ...

3. Outbound Replication Source

Read replicas cannot be configured as a source for outbound replication. Only the primary DB system can be a replication source.

4. IPv4 Subnet Support Only

Read replicas are only supported with IPv4 subnets. IPv6 subnets are not supported.

5. Bandwidth Limits

Read replicas support up to 8 Gbps bandwidth for client connections.

Implications:

High-throughput workloads may hit bandwidth limits
Monitor network metrics
Consider multiple replicas for very high throughput needs

6. Lifecycle Dependency

Read replicas are started when DB system is started, stopped when DB system is stopped, and deleted when DB system is deleted.

Cannot:

Start replica independently if source is stopped
Keep replica running when source is stopped
Preserve replica after deleting source

7. HeatWave Cluster Not Supported

Read replicas cannot have HeatWave clusters attached. If the source DB system has a HeatWave cluster, read replicas will not replicate HeatWave-specific data.

8. Configuration Changes

Prior to March 2024, configuration changes required deleting and recreating read replicas. As of March 26, 2024, you can now change shape and configuration of DB systems with read replicas without deleting replicas.

Best Practices

1. Use Multiple Read Replicas

For high availability and load distribution:

Minimum 2 replicas for redundancy
More replicas for higher read throughput
Maximum 18 replicas per DB system

2. Monitor Replication Lag

Set up alerts and regularly check:

SHOW REPLICA STATUS\G

If lag exceeds 60 seconds consistently:

Increase replica shape
Reduce read load
Optimize slow queries

3. Size Replicas Appropriately

Match replica shape to workload:

Light analytics: Same shape as source
Heavy reporting: Larger shape than source
Mixed workload: Multiple replicas with different shapes

4. Implement Read/Write Splitting in Applications

Direct writes to primary, reads to replicas:

def ex# Conclusionecute_query(query):
    if query.startswith('SELECT'):
        return replica_connection.execute(query)
    else:
        return primary_connection.execute(query)

5. Use Load Balancer for Simplicity

Automatic distribution and failover:

Configure application with load balancer endpoint only
No application changes when adding/removing replicas
Built-in health checks

6. Plan for Replication Lag

Design applications to tolerate eventual consistency:

Don't read immediately after write from replica
Use primary for read-after-write scenarios
Communicate lag tolerance to users

7. Test Failover Scenarios

Regularly test promoting replica to standalone:

Verify promotion procedure
Measure recovery time
Update application connection strings
Practice makes perfect for real disasters

8. Enable Delete Protection

For production replicas:

--is-delete-protected true

Prevents accidental deletion.

Read replicas are essential for scaling MySQL HeatWave beyond single-instance limitations. By distributing read traffic across multiple instances, you improve application performance, support business analytics, enable geographic distribution, and enhance disaster recovery capabilities.

Key takeaways:

Asynchronous replication: Data replicated with minimal lag
18 replicas maximum: Scale read capacity significantly
Shape requirement: Minimum 8 ECPUs or 4 OCPUs
Automatic load balancer: Simplifies traffic distribution
Multiple connection methods: Bastion, compute, VPN, load balancer
Rolling upgrades for HA: Minimal downtime during maintenance
Monitor lag: Critical for maintaining data freshness
Disaster recovery: Promote replica to standalone if needed

Whether supporting global applications, offloading analytics, or implementing disaster recovery, MySQL HeatWave read replicas provide the scalability and flexibility needed for modern database architectures.

MySQL HeatWave: Identifying and Testing Failover in High Availability DB Systems

Mon, 10 Nov 2025 10:01:57 +0000

In a high availability MySQL HeatWave DB system, automatic failover is a critical mechanism that ensures business continuity when the primary instance becomes unavailable. Understanding how to identify when a failover has occurred and how to properly test failover scenarios is essential for database administrators managing mission-critical applications.

This guide walks you through recognizing failover events through error messages, understanding the failover process, and implementing best practices for testing failover scenarios without disrupting production systems.

Understanding Failover in High Availability

What is Failover?

Failover is the automatic process where one of the secondary MySQL instances is promoted to become the new primary instance when the current primary fails or becomes unavailable. The promotion happens automatically without manual intervention, minimizing downtime and ensuring continuous availability.

Common Causes of Failover

Failover can be triggered by several scenarios:

Availability Domain Failure: Complete outage of the data center hosting the primary instance
Primary Database Instance Failure: Hardware failure, software crash, or critical system error on the primary instance
Network Connectivity Issues: Loss of network connectivity between the primary instance and the MySQL HeatWave Service management layer
Block Storage Issues: Persistent storage failures affecting the primary instance's ability to read or write data
Maintenance Activities: Certain planned maintenance operations that affect the primary instance

The Failover Process

When failover occurs:

MySQL HeatWave Service detects the primary instance failure
One of the secondary instances is automatically selected for promotion
The selected secondary is promoted to primary status
The DB system endpoint is reassigned to the newly promoted primary instance
The IP address remains the same, but all existing connections are closed
Client applications must reconnect to resume operations
If a HeatWave cluster was attached, it's automatically recreated and data is reloaded

Identifying a Failover: Error Messages and Indicators

Primary Indicators of Failover

1. Console Notification

The most straightforward way to verify if a failover has occurred is through the OCI Console:

Steps to Check:

Open the navigation menu
Select Databases → HeatWave MySQL → DB Systems
Choose your compartment from the List Scope
Click the name of your DB system to open the DB System Details page

If a failover has occurred, a message is displayed stating: "Current placement () differs from preferred placement, due to failover or maintenance activity"

This message indicates that the current primary instance is not in your originally preferred location due to automatic failover.

2. MySQL Event Notifications

When a failover happens, a MySQL - Automatic Recovery event is emitted on the DB system with the additionalDetails.isFailover property set to true

Configure event rules to receive notifications when these events occur.

Error Messages During Failover

When a failover is in progress or has just occurred, applications may encounter various error messages. Understanding these errors helps you identify failover events and implement appropriate retry logic.

Error 1290 (HY000): Read-Only Mode Error

Error Message:

ERROR 1290 (HY000): The MySQL server is running with the --super-read-only option 
so it cannot execute this statement

What It Means:
This error occurs when attempting to execute write operations (INSERT, UPDATE, DELETE, DDL statements) on an instance that is in super-read-only mode

When You'll See It:

During the brief window when a primary is being demoted to secondary
If your application still has connections to the old primary after failover
When attempting writes to a secondary instance

What to Do:

Implement connection retry logic in your application
Close existing connections and reconnect to get routed to the new primary
Ensure your application uses the DB system endpoint, not instance-specific IPs

Error 3100 (HY000): Replication Hook Error

Error Message:

ERROR 3100 (HY000): Error on observer while running replication hook 'before_commit'

What It Means:
This error indicates the system is starting the failover process, and transaction commits are being blocked or rolled back.

Context:
This error can also indicate transaction size errors when the message includes "Transaction of size X exceeds specified limit"

When You'll See It:

At the beginning of a failover event
When large transactions exceed the group replication transaction size limit
During the brief period when the primary instance is being transitioned

What to Do:

Transactions in progress will be rolled back automatically
Implement exponential backoff retry logic
Reconnect and retry the transaction
For transaction size errors, break large transactions into smaller batches

Error 2013 (HY000): Connection Lost During Query

Error Message:

ERROR 2013 (HY000): Lost connection to MySQL server during query

What It Means:
The connection to the MySQL server was lost while executing a query, typically because the primary instance became unavailable during failover.

When You'll See It:

During active failover when the primary instance goes offline
When long-running queries are interrupted by failover
During network disruptions affecting the primary

What to Do:

Close the broken connection
Establish a new connection to the DB system endpoint
Retry the query on the new primary
Implement idempotent query patterns where possible

Error Log Indicators

MySQL error logs provide detailed information about failover events. You can query the Performance Schema to view these logs.

Viewing Error Logs

Connect to your DB system using MySQL Shell or MySQL Client and run:

SELECT * FROM performance_schema.error_log 
WHERE SUBSYSTEM = 'Repl' 
ORDER BY LOGGED DESC 
LIMIT 50;

Key Error Codes Indicating Failover

MY-013213 and MY-011507: Primary Election Change

These errors indicate that a failover has occurred and a secondary instance has been promoted to primary through a Group Replication primary election change

MY-011608: Transaction Size Error

This error indicates a transaction size error where the transaction exceeds the group_replication_transaction_size_limit

MY-011566: Super Read-Only Mode Change

This error indicates that super_read_only has been switched off when an instance is promoted from secondary to primary, or switched on when demoted from primary to secondary.

What It Indicates:

An instance role change has occurred
The instance is transitioning between primary and secondary status
Part of the normal failover process

Additional Failover Indicators

Binary Log Position Differences

After a failover, the current binary log file name and position of the new primary may be different from the old primary

This is because binary logs of each instance are managed independently. Each transaction may be written to different binary log files and positions across instances.

To Check Binary Log Position:

SHOW MASTER STATUS;

Compare before and after failover to identify changes.

Connection String Behavior

While the DB system endpoint IP address remains constant during failover, all existing connections are terminated and must be re-established. Monitor your application's connection pool for sudden spikes in connection closures and new connection attempts.

Testing Failover: Process and Best Practices

Why Test Failover?

Regular failover testing is essential for:

Validating your disaster recovery procedures
Ensuring applications handle failovers gracefully
Training operations teams on recovery processes
Identifying configuration issues before real failures occur
Meeting compliance and audit requirements
Building confidence in your high availability architecture

Switchover vs. Failover

Failover: Automatic, unplanned promotion of a secondary to primary due to primary instance failure.

Switchover: Manual, planned promotion of a secondary to primary for testing, maintenance, or optimization purposes.

When you perform a switchover, the preferred placement and current placement change to the newly selected placement of the primary instance, and the DB system endpoint IP address does not change

Use switchover to safely test failover scenarios without actually causing a failure.

Step-by-Step Failover Testing Process

Phase 1: Set Up a Test DB System

Never test failover on production systems during business hours.

Create a Non-Production HA DB System
- Use a development or staging environment
- Configure it identically to production (same shape, configuration, HA settings)
- Ensure the system has high availability enabled
Import Representative Data
- Load a production-like dataset
- Include tables with primary keys (HA requirement)
- Consider data sensitivity and masking requirements
Deploy a Sample Application
- Use a test application that mimics production workload
- Implement connection pooling and retry logic
- Add monitoring and logging capabilities
- Configure the application to use the DB system endpoint

Phase 2: Establish Baseline Monitoring

Before triggering switchover, establish baseline metrics:

Application Performance Metrics
- Query response times
- Transaction throughput
- Connection pool statistics
- Error rates
Database Metrics
- Active connections
- Current binary log position
- Replication lag (should be minimal in HA systems)
- Current primary instance location
System Health Indicators
- CPU and memory utilization
- Storage I/O metrics
- Network latency between availability domains

Phase 3: Execute Switchover

Switchover causes a short period of downtime while the primary instance is redirected to the newly promoted instance, requiring all database connections to be reopened

Using the Console:

Navigate to your DB System details page
Click the Actions menu
Select Switchover
Choose the target availability domain or fault domain (different from current)
Confirm the switchover operation

Using OCI CLI:

oci mysql db-system update \
  --db-system-id <YOUR_DB_SYSTEM_OCID> \
  --availability-domain <TARGET_AD> \
  --force

Modern Switchover Behavior (for systems created/upgraded after February 25, 2025):

New transactions are blocked when switchover is initiated, running transactions are allowed to complete, and uncommitted transactions are rolled back when the connection to the previous primary is broken

Phase 4: Monitor the Switchover

During Switchover (typically 30-60 seconds):

Watch Application Behavior
- Monitor for connection errors (Error 1290, 2013, 3100)
- Observe connection pool drain and refill
- Track any failed transactions
- Verify automatic retry mechanisms

Check Error Logs

SELECT * FROM performance_schema.error_log 
WHERE LOGGED > DATE_SUB(NOW(), INTERVAL 5 MINUTE)
ORDER BY LOGGED DESC;

Verify HeatWave Cluster Status (if applicable)
When current placement changes in a switchover, HeatWave cluster is detached from the previous primary instance and either reused (same AD) or deleted and recreated (different AD)
```
SHOW GLOBAL STATUS LIKE 'rapid_plugin_bootstrapped';
```

After Switchover:

Verify New Primary Location
- Check OCI Console for current placement
- Confirm switchover to target AD/FD succeeded
- Verify "Current placement" matches target
Validate Application Recovery
- All connections successfully re-established
- Transaction throughput returns to baseline
- Query performance is consistent
- No persistent error conditions
Check Data Consistency
- Verify recent transactions completed successfully
- Confirm no data loss occurred
- Test read and write operations
- Validate application state consistency

Phase 5: Document and Analyze

Key Metrics to Record:

Metric	Before	During	After
Total Downtime	N/A	___ sec	N/A
Connections Dropped	0	___	0
Failed Transactions	0	___	0
Recovery Time	N/A	___ sec	N/A

Analysis Questions:

Did the application handle failover gracefully?
Were retry mechanisms effective?
Did any transactions fail permanently?
How quickly did the system return to normal operation?
Were there any unexpected issues?

Switchover to Return to Preferred Placement

After testing, you can switch back to the original preferred placement:

Option 1: Switchover Back
Perform another switchover to return the primary to the original AD/FD.

Option 2: Update Preferred Placement
You can switchover to the existing primary instance to change the preferred placement to the current placement with no downtime

Best Practices for Handling Failover

1. Test on Non-Production Systems First

Critical Rule: Never test failover on production systems during business hours or peak traffic periods.

Why This Matters:

Switchover causes brief downtime (30-60 seconds)
Connections are forcibly closed
Uncommitted transactions are rolled back
Long-running queries may be aborted

Best Practice:

Maintain a dedicated test environment that mirrors production
Perform regular failover drills quarterly or bi-annually
Test during maintenance windows for production validation
Document all test results and improvements made

2. Use Switchover to Simulate Failover

Recommendation: Use manual switchover operations to test disaster recovery procedures rather than artificially creating failures.

Benefits:

Controlled, predictable process
Ability to schedule during low-traffic periods
Minimizes risk of actual data loss
Provides clean before/after states for analysis

Testing Scenarios:

Same AD, Different Fault Domain: Minimal impact, tests intra-datacenter failover
Different AD: Simulates datacenter failure, tests cross-AD latency and behavior
With Active HeatWave Cluster: Tests analytics workload recovery
Under Load: Tests failover during peak transaction volume

3. Implement Robust Application-Level Retry Logic

Connection Retry Pattern:

import time
import mysql.connector
from mysql.connector import Error

def connect_with_retry(max_retries=5, base_delay=1):
    for attempt in range(max_retries):
        try:
            connection = mysql.connector.connect(
                host='<DB_SYSTEM_ENDPOINT>',
                database='your_database',
                user='admin',
                password='your_password',
                connect_timeout=10
            )
            return connection
        except Error as e:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt)
            print(f"Connection failed: {e}. Retrying in {delay}s...")
            time.sleep(delay)

Key Principles:

Always use the DB system endpoint, never instance-specific IPs
Implement exponential backoff (1s, 2s, 4s, 8s, 16s)
Limit retry attempts to avoid infinite loops
Make transactions idempotent when possible
Log all retry attempts for monitoring

4. Move Primary Back After Unplanned Failover

After an automatic failover occurs in production:

The current placement differs from the preferred placement, and a message is displayed indicating "Current placement differs from preferred placement, due to failover or maintenance activity"

Decision Points:

Option A: Leave in New Location

System is stable and operational
Performance is acceptable
No immediate reason to change

Option B: Switchover Back to Preferred Location

Original location has specific advantages
Schedule switchover during next maintenance window
Perform during low-traffic period

Best Practice: Don't rush to move back immediately after failover. Validate the stability of the current state first, then plan a controlled switchover during a maintenance window.

5. Configure Connection Timeouts Appropriately

Recommended Settings:

SET GLOBAL connect_timeout = 10;
SET GLOBAL interactive_timeout = 28800;
SET GLOBAL wait_timeout = 28800;

For Applications Using PrivateLink:

Set shorter connect_timeout values (5 seconds)
Implement aggressive retry logic
Failovers and switchovers take longer when using Query PrivateLink endpoints

6. Test with HeatWave Cluster

If your DB system has a HeatWave cluster attached:

Additional Considerations:

Cluster Behavior During Switchover
- If new primary is in the same AD as previous primary, the existing HeatWave cluster is reused and reattached
- If new primary is in a different AD, the existing HeatWave cluster is deleted and a new one created
Data Reload
- Tables are automatically reloaded from HeatWave Storage Layer
- Monitor reload progress: SHOW GLOBAL STATUS LIKE 'rapid%';
Long-Running Queries
- When current placement changes in a switchover, long-running queries offloaded to HeatWave cluster can get aborted
- Reconnect and rerun queries after cluster is ready

Key Takeaways

Identifying and testing failover in MySQL HeatWave high availability DB systems is a critical skill for database administrators and DevOps teams. By understanding the error messages that indicate failover, implementing proper monitoring, and regularly testing switchover scenarios in non-production environments, you can ensure your applications handle failover gracefully with minimal disruption.

Key takeaways:

Recognize failover indicators: Watch for specific error codes (1290, 3100, 2013) and console messages
Use switchover for testing: Never test on production; use manual switchover to simulate failover safely
Monitor thoroughly: Track error logs, transaction status, and application behavior
Implement retry logic: Applications must handle connection failures and transaction rollbacks gracefully
Plan recovery: After automatic failover, assess whether to switch back during a maintenance window

MySQL HeatWave: Creating High Availability DB Systems and Understanding Limitations

Fri, 07 Nov 2025 14:01:41 +0000

MySQL HeatWave: Creating High Availability DB Systems and Understanding Limitations

Introduction

MySQL HeatWave offers enterprise-grade high availability features that ensure your database remains operational even during hardware failures or maintenance events. Understanding how to properly configure high availability and its associated limitations is crucial for designing resilient database architectures.

This guide walks you through creating a high availability DB system and explores the important limitations you should consider when planning your deployment.

Creating a High Availability DB System

Overview of the Creation Process

Creating a high availability DB system in MySQL HeatWave follows a structured workflow with three main phases:

Set Up Basic DB System Configuration
Specify Database and Infrastructure Details
Configure Backup and Advanced Options

Let's explore each phase in detail.

Phase 1: Set Up Basic DB System Configuration

Step 1: Access the Console

Step 2: Navigate to DB Systems

Open the navigation menu
Under Databases, select HeatWave MySQL (OCI) or MySQL HeatWave (AWS)
Click DB Systems
Click Create DB System

Step 3: Choose Deployment Template

Select the Production template, which automatically configures your system for high availability with recommended settings for production workloads.

Alternatively, you can start with Development or Testing and manually enable high availability later.

Step 4: Basic Configuration

Compartment

Select the compartment where the DB system will reside
Compartments help organize resources and control access

Name Your DB System

Display Name: Provide a meaningful name (e.g., "prod-mysql-ha-primary")
Description: (Optional) Add details about the system's purpose

Database System Type

Ensure "High Availability" is selected if using the Production template
For manual setup, you can enable this option here

Phase 2: Specify Database Information

Administrator Credentials

Username Configuration

Specify the administrator username (default is typically "admin")
Avoid reserved usernames like "root", "mysql", "sys"
Username can be up to 32 characters

Password Requirements

Must be 8-32 characters long
Include at least one:
- Uppercase letter
- Lowercase letter
- Number
- Special character from: ,.~-+*;:_!
Store credentials securely (consider OCI Vault or AWS Secrets Manager)

Network Configuration

Virtual Cloud Network (VCN)

Select an existing VCN or create a new one
Subnet: Choose a private subnet for enhanced security
Public subnets are not recommended for production databases

Availability Domain Selection

Choose the preferred availability domain for your primary instance
This becomes your preferred placement
The system will automatically distribute secondary instances across different ADs or fault domains

Security Configuration

Ensure security lists or network security groups allow:
- Port 3306 for MySQL connections
- Port 33060 for MySQL X Protocol
Restrict source CIDR blocks to known application IP ranges

Hardware Configuration (Shape)

Select DB System Shape

Choose a shape that meets your performance requirements:

ECPU Shapes (Recommended)

Architecture-agnostic and future-proof
Examples: MySQL.8, MySQL.16, MySQL.32, MySQL.64, MySQL.128, MySQL.256
Can be changed after deployment
Support high availability and HeatWave clusters

Shape Characteristics

Defines CPU count, memory, and network bandwidth
Higher shapes provide better performance but cost more
Use Autopilot Shape Advisor for recommendations based on workload

Important: All OCPU shapes are deprecated and unavailable to new users created after September 5, 2024. Existing OCPU shapes will be unavailable after March 13, 2026.

Storage Configuration

Data Storage Size

Minimum: 50 GB
Maximum: 131,072 GB (128 TB)
Plan for current data plus growth projections
Storage can only be increased, never decreased

Automatic Storage Expansion

Recommended: Enable this feature
Automatically increases storage when usage approaches limits
Prevents downtime due to storage exhaustion
Based on Health Monitor read-only thresholds

Storage Performance

Higher storage allocations provide better IOPS
Storage is backed by high-performance block storage

Phase 3: Configure Backups

Enable Automatic Backups

Why Backups Matter

Protects against data loss
Enables point-in-time recovery
Required for disaster recovery planning

Backup Configuration Options

Enable Automatic Backups

Toggle: ON (highly recommended for HA systems)
Backups run automatically at scheduled times

Backup Retention Period

Range: 1 to 35 days
Recommended: At least 7 days for production
Longer retention = higher storage costs but better recovery options

Backup Window Selection

Choose a time when database activity is lowest
Typically during off-peak hours (e.g., 2:00 AM - 4:00 AM)
Avoid peak transaction periods to minimize performance impact

Point-in-Time Recovery (PITR)

Enable to restore to any point within retention period
Uses binary logs for granular recovery
Essential for recovering from accidental data modifications

Regional Backup Copy (OCI only)

Schedule automatic backup copies to another OCI region
Provides geographic redundancy
Protects against regional disasters

Advanced Backup Options

Backup Validation

Verifies backup integrity
Ensures backups are restorable
Available on supported MySQL versions

Soft Delete for Backups

Enabled by default
Backups remain in DELETED state before permanent deletion
Provides grace period for recovery of accidentally deleted backups

Additional Configuration Options

HeatWave Cluster Configuration (Optional)

If you need analytics capabilities:

Enable HeatWave Cluster

Checkbox: Select to add analytics processing
Choose HeatWave node shape (HeatWave.512GB, HeatWave.256GB, HeatWave.128GB)
Specify number of nodes (1 to 512 depending on shape and region)

Important HA Consideration: On OCI, HeatWave clusters are fully supported with high availability. On AWS, HeatWave features (including Lakehouse, GenAI, and AutoML) are currently not supported on HA DB systems.

MySQL Configuration

Configuration Selection

Use default HA-compatible configuration (recommended)
Configurations with "HA" suffix support high availability
Custom configurations must be based on HA-compatible templates

Configuration Characteristics

Defines system variables and initialization parameters
Can be changed after deployment with rolling restarts
Affects memory allocation, connection limits, and performance tuning

Tagging (Optional)

Resource Tags

Add metadata for organization
Enable cost tracking and allocation
Support automation and bulk operations

Tag Types

Defined Tags: Created by administrators with controlled vocabularies
Free-form Tags: Simple key-value pairs for flexible labeling

Finalizing Creation

Review Configuration

Before creating:

Verify all settings are correct
Check estimated monthly cost
Ensure network connectivity is properly configured
Confirm backup settings meet compliance requirements

Create DB System

Click Create to provision your high availability DB system.

Provisioning Timeline

DB system enters CREATING state
Typical creation time: 15-30 minutes for HA systems
Three instances are provisioned across availability/fault domains
Status changes to ACTIVE when ready

What Happens During Creation

Primary instance is provisioned in preferred placement
Two secondary instances created in different domains
MySQL Group Replication is configured automatically
Initial backup is taken (if enabled)
Monitoring and health checks are activated

Understanding High Availability Limitations

While MySQL HeatWave high availability provides robust fault tolerance, it's important to understand its limitations to design systems appropriately.

1. Rolling Upgrades and Brief Downtime

Characteristic: High availability DB systems require rolling upgrades for maintenance, during which there is a brief period of downtime before the newly promoted MySQL instance resumes connections.

What This Means:

During maintenance, instances are upgraded one at a time
When the primary instance is upgraded, a failover occurs
Connection strings remain the same, but applications must reconnect
Downtime is typically measured in seconds, not minutes

Best Practice:

Implement connection retry logic in applications
Use connection pooling with automatic reconnection
Schedule maintenance during low-traffic periods
Test application behavior during failovers

2. MySQL Version Requirements

Limitation: High availability is only supported for MySQL version 8.0.24 or higher.

Implications:

Older MySQL versions cannot use HA features
Must upgrade to 8.0.24+ before enabling high availability
Consider upgrading to MySQL 8.4.x (LTS) or 9.x (Innovation) for latest features

Migration Path:

Upgrade standalone DB system to 8.0.24 or higher
Ensure all prerequisites are met (primary keys, etc.)
Enable high availability after upgrade completes

3. Direct Access Limitations

Limitation: You can only access the primary MySQL instance directly; the two secondary instances cannot be accessed using MySQL Shell or any other client.

Why This Matters:

Cannot distribute read load across secondary instances for HA systems
All connections go through the primary instance
Read replicas are separate feature for read scaling (not part of HA)

Alternative Solutions:

Use Read Replicas for read scaling (separate from HA)
Read replicas can be created on HA or standalone systems
Offload analytics queries to HeatWave cluster instead

4. Transaction Size Limits Based on Shape Memory

Limitation: The maximum transaction size depends on the amount of memory of the shape.

Transaction Size Limits by Shape:

Smaller shapes have lower transaction size limits
Larger transactions require more memory for Group Replication
Exceeding limits causes transaction failure

Example Limits (approximate):

MySQL.8 (8 ECPU, 128 GB RAM): ~2 GB transaction limit
MySQL.16 (16 ECPU, 256 GB RAM): ~4 GB transaction limit
MySQL.32 (32 ECPU, 512 GB RAM): ~8 GB transaction limit

Workarounds:

Break large transactions into smaller batches
Use bulk loading features with smaller commit sizes
Select appropriate shape for your transaction patterns
Consider upgrading shape if hitting limits regularly

5. HeatWave Cluster Operations and HA Changes

Limitation: If a HeatWave cluster load or reload operation is ongoing, enabling or disabling high availability can fail.

Why This Happens:

HeatWave cluster operations lock resources during data loading
HA changes require cluster reconfiguration
Concurrent operations can cause conflicts

Best Practice:

Wait for HeatWave operations to complete before HA changes
Monitor HeatWave operation status
Schedule HA changes during maintenance windows
Avoid simultaneous infrastructure and data operations

6. HeatWave Lakehouse Compatibility

Previous Limitation: Prior to MySQL 8.3.0-u2, you could not enable high availability on a DB system with HeatWave Lakehouse enabled.

Current Status: This limitation has been lifted in MySQL 8.3.0-u2 and later versions.

Upgrade Path:

Systems on older versions must upgrade to use HA with Lakehouse
After upgrade, both features can coexist
On OCI, full support is available
On AWS, HeatWave features remain unsupported on HA systems

7. Platform-Specific Limitations

MySQL HeatWave on AWS

On AWS, MySQL HeatWave Clusters and features requiring a cluster (MySQL HeatWave Lakehouse, GenAI, AutoML) are not currently supported on high availability DB systems.

Impact:

HA systems on AWS are limited to OLTP workloads
Analytics features require standalone systems
Must choose between HA and advanced analytics on AWS

Query PrivateLink Considerations

When using Query PrivateLink endpoints, failovers and switchovers take longer than on public endpoints.

Mitigation:

Set connect_timeout to small value (e.g., 5 seconds)
Implement aggressive connection retry logic
Monitor failover times
Consider public endpoints for latency-sensitive applications

8. Configuration and Infrastructure Limitations

Bulk Ingest: The Bulk Ingest feature is not supported on high availability DB systems.

Alternative: Use standard data loading methods like LOAD DATA INFILE or MySQL Shell utilities.

PrivateLink Management: High availability cannot be enabled or disabled on a DB system with PrivateLink(s) configured.

Workaround: Delete PrivateLinks, make HA changes, then recreate PrivateLinks.

9. Rolling Restart for Configuration Changes

Behavior: When updating configuration of a high availability DB system that changes only dynamic variables, a rolling restart is performed, which shortens downtime and causes a change in current primary placement.

What Happens:

Configuration updated on primary instance
Controlled failover to secondary instance
Original primary updated and becomes secondary
System automatically rebalances if needed
HeatWave data reloaded automatically if cluster attached

Implications:

Brief connection interruption during rolling restart
Current placement may differ from preferred placement
Applications must handle reconnections
HeatWave queries may be briefly unavailable during reload

Best Practices for HA DB Systems

1. Application Design

Connection Management

Implement exponential backoff for reconnection attempts
Use connection pooling with health checks
Set appropriate timeout values
Handle transient errors gracefully

Transaction Design

Keep transactions small and fast
Avoid long-running transactions that span minutes
Break large data modifications into batches
Monitor transaction sizes against shape limits

2. Monitoring and Alerting

Key Metrics to Monitor

Replication lag between primary and secondary instances
Storage utilization and growth trends
Transaction throughput and latency
Failover events and frequency
Backup success rates

Set Up Alerts For

Failover events
High replication lag
Storage approaching limits
Backup failures
Configuration change events

3. Testing and Validation

Regular Testing

Perform manual switchovers during maintenance windows
Test application behavior during failovers
Validate backup and restore procedures
Verify monitoring and alerting systems

Chaos Engineering

Simulate instance failures
Test network partition scenarios
Validate automatic recovery mechanisms
Document recovery procedures

4. Maintenance Planning

Scheduling

Plan configuration changes during low-traffic periods
Coordinate with application teams for maintenance windows
Avoid changes during critical business periods
Document rollback procedures

Pre-Change Validation

Test configuration changes in non-production first
Verify compatibility with HA requirements
Ensure backups are current
Have rollback plan ready

5. Cost Optimization

Resource Planning

Start with appropriate shape size
Monitor utilization with Autopilot Shape Advisor
Adjust shapes based on actual workload patterns
Balance performance needs with cost constraints

Backup Management

Set retention periods based on compliance requirements
Use regional backup copies only when needed
Enable backup validation selectively
Clean up old backups per policy

When NOT to Use High Availability

While high availability is excellent for many use cases, it may not be appropriate when:

Development/Testing: Standalone systems are more cost-effective
Analytics-Heavy on AWS: HA doesn't support HeatWave features on AWS
Very Large Transactions: Shape memory limits may be restrictive
Budget Constraints: HA requires 3x compute resources vs standalone
Read-Heavy Workloads: Read replicas might be more appropriate than HA

Alternative High Availability Strategies

If MySQL HeatWave HA limitations are too restrictive:

Read Replicas for Read Scaling

Distribute read traffic across multiple instances
Works with both standalone and HA systems
Up to 10 read replicas per DB system
Better for read-heavy workloads

Multi-Region Deployment

Deploy separate DB systems in different regions
Use application-level failover logic
Replicate data using MySQL replication channels
Provides geographic redundancy

Hybrid Architectures

Use HA for OLTP workloads
Separate standalone system with HeatWave for analytics (AWS)
Replicate data between systems using channels
Best of both worlds but adds complexity

Conclusion

MySQL HeatWave high availability provides enterprise-grade fault tolerance with automatic failover, data protection, and minimal downtime. Understanding the limitations—particularly around rolling upgrades, transaction sizes, and platform-specific constraints—is essential for designing robust database architectures.

By following best practices for application design, monitoring, and maintenance, you can build resilient systems that meet stringent uptime requirements while staying within the boundaries of high availability limitations.

The key is matching your requirements to the capabilities: if you need maximum uptime for OLTP workloads with automatic failover, MySQL HeatWave HA is an excellent choice. If you need advanced analytics features on AWS or have very specific requirements, consider alternative architectures that better fit your needs.

Additional Resources:

MySQL HeatWave: Creating DB Systems and High Availability Setup

Tue, 04 Nov 2025 11:52:49 +0000

MySQL HeatWave: Creating DB Systems and High Availability Setup

Introduction

MySQL HeatWave is Oracle's fully managed database service that combines transactional and analytical workloads in a single platform. This guide walks you through creating a MySQL HeatWave DB System and implementing high availability for enterprise-grade reliability.

Creating a MySQL HeatWave DB System

Step 1: Access OCI Console

Step 2: Navigate to DB Systems

Open the navigation menu (hamburger icon)
Under Databases, select HeatWave MySQL
Click DB Systems from the menu

Step 3: Initiate DB System Creation

Click the Create DB System button. This opens the Create DB System dialog with several configuration options.

Step 4: Choose a Template

Oracle provides three templates to streamline setup:

Production: Creates a high availability DB system with recommended settings for production environments
Development or Testing: Creates a standalone DB system optimized for dev/test workloads
Always Free: Creates a standalone system with limited resources (50GB storage, MySQL.Free shape) suitable for learning and small projects

Step 5: Configure Basic Information

Compartment Selection

Choose the compartment where you want to create the DB system
Compartments help organize and manage access to resources

System Identification

Name: Provide a display name for your DB system (doesn't need to be unique)
Description: (Optional) Add details about the system's purpose
OCID: Oracle automatically generates a unique identifier

Step 6: Configure Database Credentials

Administrator Credentials

Username: Specify the administrator username (avoid reserved usernames)
Password: Create a strong password that meets security requirements
Important: Save these credentials securely – you'll need them to connect to the database

Step 7: Configure Networking

Virtual Cloud Network (VCN)

Select an existing VCN or create a new one
Choose the private subnet for the DB system (recommended for security)
Availability Domain: Select the AD where the primary instance will run
Fault Domain: Oracle automatically selects an appropriate fault domain

Configure Security List

Ensure your VCN's security list allows MySQL traffic:

Source CIDR: 0.0.0.0/0 (or restrict to specific IP ranges)
Destination Ports: 3306 (MySQL) and 33060 (MySQL X Protocol)

Step 8: Select Shape

Choose a shape that defines your system's compute resources:

ECPU Shapes (Recommended as of 2024): Architecture-agnostic shapes that can be changed after deployment
OCPU Shapes: Being deprecated for new users (sunset date: March 13, 2026)
Examples: MySQL.8, MySQL.16, MySQL.32, MySQL.HeatWave.VM.Standard

The shape determines CPU count, memory, and network bandwidth. You can scale up or down later based on workload demands.

Step 9: Configure Storage

Initial Data Storage Size

Specify storage in GB (minimum: 50 GB, maximum: 131,072 GB)
Consider your current data size plus growth projections
Storage can be increased later but not decreased

Automatic Storage Expansion

Enable this feature to automatically grow storage when space usage approaches the Health Monitor read-only limits
Helps prevent storage-related outages

Step 10: Configure HeatWave Cluster (Optional)

Enable HeatWave Cluster

Select this option to add analytics capabilities
Configure the number of HeatWave nodes (minimum: 1, maximum: depends on shape and region)
Choose the HeatWave node shape

MySQL HeatWave Lakehouse

Enable to query data in Object Storage (CSV, Parquet, Aurora/Redshift backups)
Note: Prior to MySQL 8.3.0-u2, enabling Lakehouse required disabling HA, point-in-time recovery, and read replicas

Step 11: Configure Backup Plan

Backup Settings

Enable automatic backups: Recommended for production
Backup retention period: Choose from 1 to 35 days
Point-in-time recovery: Enables restoration to any point within the retention window
Backup window: Select a time when database activity is lowest

Step 12: Advanced Options (Optional)

Configuration

Apply a custom configuration or use the default
High availability requires an HA-compatible configuration (configurations with "HA" suffix)

Hostname Label

Define a DNS hostname for your DB system (requires DNS hostnames enabled in VCN)

Tags

Add metadata for organization and cost tracking

Step 13: Create the DB System

Review all settings and click Create. The DB system enters the CREATING state and typically becomes available in 15-30 minutes.

Understanding High Availability in MySQL HeatWave

High availability is a critical feature for production workloads that require maximum uptime and fault tolerance.

What is High Availability?

A high availability DB system consists of three MySQL instances: a primary instance and two secondary instances, with the primary functioning as a read/write endpoint and secondary instances receiving asynchronous data replication from the primary.

Key Benefits

Business Continuity: Three MySQL instances distributed across different availability or fault domains
Minimal Downtime: Automatic failover promotes a secondary instance to primary when the primary fails, resuming availability with no data loss
Flexible Placement: Choose your preferred availability domain and fault domain for the primary instance
Uptime SLA: Supports 99.99% availability requirements
HeatWave Compatible: You can enable HeatWave cluster on high availability DB systems

Architecture Components

Primary Instance

Handles all write operations and read queries
Located in your chosen availability domain
Provides the read/write endpoint with a consistent IP address

Secondary Instances (2)

Receive replicated data from the primary asynchronously
Placed in different availability or fault domains from the primary
Ready to be promoted to primary during failover

Placement Models

There are two deployment models:

Multiple Availability Domains with Regional Subnet: Three instances spread across three different ADs (AD1, AD2, AD3)
Single Availability Domain with Multiple Fault Domains: All three instances in one AD but different fault domains

Replication Mechanism

MySQL HeatWave uses MySQL Group Replication for high availability, with primary keys required as a prerequisite for the group replication used by high availability. Data is copied from the primary to secondary instances over a secure, managed internal network.

Important: The binary logs of each MySQL instance are managed independently, meaning even though instances have the same data, they can have different binary log files with different names and sizes.

Prerequisites for Enabling High Availability

Before enabling high availability on a DB system, ensure the following requirements are met:

1. MySQL Version Requirement

MySQL version must be 8.0.24 or higher to enable high availability. If using an older version, upgrade before enabling HA.

2. Primary Keys on All Tables

If you are migrating data to MySQL HeatWave and have not defined primary keys on your tables, you must add them.

Why Primary Keys Matter

Required for MySQL Group Replication
Ensure data consistency across instances
Impact load performance if composite or non-INTEGER

How to Check and Add Primary Keys

Use MySQL Shell to identify tables without primary keys:

SELECT DISTINCT
    table_schema,
    table_name
FROM information_schema.columns
WHERE table_schema NOT IN ('information_schema', 'mysql', 'performance_schema', 'sys')
AND table_name NOT IN (
    SELECT table_name
    FROM information_schema.table_constraints
    WHERE constraint_type = 'PRIMARY KEY'
);

Adding Primary Keys Using Invisible Columns

Using invisible columns to add primary keys is a low-impact way to update existing data for use with a high availability DB system:

ALTER TABLE your_table 
ADD COLUMN id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY INVISIBLE;

3. Crash Recovery Enabled

Verify crash recovery is enabled on your DB system. This is typically enabled by default but can be checked in the configuration settings.

4. Inbound Replication Stopped

If you intend to configure an inbound replication channel on this DB system, you must import data before enabling high availability and configure your channel after high availability is enabled.

5. Compatible Configuration

All default configurations that are compatible with high availability have the HA suffix in the configuration name. If using a custom configuration, copy it from an existing HA-compatible configuration.

6. HeatWave Cluster Considerations

If a HeatWave cluster load or reload operation is ongoing, enabling or disabling high availability can fail. Wait for these operations to complete before making HA changes.

Enabling High Availability on an Existing DB System

Using the Console

Navigate to your DB System details page
From the Actions menu, select Enable high availability
Review the prerequisites dialog
Click Enable high availability

The process creates two secondary instances by cloning from the primary instance and does not cause any downtime.

Using OCI CLI

oci mysql db-system update \
  --db-system-id <DB_SYSTEM_OCID> \
  --is-highly-available true

Note: If the configuration is not compatible with high availability, you must first change the DB system to use a high availability compatible configuration before enabling it.

Understanding Failover and Switchover

Automatic Failover

When the primary instance becomes unavailable due to hardware failure, network issues, or other problems, MySQL HeatWave Service automatically initiates a failover.

Failover Process:

Service detects primary instance failure
One secondary instance is automatically promoted to primary
The IP address of the read/write endpoint does not change, regardless of the placement of the primary instance
Applications must reconnect, but the endpoint remains the same
No data loss occurs

For a high availability DB system with HeatWave cluster, if the primary instance fails, HeatWave Service deletes the HeatWave cluster attached to the failed instance and adds a new one to the new primary instance, with tables automatically loaded.

Post-Failover State:

The current placement changes to the new primary's location
The preferred placement (your original choice) remains unchanged
A message appears: "Current placement differs from preferred placement, due to failover"

Manual Switchover

You can manually select one of the MySQL instances as the primary instance through switchover, which changes both the preferred placement and current placement to the newly selected placement.

When to Use Switchover:

Planned maintenance on the current primary's availability domain
Testing disaster recovery procedures
Rebalancing after a failover to restore preferred placement

Switchover Process (for systems created/upgraded after February 25, 2025):

New transactions are blocked when switchover is initiated, running transactions are allowed to complete, and the switchover process waits until all running transactions finish and commit.

Navigate to DB System details page
Click Switchover from the Actions menu
Select the target availability domain or fault domain
Confirm the switchover

When the current placement changes in a switchover, it results in a short period of downtime while the primary instance is redirected to the newly promoted instance, and all database connections must be reopened.

Using CLI:

oci mysql db-system update \
  --db-system-id <DB_SYSTEM_OCID> \
  --availability-domain <TARGET_AD>

Important Notes:

Uncommitted transactions are rolled back when the connection to the previous primary instance is broken
Long-running queries offloaded to HeatWave cluster may be aborted
After a switchover, the current binary log file name and position of the new primary may be different from the old primary

HeatWave Cluster with High Availability

You can enable HeatWave cluster on a high availability DB system for combined OLTP and analytics capabilities.

HeatWave Behavior During Placement Changes

Same Availability Domain:
If both the previous primary instance and the new primary instance are located in the same availability domain, the existing HeatWave cluster can be reused by detaching from the previous primary and reattaching to the new primary.

Different Availability Domains:
If the previous primary and new primary are in different availability domains, the existing HeatWave cluster must be detached and deleted, with a new HeatWave cluster created in the same AD as the new primary instance.

Data Recovery

The data in the HeatWave cluster is automatically recovered from:

The HeatWave Storage Layer (fastest option)
Reloaded from the DB system
Reloaded from Lakehouse Object Storage

This ensures minimal disruption to analytics workloads even during failover scenarios.

Disabling High Availability

If you need to convert a high availability DB system back to standalone:

Disabling high availability creates a standalone DB system from the instance running in the preferred primary placement, with no downtime if the primary is already in the preferred placement, but a controlled failover and short downtime if it is not.

Console Method:

Navigate to DB System details
Actions menu → Disable high availability
Confirm the action

CLI Method:

oci mysql db-system update \
  --db-system-id <DB_SYSTEM_OCID> \
  --is-highly-available false

Tip: You can avoid downtime by switching the preferred primary placement to the current primary instance before disabling high availability.

Monitoring and Observability

Health Monitoring

MySQL HeatWave provides built-in monitoring:

Instance health status
Storage utilization
Connection metrics
Replication lag (for HA systems)

Event Notifications

When a failover happens, a MySQL - Automatic Recovery event is emitted on the DB system with the additionalDetails.isFailover property set to true.

Configure operational notifications to receive alerts for:

Failover events
Storage threshold warnings
Backup completion/failure
Maintenance notifications

Integration with Monitoring Services

OCI Monitoring: Native integration with OCI Monitoring service
AWS CloudWatch: Available for MySQL HeatWave on AWS
Custom Metrics: Export metrics to third-party monitoring tools

Best Practices

Planning for High Availability

Always Use HA for Production: Any workload requiring uptime guarantees should use high availability
Test Failover Procedures: Regularly perform switchovers to verify your disaster recovery processes
Monitor Replication Lag: Though minimal, monitor lag between primary and secondary instances
Configure Application Retry Logic: Ensure applications can reconnect automatically after failover

Storage Planning

Oversize Storage Initially: Leave room for growth to avoid frequent expansions
Enable Automatic Expansion: Prevents storage-related outages
Monitor Storage Trends: Track usage patterns to predict future needs

Security Considerations

Use Private Subnets: Keep DB systems in private subnets
Restrict Security Lists: Limit access to known IP ranges
Secure Credentials: Use OCI Vault for secrets management
Regular Patches: Enable automatic security updates

Cost Optimization

Right-Size Shapes: Start with appropriate shapes and adjust based on metrics
Optimize Backup Retention: Balance compliance needs with storage costs
Use Development Templates: For non-production workloads, use lower-cost configurations
Monitor Resource Utilization: Use Autopilot Shape Advisor for recommendations

Conclusion

MySQL HeatWave provides a comprehensive, fully managed database service with enterprise-grade high availability features. By following this guide, you can create robust DB systems that meet demanding uptime requirements while benefiting from automatic failover, integrated analytics, and seamless scalability.

The combination of automated management, built-in high availability, and HeatWave analytics capabilities makes MySQL HeatWave an excellent choice for modern applications requiring both transactional processing and real-time analytics without the complexity of managing multiple database systems.

For the latest information, consult the official MySQL HeatWave documentation.

MySQL HeatWave Architecture: A Complete Guide

Sun, 02 Nov 2025 09:22:57 +0000

Introduction

MySQL HeatWave is Oracle's fully managed database service that uniquely combines Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) workloads in a single platform. Built on MySQL Enterprise Edition, it delivers real-time analytics on transactional data without requiring ETL duplication to a separate analytics database.

DB System Architecture Overview

A MySQL HeatWave DB System consists of several key components:

Core Components

Compute Instance: The foundational cloud-based compute resource
MySQL Server: MySQL Enterprise Edition with advanced security features including encryption, data masking, authentication, and a database firewall
Operating System: Oracle Linux OS
Virtual Network Interface Card (VNIC): Enables network connectivity
Block Storage: Persistent storage for data and transaction logs

Prerequisites for Deployment

Before creating a DB System, you need:

Tenancy: Your Oracle Cloud Infrastructure account
Compartment: Logical container for organizing resources
Policies: IAM policies for access control
Virtual Cloud Network (VCN): Network infrastructure for connectivity

DB System Provisioning

Shape Configuration

Shapes specify CPU, memory, and network bandwidth. As of 2024, you can provision MySQL HeatWave DB systems using ECPU shapes, which are architecture-agnostic. Shapes can be changed after deployment to scale your resources up or down based on workload demands.

Storage Capacity

Storage stores both data and logs and can be increased as your database grows. The system automatically manages storage allocation for optimal performance.

Endpoint Configuration

The endpoint comprises private IP addresses and port numbers that accept client connections. This provides secure, private connectivity to your database instance.

Data Security in HeatWave

MySQL HeatWave implements comprehensive security measures: data is encrypted and compressed both at rest and in transit. Client applications connect securely to the HeatWave DB System through encrypted channels.

Data Masking is commonly implemented for protecting sensitive information such as:

Social Security Numbers (SSNs)
Credit card numbers
Phone numbers
Personally Identifiable Information (PII)

High Availability Architecture

Standalone vs. High Availability DB Systems

A High Availability (HA) DB System provides enterprise-grade reliability through:

Three MySQL instances: one primary instance and two secondary instances, with all data written to the primary instance automatically copied to the secondary instances.

Distribution Across Availability Domains

The three instances are deployed across different availability domains (AD1, AD2, AD3) and fault domains. This geographic distribution ensures resilience against localized failures.

Automatic Failover Mechanism

In case the primary instance fails, MySQL HeatWave Service automatically promotes one of the secondary instances to function as the primary instance, resuming availability with no data loss.

Important Note: MySQL HeatWave uses MySQL Group Replication to replicate data from the primary to secondary instances over a secure, managed internal network.

HeatWave Cluster Architecture

Components and Features

The HeatWave cluster extends your DB System with powerful analytics capabilities:

HeatWave Plugin: Manages cluster operations and query routing
HeatWave Nodes: Store data in memory and process analytics queries, with each node containing an instance of HeatWave
Data Loading: Tables are loaded from MySQL into the HeatWave cluster
Query Offloading: Analytics queries are automatically offloaded to the cluster

Scalability

HeatWave's scale-out architecture enables massive parallelism, allowing the cluster to scale up to 512 nodes for provisioning, loading data, and processing queries.

Query Processing

MySQL HeatWave Cluster stores data in main memory using a hybrid columnar format that achieves the benefits of columnar processing while avoiding the materialization and update costs of pure columnar format.

Queries that meet specific prerequisites are automatically offloaded from the MySQL DB System to the HeatWave cluster for accelerated processing, with results returned to the client through the MySQL DB System.

HeatWave Storage Layer

The HeatWave Storage Layer is a critical component that provides data persistence and rapid recovery:

Key Features

Data loaded into MySQL HeatWave Cluster is automatically persisted to network-attached block storage, known as the MySQL HeatWave Storage Layer, which allows data to be reloaded quickly when the cluster resumes after a pause or recovers from a node or cluster failure.

Storage Layer Capabilities

Automated Persistence: Stores a copy of in-memory data automatically
Transparent Updates: Changes are propagated without manual intervention
Fast Recovery: Enables rapid restoration after failures
Data Reload: Supports efficient data restoration to memory
Automatic Expansion: On AWS, the storage layer is built on S3, allowing data to be reloaded to each MySQL HeatWave node independently and in parallel

Self-Managing Architecture

This automated, self-managing storage layer scales to the size required for your MySQL HeatWave Cluster and operates independently in the background.

HeatWave Lakehouse

MySQL HeatWave Lakehouse allows enterprises to process and query data across file formats such as CSV and Parquet, as well as Aurora and Redshift backups from AWS, even when data is not stored inside a MySQL database.

The Lakehouse capability enables organizations to:

Query up to 400TB of data in object storage
Combine OLTP data with data lake information
Eliminate ETL complexity and costs
Process structured, semi-structured, and unstructured data

Performance and Capabilities

AutoML Integration

MySQL HeatWave includes built-in machine learning capabilities through HeatWave AutoML, which:

Automates model training and selection
Eliminates the need for separate ML services
Processes data in-database for security and performance

GenAI and Vector Store

Recent additions include:

In-database Large Language Models (LLMs)
Automated vector store for AI applications
Natural language query capabilities
Contextual conversations with your data

Performance Benchmarks

Oracle claims HeatWave provides 1,400x faster analytics performance than Amazon RDS Aurora and 6-7x faster than AWS Redshift and Snowflake, based on their benchmarks.

Multicloud Availability

MySQL HeatWave is available across multiple cloud platforms:

Oracle Cloud Infrastructure (OCI): Native availability in all OCI regions
Amazon Web Services (AWS): Native deployment within AWS
Microsoft Azure: Available through Oracle Database Service for Azure

Best Practices

Enable High Availability: For production workloads requiring 99.99% uptime
Monitor Shape Performance: Use Autopilot Shape Advisor for optimization
Implement Data Masking: Protect sensitive information at the column level
Leverage Storage Layer: Trust automatic persistence for recovery scenarios
Plan Node Capacity: Use the Estimate Node Count feature for proper cluster sizing

Conclusion

MySQL HeatWave represents a significant advancement in database technology by unifying OLTP and OLAP workloads in a single, fully managed service. Its innovative architecture, featuring automated storage management, high availability, and integrated machine learning capabilities, makes it a compelling choice for organizations seeking to eliminate data duplication, reduce costs, and accelerate insights from their data.

For more detailed information, consult the official MySQL HeatWave documentation.

Building Resilient Oracle Databases: Essential Strategies for Security and Maximum Availability

Wed, 10 Sep 2025 16:52:51 +0000

Building Resilient Oracle Databases: Essential Strategies for Security and Maximum Availability Architecture

In today's data-driven landscape, ensuring database resilience through robust security measures and high availability architecture is not just best practice—it's business critical. Oracle provides a comprehensive framework for achieving both maximum security and availability through integrated technologies and proven methodologies.

Securing Data at Its Source: A Multi-Layered Approach

Core Security Principles

Modern database security requires a defense-in-depth strategy that protects data throughout its entire lifecycle. Oracle Database offers comprehensive encryption, key management, and masking capabilities that scale to enterprise-level workloads, addressing the requirements of most data privacy regulations.

1. Data Encryption and Key Management

Data at Rest: Transparent Data Encryption (TDE) for tablespaces and databases
Data in Motion: Network encryption using SSL/TLS protocols
Key Management: Oracle Key Vault for centralized key lifecycle management
Advanced Features: Always Encrypted columns for sensitive data protection

2. Access Control and Authentication

Principle of Least Privilege: Grant necessary privileges only. Do not provide database users or roles more privileges than are necessary
Strong Authentication: Multi-factor authentication, Kerberos integration
Database Vault: Separation of duties for privileged users
Virtual Private Database (VPD): Row-level security policies

3. Activity Monitoring and Auditing

Unified Auditing: Comprehensive tracking of database activities
Real-time Monitoring: Immediate detection of suspicious activities
Compliance Reporting: Automated generation of audit trails
Database Firewall: Real-time protection against SQL injection attacks

Security Zones of Control

Assess
Continuously evaluate your database's current security state through automated assessments and vulnerability scans.

Detect
Identify attempts to access data outside established policies using advanced analytics and machine learning.

Prevent
Block unauthorized access attempts that bypass database control mechanisms through real-time protection.

Establishing Baseline Security Posture

1. System State Assessment
Regularly evaluate database configurations against security benchmarks and industry standards.

2. Security Patch Management
Oracle releases Critical Patch Updates quarterly to address security vulnerabilities. Implement automated patch management processes to ensure timely updates.

3. Comprehensive Database Auditing
Monitor all database connections, privileged operations, and data access patterns to establish behavioral baselines.

Oracle Security Tools and Services

Oracle Data Safe
Oracle Data Safe helps assess your security posture, close gaps that may result in noncompliance, and generate reports to share with auditors. Key capabilities include:

Security Assessment: Updated with additional checks from Oracle Database 19c CIS Benchmark v1.2 and STIG V2R8
Data Discovery: Automated identification of sensitive data
Data Masking: Replace sensitive data with realistic, yet obscured, data for safe use in nonproduction environments
Activity Auditing: Comprehensive user and privilege monitoring

Database Security Assessment Tool (DBSAT)
Oracle Database Security Assessment Tool provides prioritized recommendations on how to mitigate identified security risks or gaps within Oracle Database. Currently in version 3.1.0 (January 2024), DBSAT is a command-line tool focused on detecting potential security vulnerabilities across database, operating system, and network components.

Oracle Maximum Availability Architecture (MAA)

Oracle Maximum Availability Architecture (MAA) provides architecture, configuration, and lifecycle best practices for Oracle Databases and applications, enabling high-availability service levels for databases and applications residing in on-premises, cloud, or hybrid configurations.

Core MAA Objectives

High Availability
Provides redundant components and automated failover mechanisms to ensure uninterrupted service during hardware failures, planned maintenance, and other disruptions.

Scalability
Ensures consistent performance as application usage grows, automatically scaling resources to meet demand without degrading user experience.

Rolling Updates and Patches
Minimizes planned maintenance downtime through online patching and rolling upgrade capabilities.

Disaster Recovery
Protects against complete site outages through geographically distributed backup sites and automated recovery procedures.

Key Performance Metrics

Recovery Point Objective (RPO)
Measures acceptable data loss in time units. Modern MAA implementations can achieve near-zero RPO through synchronous replication.

Recovery Time Objective (RTO)
The key goal of Oracle MAA is to meet recovery time objectives (RTO) by minimizing system downtime during failures and recovery operations.

Chaos Engineering and Resilience Testing

Proactive Failure Testing
Intentionally introduce controlled failures to validate system resilience and identify potential weaknesses before they impact production.

Common Failure Scenarios:

Network, server, and storage component failures
Human errors and data corruption incidents
Power outages and complete site failures
Data reorganization and schema changes
Application bugs and performance degradation

MAA Technology Stack

Real Application Clusters (RAC)
Provides horizontal scalability and high availability through clustered database instances sharing the same storage.

Oracle Active Data Guard
Oracle Active Data Guard adds high availability (HA) and disaster recovery (DR) capabilities to an Oracle Database through real-time data replication and automatic failover.

Key MAA Capabilities:

Optimal High Availability: Automated failover with minimal downtime
Data Protection: Multiple levels of data redundancy and backup
Active Replication: Real-time synchronization across sites
Scale-Out Architecture: Horizontal scaling capabilities
Application Continuity: Transparent recovery for application connections

Oracle Cloud Database Services

Service Portfolio

Oracle Cloud Database: Traditional database services with full administrative control and customization options.
Oracle Exadata Database Service: High-performance database platform optimized for Oracle workloads with built-in MAA features.
Oracle Autonomous Database: Fully automated, standardized Autonomous Database infrastructure means that achieving robust availability and disaster recovery for your mission-critical applications is built into the service foundation.

Cloud MAA Benefits

Automated Operations
Self-healing, self-securing, and self-tuning capabilities reduce operational overhead and human error risks.

Global Distribution
Multi-region deployment options for optimal disaster recovery and performance.

Elastic Scaling
Dynamic resource allocation based on workload demands without manual intervention.

Implementation Best Practices

Security Implementation Strategy

Risk Assessment: Conduct thorough security assessments using Oracle Data Safe and DBSAT
Defense in Depth: Implement multiple security layers rather than relying on single controls
Continuous Monitoring: Establish real-time monitoring and automated response procedures
Regular Updates: Maintain current security patches and configuration updates
Compliance Validation: Regular compliance checks against industry standards and regulations

MAA Implementation Approach

Architecture Planning: Design for your specific RTO/RPO requirements
Redundancy Strategy: Implement appropriate levels of redundancy across all tiers
Testing Protocol: Regular disaster recovery and failover testing
Monitoring and Alerting: Comprehensive observability across all components
Documentation: Maintain detailed runbooks and recovery procedures

Integrated Security and Availability

Unified Management
Use Oracle Enterprise Manager or OCI Console for centralized management of both security and availability features.

Automated Responses
Implement automated security incident response that considers availability requirements.

Compliance Integration
Ensure high availability solutions meet regulatory compliance requirements for data protection and audit trails.

Future-Proofing Your Oracle Environment

Emerging Technologies

Machine Learning Integration
Oracle's ML capabilities enhance both security (anomaly detection) and availability (predictive maintenance).

Container and Kubernetes Support
Modern deployment patterns with built-in resilience and security features.

Multi-Cloud Strategies
Build Multicloud Networks for Business continuity using Oracle's hybrid cloud capabilities.

Continuous Improvement

Regular Architecture Reviews
Periodic assessment of MAA implementations against evolving business requirements.

Security Posture Evolution
Continuous enhancement of security controls based on threat landscape changes.

Performance Optimization
Ongoing tuning of both security and availability components for optimal performance.

Conclusion

Building resilient Oracle database environments requires a holistic approach that integrates security and availability from the ground up. Oracle considers cloud security its highest priority, and the security responsibilities are shared between Oracle and you.

By implementing Oracle's Maximum Availability Architecture alongside comprehensive security controls, organizations can achieve:

99.99%+ uptime through redundant, self-healing infrastructure
Enterprise-grade security that meets the most stringent compliance requirements
Automated operations that reduce human error and operational overhead
Future-ready architecture that scales with business growth

The combination of Oracle Data Safe, MAA best practices, and cloud-native services provides a robust foundation for mission-critical applications that demand both maximum security and availability.

Whether you're running traditional on-premises environments, hybrid configurations, or fully cloud-native deployments, Oracle's integrated approach ensures your data remains secure, available, and performant under all conditions.