Coder Legion - Recent posts tagged database-replication

MySQL HeatWave: Replicating to MySQL Server (Outbound Replication)

Mon, 24 Nov 2025 09:16:13 +0000

Outbound replication enables you to copy data from a MySQL HeatWave DB system to external MySQL instances, supporting data distribution, disaster recovery, and hybrid cloud architectures. Unlike inbound replication where HeatWave acts as the replica, outbound replication positions HeatWave as the source, allowing you to replicate data to on-premises servers, cloud compute instances, or other DB systems.

This guide covers outbound replication scenarios, configuration, prerequisites, and best practices.

What is Outbound Replication?

Outbound replication uses a replication channel to copy transactions from a DB system to another location asynchronously. A channel connects the source DB system (HeatWave) to a replica database (external MySQL instance or another DB system), copying data changes from source to replica.

Key Characteristics:

Source: MySQL HeatWave DB system
Replica: External MySQL instance or another DB system
Asynchronous: Replica doesn't need constant connection
Channel Configuration: Always configured on the replica
Scale-Out: Enables read scaling by distributing load

Important: You can configure only a DB system as the source; you cannot configure a read replica as the source.

Supported Outbound Replication Scenarios

MySQL HeatWave supports three primary outbound replication scenarios. In all scenarios, the replica must be the same version as, or a higher version of MySQL than, the DB system source.

1. DB System to On-Premises MySQL Instance

Replicate from a DB system to an on-premises MySQL server.

Use Cases:

Data repatriation from cloud
Hybrid cloud architectures
Local reporting and analytics
Disaster recovery to on-premises
Compliance requirements for local data

2. DB System to MySQL on Cloud Compute

Replicate to a MySQL instance installed on OCI compute instance or other cloud providers (AWS, Azure, GCP).

Use Cases:

Custom MySQL configurations not available in managed service
Testing new MySQL versions
Migration staging environments
Cross-cloud data distribution

3. DB System to DB System

Replicate from one DB system to another DB system, both on OCI.

Use Cases:

Cross-region replication
Data distribution across geographies
Reporting replicas
Development/test environment refresh

Note: Prior to MySQL 8.3.0-u2, you cannot perform outbound replication from a DB system with MySQL HeatWave Lakehouse enabled. As of MySQL 8.3.0-u2+, outbound replication is supported with Lakehouse, but Lakehouse tables can only replicate to replica DB systems, not external replicas.

Outbound vs. Inbound Replication

Aspect	Outbound Replication	Inbound Replication
Source	HeatWave DB System	External MySQL or DB System
Replica	External MySQL or DB System	HeatWave DB System
Channel Config	On replica	In HeatWave Service
Use Case	Data out of cloud	Data into cloud
Management	Manual on replica	Managed in OCI Console

Prerequisites for Outbound Replication

Source DB System Requirements

1. Active DB System

An active MySQL HeatWave DB system must be available as the replication source.

2. Binary Logging Enabled

Binary logging is enabled by default on all DB systems. Verify:

-- On source DB system
SHOW VARIABLES LIKE 'log_bin';

3. Row-Based Replication

Row-based binary logging must be enabled (default):

SHOW VARIABLES LIKE 'binlog_format';
-- Should return: ROW

4. GTID Mode Enabled (Recommended)

GTID mode is disabled by default on DB systems but strongly recommended for outbound replication.

Why GTID Matters:

Simplifies replication setup with auto-positioning
Easier failover and recovery
Better tracking of transactions
Reduces configuration complexity

Note: GTID mode is a compatibility option you must consider when creating your DB system.

5. Replication User Account

Create a dedicated replication user on the source DB system with required privileges.

Create replication user:

-- Connect to source DB system
CREATE USER 'repl_user'@'%' 
  IDENTIFIED BY 'SecurePassword123!' 
  REQUIRE SSL;

GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%';
FLUSH PRIVILEGES;

Replica MySQL Server Requirements

1. MySQL Version Compatibility

The replica must be the same version as, or a higher version than, the DB system source. For example, you can replicate from MySQL 8.0 to MySQL 8.4, but not from MySQL 8.4 to MySQL 8.0.

2. Compatible Configuration Options

The replica must have matching configuration for certain system variables:

lowercase_table_names: The source and replica must run with the same lower_case_table_names system variable value (0, 1, or 2).

Enable Partial Revoke (if used on source): If your source DB system uses partial revokes, the replica must also have this enabled:

-- On replica
SET GLOBAL partial_revokes = ON;

GTID Mode (recommended): If source uses GTID, replica should have GTID enabled:

-- On replica
SET GLOBAL enforce_gtid_consistency = ON;
SET GLOBAL gtid_mode = OFF_PERMISSIVE;
SET GLOBAL gtid_mode = ON_PERMISSIVE;
SET GLOBAL gtid_mode = ON;

Timezone Information Loaded: If your source DB system uses named timezones, load timezone information on the replica:

# On replica server
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p mysql

3. Bi-Directional Communication Enabled

The replica must be able to connect to the source DB system:

Network path exists (VPN, FastConnect, direct connectivity)
Firewall rules allow MySQL port (3306)
Security lists/NSGs permit traffic

4. Row-Based Binary Logging

The replica should also use row-based replication:

SET GLOBAL binlog_format = 'ROW';

Network Connectivity Options

Connecting External MySQL to HeatWave Source

For external MySQL servers replicating FROM HeatWave, establish connectivity:

Option 1: Network Load Balancer

Set up a Network Load Balancer to provide external access to the DB system:

Steps:

Create Network Security Group allowing access from replica IP
Configure LB Listener (add NSG)
Configure LB Backend
Specify Health Check Policy
Configure backend to access MySQL source

Security: Enforce SSL/TLS for all connections.

Option 2: Site-to-Site VPN

For on-premises replicas:

IPSec VPN tunnel over internet
Encrypted connectivity
Up to 250 Mbps per tunnel

Option 3: FastConnect

For production on-premises replicas:

Dedicated private connection
Up to 10 Gbps bandwidth
Predictable performance
Lower latency

Loading Initial Data from DB System to MySQL Server

Before starting replication, export existing data from the source to the replica.

Step 1: Export Data from Source DB System

Use MySQL Shell Instance Dump utility to export data:

mysqlsh admin@<DB_SYSTEM_IP>:3306 -- util dump-instance /path/to/dump \
  --excludeSchemas=mysql,sys,performance_schema,information_schema \
  --ocimds=true

What This Exports:

All user schemas and data
User accounts and privileges
GTID executed position
Excludes system schemas

Why Exclude System Schemas:

System schemas managed differently
Prevents replication conflicts
User accounts can be recreated separately
Best practice for replication setup

Important: If using GTID-based replication, capture the value of the source server's system variable gtid_executed from the dump metadata.

Step 2: Import Data into Replica MySQL Server

Import the dump into your replica MySQL instance:

mysqlsh admin@<REPLICA_IP>:3306 -- util load-dump /path/to/dump \
  --includeSchemas=<schema_list> \
  --progressFile=/path/to/progress.json

Import User Accounts:

mysqlsh admin@<REPLICA_IP>:3306 -- util load-dump /path/to/dump \
  --includeUsers=true \
  --excludeUsers=admin,root

Step 3: Update gtid_purged Settings

If using GTID-based replication, set gtid_purged on the replica:

-- On replica
RESET MASTER;
SET GLOBAL gtid_purged='<gtid_executed_value>';

Example:

SET GLOBAL gtid_purged='3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5';

Configuring Outbound Replication Channel

The replication channel is always configured on the replica (not in HeatWave Service for outbound replication).

Configuration Parameters

Source Connection Settings

source_host: IP address or hostname of the source DB system

SOURCE_HOST='10.0.1.100'

source_port: MySQL port (default 3306)

SOURCE_PORT=3306

source_user: Replication user account

SOURCE_USER='repl_user'

source_password: Replication user password

SOURCE_PASSWORD='SecurePassword123!'

source_ssl: Enable SSL/TLS (recommended)

SOURCE_SSL=1

Replication Positioning

GTID Auto-Positioning (recommended):

SOURCE_AUTO_POSITION=1

Manual Binary Log Positioning (if GTID not used):

SOURCE_LOG_FILE='mysql-bin.000003'
SOURCE_LOG_POS=154

Setting Up the Replication Channel

Connect to the replica MySQL server and run:

-- Using GTID auto-positioning (recommended)
CHANGE REPLICATION SOURCE TO
  SOURCE_HOST='<DB_SYSTEM_IP>',
  SOURCE_PORT=3306,
  SOURCE_USER='repl_user',
  SOURCE_PASSWORD='SecurePassword123!',
  SOURCE_SSL=1,
  SOURCE_AUTO_POSITION=1;

Without GTID (manual positioning):

CHANGE REPLICATION SOURCE TO
  SOURCE_HOST='<DB_SYSTEM_IP>',
  SOURCE_PORT=3306,
  SOURCE_USER='repl_user',
  SOURCE_PASSWORD='SecurePassword123!',
  SOURCE_SSL=1,
  SOURCE_LOG_FILE='mysql-bin.000003',
  SOURCE_LOG_POS=154;

Starting Replication

START REPLICA;

Or for older MySQL versions:

START SLAVE;

Managing Replication

Checking Replication Status

SHOW REPLICA STATUS\G

Key Metrics:

Replica_IO_Running: Should be "Yes"
Replica_SQL_Running: Should be "Yes"
Seconds_Behind_Source: Replication lag
Last_IO_Error: IO thread errors
Last_SQL_Error: SQL thread errors

Stopping Replication

STOP REPLICA;

Resetting Replication

Warning: This clears all replication position data.

STOP REPLICA;
RESET REPLICA ALL;

Reconfiguring the Channel

To change replication settings:

STOP REPLICA;

CHANGE REPLICATION SOURCE TO
  SOURCE_HOST='<NEW_IP>',
  SOURCE_USER='<NEW_USER>',
  -- other parameters
  SOURCE_AUTO_POSITION=1;

START REPLICA;

Monitoring Replication

Using Performance Schema

Query replication metrics:

-- Connection status
SELECT * FROM performance_schema.replication_connection_status\G

-- Applier status
SELECT * FROM performance_schema.replication_applier_status\G

-- Worker threads
SELECT * FROM performance_schema.replication_applier_status_by_worker\G

Setting Up Alerts

Monitor these critical metrics:

Replication lag exceeding threshold (e.g., 60 seconds)
IO or SQL thread stopped
Replication errors
Connection failures

Using Database Management

If your replica is also a DB system on OCI, use Database Management to monitor:

Active connections
Statement count
Fetch state
Apply state
Time behind source
GTID executed

Common Outbound Replication Issues

Issue 1: Replica Cannot Connect to Source

Error: "Can't connect to MySQL server on ''"

Causes:

Network Load Balancer not configured
Security lists blocking traffic
VPN/FastConnect not established
Wrong IP or hostname

Solutions:

Verify Network Load Balancer configuration
Check security lists allow port 3306
Test connectivity: telnet <DB_SYSTEM_IP> 3306
Verify VPN/FastConnect is active

Issue 2: Authentication Failures

Error: "Access denied for user 'repl_user'@''"

Causes:

Replication user not created on source
Wrong credentials
User not allowed from replica IP

Solutions:

Verify user exists on source: SELECT User, Host FROM mysql.user WHERE User = 'repl_user';
Check credentials are correct
Verify user grants: SHOW GRANTS FOR 'repl_user'@'%';

Issue 3: Replication Lag Increasing

Symptoms: Seconds_Behind_Source growing

Causes:

High transaction volume on source
Replica underpowered
Network bandwidth limitations
Large transactions

Solutions:

Increase replica resources
Optimize slow queries
Check network bandwidth
Break large transactions into batches

Issue 4: SSL Connection Failures

Error: SSL-related errors in replication status

Causes:

SSL not configured on source
Certificate issues
Wrong SSL mode

Solutions:

Verify source supports SSL
Use SOURCE_SSL=1 for encrypted connection
Check certificate validity if using certificate verification

Best Practices

1. Always Use GTID Mode

Enable GTID on both source and replica for simplified management:

Easier failover
Automatic position tracking
Simplified reconfiguration

2. Enforce SSL/TLS Encryption

Always encrypt replication traffic:

CREATE USER 'repl_user'@'%' 
  IDENTIFIED BY 'SecurePassword!' 
  REQUIRE SSL;

3. Monitor Replication Lag

Set up alerts for:

Lag > 60 seconds
IO or SQL thread stopped
Replication errors

4. Use Network Load Balancer

For production external replicas:

Provides stable endpoint
Enables health checks
Supports SSL termination
Better security controls

5. Plan for Network Interruptions

Use appropriate binary log retention on source
Monitor relay log space on replica
Test reconnection scenarios

6. Regular Health Checks

Implement automated checks:

#!/bin/bash
# Check replication health
REPLICA_STATUS=$(mysql -e "SHOW REPLICA STATUS\G" | grep "Replica_.*_Running:" | awk '{print $2}')
if [[ "$REPLICA_STATUS" != "Yes" ]]; then
  echo "Replication issue detected"
  # Send alert
fi

7. Test Failover Procedures

Regularly test:

Stopping and starting replication
Switching to different source
Replica promotion scenarios
Application behavior during failures

8. Document Configuration

Maintain documentation of:

Source and replica connection details
Replication user credentials
Network paths used
Failover procedures

Limitations for Outbound Replication

Lakehouse Table Restrictions

Lakehouse tables can only replicate to replica DB systems. Replicating Lakehouse tables to external replicas (on-premises MySQL) causes replication to break with errors.

Workaround: Exclude Lakehouse tables from replication or only replicate to DB system replicas.

Prior to MySQL 8.3.0-u2

Cannot perform outbound replication from a DB system with HeatWave Lakehouse enabled
Cannot enable Lakehouse on HA DB system with HeatWave cluster

Read Replicas as Source

You cannot configure a read replica as the source for outbound replication. Only the primary DB system can serve as the replication source.

Outbound replication from MySQL HeatWave enables flexible data distribution strategies, supporting hybrid cloud architectures, disaster recovery, and read scaling scenarios. By understanding configuration requirements, network connectivity options, and best practices, you can implement reliable outbound replication for your specific needs.

Key takeaways:

Source is HeatWave: DB system acts as replication source
Channel on replica: Always configure replication on the replica side
Use GTID: Simplifies setup and management
Encrypt traffic: Always use SSL/TLS
Network connectivity: Use NLB, VPN, or FastConnect for external replicas
Monitor continuously: Track lag, errors, and thread status
Version compatibility: Replica must be same or higher MySQL version

Whether replicating to on-premises servers, cloud compute instances, or other DB systems, MySQL HeatWave outbound replication provides the flexibility needed for modern distributed architectures.

MySQL HeatWave: Replication Channels and Troubleshooting

Fri, 21 Nov 2025 16:28:02 +0000

Replication channels are the cornerstone of MySQL HeatWave inbound replication, establishing the connection between source databases and replica DB systems. Understanding channel states, operations, and troubleshooting techniques is essential for maintaining reliable data replication. This guide covers channel management, lifecycle states, common errors, and proven troubleshooting strategies.

Understanding Replication Channels

A replication channel represents a path through which changes produced by transactions on a source MySQL server or DB system are replicated to a target DB system.

What Channels Do

Each channel identifies:

A replication source (hostname, port, credentials)
Configuration required to connect to that source
How to copy transactions from source to replica
Replication positioning (GTID or binary log)

Important: Channels have their own lifecycle and can be enabled, disabled, reset, and resumed independently.

Channel States

MySQL HeatWave channels exist in either final states (stable) or transient states (temporary during operations).

Final States

1. ACTIVE

Channel is replicating data successfully.

Characteristics:

Both IO and SQL threads running
Transactions flowing from source to replica
May have slight replication lag (normal)
No errors reported

Console Indicator: Green icon with "Active" text

2. INACTIVE

Channel is not replicating; replication has been stopped.

Common Reasons:

Channel manually disabled by administrator
Created with "Enable automatically" unchecked
DB system is stopped

Console Indicator: Gray icon with "Inactive" text

To Resume: Enable the channel through console or CLI

3. NEEDS_ATTENTION

Channel is not replicating due to an error.

Common Causes:

Source server unreachable
Authentication failure
Binary log files purged on source
SSL/TLS configuration mismatch
Incompatible transactions
DB system is in INACTIVE state

Console Indicator: Red/orange icon with "Needs Attention" text

Action Required: Check channel details page for error messages, resolve issue, then resume channel

4. DELETED

Channel has been permanently removed.

Characteristics:

Appears briefly before removal from list
Cannot be recovered
All channel configuration deleted

Transient States

These are temporary states during channel operations:

1. CREATING

Channel is being created and configured.

Duration: Typically 1-3 minutes

Next State: ACTIVE (if auto-enabled) or INACTIVE

2. UPDATING

Channel is performing an update operation.

Occurs During:

Enabling or disabling channel
Modifying channel configuration
Resuming after error resolution
DB system operations affecting channel

Duration: Typically 30 seconds to 2 minutes

3. DELETING

Channel deletion is in progress.

Duration: Typically 1-2 minutes

Next State: DELETED

Channel Operations

Enabling a Channel

Starts replication from the source if no errors exist.

Console:

Navigate to Databases → HeatWave MySQL → Channels
Select channel → Actions → Enable
Channel transitions: INACTIVE → UPDATING → ACTIVE

CLI:

oci mysql channel update \
  --channel-id <CHANNEL_OCID> \
  --is-enabled true

Disabling a Channel

Stops replication and puts channel into INACTIVE state.

When to Disable:

Maintenance on source server
Temporary halt of replication
Troubleshooting issues
Before configuration changes

Console: Channel → Actions → Disable

CLI:

oci mysql channel update \
  --channel-id <CHANNEL_OCID> \
  --is-enabled false

Resuming a Channel

Restarts replication after error is resolved. Used when channel is in NEEDS_ATTENTION state.

Console: Channel → Actions → Resume

CLI:

oci mysql channel resume \
  --channel-id <CHANNEL_OCID>

Behavior: Channel transitions to UPDATING, then ACTIVE if error resolved

Resetting a Channel

Removes all replication position data except channel configuration. Channel must be INACTIVE first.

Warning: Resets replication position - will re-replicate transactions

Console: Channel → Actions → Reset

CLI:

oci mysql channel reset \
  --channel-id <CHANNEL_OCID> \
  --from-json '{"resetType": "POSITION"}'

Use Cases:

Recovering from corrupted replication position
Starting replication from beginning
After major source database changes

Deleting a Channel

Permanently removes channel and all associated configuration.

Console: Channel details → Delete

CLI:

oci mysql channel delete \
  --channel-id <CHANNEL_OCID> \
  --force

Troubleshooting Inbound Replication

Step 1: Identify Source of Failure

When channel enters NEEDS_ATTENTION state, determine the root cause:

Check Channel Details Page

Navigate to channel in console
Click channel name to view details
Review Error section for messages
Check Last Error timestamp
Note error code and description

Check Replication Status on Replica

Connect to replica DB system and query:

SHOW REPLICA STATUS\G

Key Fields:

Replica_IO_Running: Should be "Yes" (checks source connection)
Replica_SQL_Running: Should be "Yes" (checks transaction application)
Last_IO_Errno: IO thread error code
Last_IO_Error: IO thread error message
Last_SQL_Errno: SQL thread error code
Last_SQL_Error: SQL thread error message
Seconds_Behind_Source: Replication lag

Performance Schema Query:

SELECT * FROM performance_schema.replication_connection_status\G

Step 2: Identify Communication Failures

Verify connectivity between replica and source:

Test Network Connectivity

From replica DB system or compute instance in same VCN:

# Test if source is reachable
telnet <source_hostname> 3306

# Or use netcat
nc -zv <source_hostname> 3306

# Check DNS resolution
nslookup <source_hostname>

Test Replication User Authentication

mysql -h <source_hostname> \
      -P 3306 \
      -u <repl_user> \
      -p

If successful, authentication works; issue is elsewhere.

Common Error Codes and Solutions

Error 2003: Can't Connect to MySQL Server

Error Message: "Can't connect to MySQL server on '' (error_code)"

Meaning: Replica cannot establish network connection to source

Common Causes:

Source MySQL server not running
Firewall blocking port 3306
Network path doesn't exist
Wrong hostname or IP address
Source server not listening on expected port

Solutions:

Verify source is running:

-- On source server
SHOW GLOBAL STATUS LIKE 'Uptime';

Check firewall rules:

# On source server (Linux)
sudo iptables -L | grep 3306

# Or for firewalld
sudo firewall-cmd --list-all | grep 3306

Verify MySQL listening on port:

# On source server
sudo netstat -tlnp | grep 3306
# Or
sudo ss -tlnp | grep 3306

Check bind-address:

-- On source server
SHOW VARIABLES LIKE 'bind_address';

Should be 0.0.0.0 or specific IP (not 127.0.0.1 for remote connections)

Test from replica network:

telnet <source_ip> 3306

Error 1045: Access Denied

Error Message: "Access denied for user ''@'' (using password: YES/NO)"

Meaning: Authentication failed - wrong credentials or user doesn't have access from replica IP

Common Causes:

Incorrect username or password
User not granted access from replica IP
User exists but lacks REPLICATION SLAVE privilege
Blank users on source conflicting

Solutions:

Verify replication user exists:

-- On source server
SELECT User, Host FROM mysql.user WHERE User = 'repl_user';

Check user privileges:

SHOW GRANTS FOR 'repl_user'@'%';

Should include: GRANT REPLICATION SLAVE ON *.*

Verify replica IP is allowed:

-- On source
SELECT User, Host FROM mysql.user WHERE User = 'repl_user';

Host should be %, replica subnet (e.g., 10.0.1.%), or specific IP

Remove blank users (common issue):

-- On source
SELECT User, Host FROM mysql.user WHERE User = '';
DROP USER ''@'localhost';
DROP USER ''@'<hostname>';
FLUSH PRIVILEGES;

Test authentication:

mysql -h <source> -u repl_user -p

Error 3159: Insecure Connection

Error Message: "Connections using insecure transport are prohibited while --require_secure_transport=ON"

Meaning: Source requires SSL/TLS but channel not configured for it

Solutions:

Option 1: Configure SSL in channel (recommended)

Edit channel configuration
Set SSL mode to "REQUIRED" or higher
Provide SSL certificates if using VERIFY_CA or VERIFY_IDENTITY

Option 2: Disable require_secure_transport on source (not recommended)

-- On source server
SET GLOBAL require_secure_transport = OFF;

Verify SSL configuration:

-- On source
SHOW VARIABLES LIKE 'require_secure_transport';
SHOW VARIABLES LIKE 'have_ssl';

Error 13114: Binary Log Files Purged

Error Message: "Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file'" or "The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires"

Meaning: Binary log files needed by replica have been deleted from source

Common Causes:

GTID not supported or configured
Binary log retention too short
Long replication outage
Binary logs manually purged

Solution 1: Fix GTID-Purged (For GTID-based replication)

Step 1: Retrieve gtid_executed from logical dump

When exporting source data with MySQL Shell:

mysqlsh user@source -- util dump-instance /path/to/dump \
  --ocimds=true

Capture gtid_executed value from dump metadata.

Or query source directly:

-- On source
SHOW GLOBAL VARIABLES LIKE 'gtid_executed';

Step 2: Set gtid_purged on replica

Connect to replica and execute:

-- On replica
CALL sys.SET_GTID_PURGED('<gtid_executed_value>');

Example:

CALL sys.SET_GTID_PURGED('3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5');

Step 3: Resume channel

After setting gtid_purged, resume replication:

oci mysql channel resume --channel-id <CHANNEL_OCID>

Solution 2: Rebuild Replication (If GTID fix fails)

For severe cases:

Step 1: Delete channel and replica DB system

oci mysql channel delete --channel-id <CHANNEL_OCID> --force
oci mysql db-system delete --db-system-id <DB_SYSTEM_OCID> --force

Step 2: Take new snapshot of source

mysqlsh user@source -- util dump-instance /path/to/new-dump \
  --excludeSchemas=mysql,sys,performance_schema,information_schema \
  --ocimds=true

Step 3: Create new DB system

Create DB system and import data during creation or after.

Step 4: Create and enable channel

Create new channel with proper GTID positioning from new dump.

Best Practices for Channel Management

1. Monitor Channel Health

Set up alerts:

Channel state changes to NEEDS_ATTENTION
Replication lag exceeds threshold (e.g., 60 seconds)
IO or SQL thread stops

Regular checks:

SHOW REPLICA STATUS\G

2. Maintain Adequate Binary Log Retention

On source server:

-- Set binary log expiration (seconds)
SET GLOBAL binlog_expire_logs_seconds = 172800; -- 48 hours

For AWS RDS:

CALL mysql.rds_set_configuration('binlog retention hours', 48);

3. Document Channel Configuration

Maintain records of:

Source connection details
Replication user credentials
SSL configuration
GTID settings
Network paths used

4. Test Before Production

Create test channel in non-production
Verify connectivity
Test failover scenarios
Document lessons learned

5. Plan for Maintenance Windows

Before source maintenance:

Verify replication lag is minimal
Optionally disable channel
Perform maintenance
Re-enable channel
Monitor for catch-up

6. Use GTID When Possible

Benefits:

Auto-positioning simplifies management
Easier recovery from failures
Better failover capabilities

Advanced Troubleshooting

Replication Lag Increasing

Symptoms: Seconds_Behind_Source continuously growing

Diagnose:

-- Check what's being executed
SHOW REPLICA STATUS\G

-- Check running queries on replica
SHOW PROCESSLIST;

-- Check for lock waits
SELECT * FROM sys.innodb_lock_waits;

Solutions:

Increase replica resources (upgrade shape)
Optimize slow queries on source
Break large transactions into smaller batches
Check for long-running transactions

Channel Constantly Failing

Symptoms: Channel repeatedly enters NEEDS_ATTENTION

Diagnose:

Check error patterns (same error repeating?)
Review source server logs
Check network stability
Verify source server load

Solutions:

Fix root cause (network, authentication, binary logs)
Consider increasing connection timeouts
Upgrade source server resources if overloaded

MySQL Schema Changes Causing Failures

Remember: Changes to mysql schema are not replicated and cause replication to stop

Solution:

Manage user accounts independently

Skip problematic transaction:

SET GLOBAL sql_replica_skip_counter = 1;
START REPLICA;

Better: Avoid mysql schema changes during active replication

Effective channel management and troubleshooting are crucial for reliable MySQL HeatWave inbound replication. By understanding channel states, recognizing common error codes, and following best practices, you can maintain healthy replication and quickly resolve issues.

Key takeaways:

Monitor channel states: ACTIVE, INACTIVE, NEEDS_ATTENTION are key indicators
Check channel details: Error messages provide crucial troubleshooting information
Common errors have common fixes: Network (2003), authentication (1045), SSL (3159), binary logs (13114)
Use GTID: Simplifies management and recovery
Maintain binary log retention: Prevent purged log issues
Test thoroughly: Validate connectivity and authentication before production

MySQL HeatWave: Replicating to DB System (Inbound Replication)

Tue, 18 Nov 2025 09:39:47 +0000

Inbound replication enables data replication from external MySQL sources into HeatWave DB systems, supporting migrations, disaster recovery, and hybrid cloud architectures.

What is Inbound Replication?

Uses a replication channel to copy transactions from a MySQL source to a HeatWave DB system replica.

Key Characteristics:

Asynchronous: Replicas don't need permanent connection; can reconnect to receive updates
Single Source: One source per replica (multi-source not supported)
Source Independence: Source doesn't control replica; replica connects using replication credentials
Not Managed: You configure and maintain the channel and network connectivity

Supported Scenarios

On-Premises to DB System: Cloud migrations, hybrid architectures, disaster recovery
Cloud Compute to DB System: Cross-cloud replication, migrating self-managed to HeatWave
DB System to DB System: Cross-region replication, geographic redundancy

Replication Architecture

Source Database:

MySQL 5.7+ (on-premises, OCI compute, HeatWave DB System, or other clouds)
Binary logging enabled
Row-based replication format
GTID mode enabled (recommended)

Replica Database:

MySQL HeatWave DB System in private subnet
Applies transactions from source asynchronously

Flow: Source commits to binary log → Channel retrieves transactions → Replica receives and applies

Limitations

Row-Based Only: Statement-based and mixed replication not supported
Asynchronous Only: Semi-synchronous not supported
Single Source: Multi-source replication not supported
MySQL Schema Not Replicated: User/privilege changes don't replicate; manage separately
GTID Tags: Pre-May 2024 systems need upgrade for GTID tag support
Applier Privileges: Replication fails if applier lacks privileges for statements
HeatWave Cluster: ALTER TABLE SECONDARY_LOAD/UNLOAD don't affect target cluster
High Availability: Channel suspends during HA upgrades, resumes after completion

Prerequisites

Source Requirements

1. Enable GTID (Recommended):

SET GLOBAL enforce_gtid_consistency = ON;
SET GLOBAL gtid_mode = OFF_PERMISSIVE;
SET GLOBAL gtid_mode = ON_PERMISSIVE;
SET GLOBAL gtid_mode = ON;

2. Enable Binary Logging:

SET GLOBAL log_bin = ON;
SET GLOBAL binlog_format = 'ROW';

3. Extend Binary Log Retention:

-- For AWS RDS:
CALL mysql.rds_set_configuration('binlog retention hours', 24);

4. Create Replication User:

CREATE USER 'repl_user'@'%' 
  IDENTIFIED BY 'StrongPassword123!' 
  REQUIRE SSL;
GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%';
FLUSH PRIVILEGES;

5. Network Connectivity: Allow replica access to source MySQL port (3306) with SSL/TLS

Replica Requirements

Active HeatWave DB system
Network path to source (VPN, FastConnect, PrivateLink)
Matching lower_case_table_names value with source

Connectivity Options

On-Premises to HeatWave:

FastConnect: Up to 10 Gbps, low latency, production workloads
Site-to-Site VPN: 250 Mbps per tunnel, cost-effective, dev/test

Cloud to HeatWave:

AWS: PrivateLink or VPN
Azure: OCI-Azure interconnect or VPN
GCP: Partner Interconnect or VPN

Dedicated Replication User Benefits

Minimum Privileges: Grant only REPLICATION SLAVE
Restrict Host: Limit to replica IP/subnet
Enforce Encryption: Use REQUIRE SSL

Example:

CREATE USER 'repl_user'@'10.0.1.%' 
  IDENTIFIED BY 'SecurePassword!' 
  REQUIRE SSL;
GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'10.0.1.%';

Loading Initial Data

Export from Source:

mysqlsh user@source_host:3306 -- util dump-instance /path/to/dump \
  --excludeSchemas=mysql,sys,performance_schema,information_schema \
  --ocimds=true

Import to Replica:

mysqlsh admin@replica_host:3306 -- util load-dump /path/to/dump \
  --progressFile=/path/to/progress.json

Creating Replication Channel

Using OCI Console

Navigate to DB System → Resources → Channels
Click Create channel
Configure settings (source hostname, port, credentials, SSL mode)
Click Create

Using OCI CLI

oci mysql channel create-from-mysql --generate-full-command-json-input > channel-config.json
# Edit channel-config.json with your settings
oci mysql channel create-from-mysql --from-json file://channel-config.json

Positioning Options

GTID Auto-Positioning (Recommended):

Automatic transaction tracking
No manual position needed
Requires gtid_mode=ON on source

Binary Log Positioning:

-- Get position on source:
SHOW MASTER STATUS;

Use File and Position values when creating channel

Managing Channels

Start: oci mysql channel update --channel-id <ID> --is-enabled true

Stop: oci mysql channel update --channel-id <ID> --is-enabled false

Reset: oci mysql channel reset --channel-id <ID> --from-json '{"resetType": "POSITION"}'

Delete: oci mysql channel delete --channel-id <ID> --force

Monitoring Replication

Check Status:

SHOW REPLICA STATUS\G

Key Metrics:

Replica_IO_Running: Should be "Yes"
Replica_SQL_Running: Should be "Yes"
Seconds_Behind_Source: Replication lag
Last_Error: Error messages

Channel States:

ACTIVE: Running normally
NEEDS_ATTENTION: Requires intervention
INACTIVE: Stopped

Best Practices

Enable GTID: Simplifies management
Use SSL/TLS: Encrypt replication traffic
Monitor Lag: Alert on >60 seconds lag
Test Failover: Regular failover testing
Binary Log Retention: 24-48 hours recommended
Health Checks: Automate monitoring
Gradual Migration: Phased approach for migrations

Troubleshooting

NEEDS_ATTENTION State:

Check error logs in channel details
Verify connectivity and credentials
Confirm SSL configuration and firewall rules

Increasing Lag:

Increase replica resources
Check network bandwidth
Optimize source queries
Break large transactions

MySQL Schema Errors:

Manage users independently on replica
Avoid mysql schema changes during replication

SSL Failures:

Verify source SSL enabled
Use "REQUIRED" mode
Check certificate validity

Key Takeaways

Asynchronous replication tolerates disconnections
Use GTID mode for simplified management
Always encrypt replication traffic
Monitor lag, errors, and channel health continuously
Understand limitations: row-based only, single source, mysql schema restrictions
Test thoroughly before production cutover

Inbound replication provides flexible, reliable data migration and hybrid cloud capabilities for MySQL HeatWave workloads.