Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/worktrees/agent-a11dcc9a
Submodule agent-a11dcc9a added at 516d66
1 change: 1 addition & 0 deletions .claude/worktrees/agent-a2c352e9
Submodule agent-a2c352e9 added at 536a07
1 change: 1 addition & 0 deletions .claude/worktrees/agent-a3ae8388
Submodule agent-a3ae8388 added at 2b0532
1 change: 1 addition & 0 deletions .claude/worktrees/agent-a6ed0246
Submodule agent-a6ed0246 added at 58cb36
1 change: 1 addition & 0 deletions .claude/worktrees/agent-a82614b2
Submodule agent-a82614b2 added at 783200
1 change: 1 addition & 0 deletions .claude/worktrees/agent-a8a6fb28
Submodule agent-a8a6fb28 added at 678b63
Empty file.
1 change: 1 addition & 0 deletions .claude/worktrees/agent-ab100d9d
Submodule agent-ab100d9d added at 87a851
10 changes: 9 additions & 1 deletion conf/orchestrator-sample.conf.json
Original file line number Diff line number Diff line change
Expand Up @@ -143,5 +143,13 @@
"GraphiteConvertHostnameDotsToUnderscores": true,
"ConsulAddress": "",
"ConsulAclToken": "",
"ConsulKVStoreProvider": "consul"
"ConsulKVStoreProvider": "consul",
"ProxySQLAdminAddress": "",
"ProxySQLAdminPort": 6032,
"ProxySQLAdminUser": "admin",
"ProxySQLAdminPassword": "",
"ProxySQLAdminUseTLS": false,
"ProxySQLWriterHostgroup": 0,
"ProxySQLReaderHostgroup": 0,
"ProxySQLPreFailoverAction": "offline_soft"
}
100 changes: 100 additions & 0 deletions docs/proxysql-hooks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# ProxySQL Failover Hooks

Orchestrator has built-in support for updating [ProxySQL](https://proxysql.com) hostgroups during failover. When configured, orchestrator will automatically:

1. **Before failover:** Drain the old master in ProxySQL (set `OFFLINE_SOFT` or `weight=0`)
2. **After failover:** Update ProxySQL hostgroups to route traffic to the new master

No custom scripts needed — orchestrator + ProxySQL works out of the box.

## Configuration

Add these settings to your `orchestrator.conf.json`:

```json
{
"ProxySQLAdminAddress": "127.0.0.1",
"ProxySQLAdminPort": 6032,
"ProxySQLAdminUser": "admin",
"ProxySQLAdminPassword": "admin",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The sample configuration uses "admin" as the ProxySQLAdminPassword. While this is a sample, it's generally a good practice to avoid suggesting default, insecure passwords even in examples, as users might copy-paste without changing. Consider using an empty string or a placeholder like "your_password" to encourage secure practices.

Suggested change
"ProxySQLAdminPassword": "admin",
"ProxySQLAdminPassword": "",

"ProxySQLWriterHostgroup": 10,
"ProxySQLReaderHostgroup": 20,
"ProxySQLPreFailoverAction": "offline_soft"
}
```

### Settings

| Setting | Default | Description |
|---------|---------|-------------|
| `ProxySQLAdminAddress` | (empty) | ProxySQL Admin host. Leave empty to disable hooks. |
| `ProxySQLAdminPort` | 6032 | ProxySQL Admin port |
| `ProxySQLAdminUser` | admin | Admin interface username |
| `ProxySQLAdminPassword` | (empty) | Admin interface password |
| `ProxySQLAdminUseTLS` | false | Use TLS for Admin connection |
| `ProxySQLWriterHostgroup` | 0 | Writer hostgroup ID. Must be > 0 to enable hooks. |
| `ProxySQLReaderHostgroup` | 0 | Reader hostgroup ID. Optional. |
| `ProxySQLPreFailoverAction` | offline_soft | Pre-failover action: `offline_soft`, `weight_zero`, or `none` |

## How It Works

### Pre-Failover

When orchestrator detects a dead master and begins recovery:

- **`offline_soft`**: Sets the old master's status to `OFFLINE_SOFT` in ProxySQL. Existing connections are allowed to complete, but no new connections are routed to it.
- **`weight_zero`**: Sets the old master's weight to 0. Similar effect but preserves the server entry's status.
- **`none`**: Skips pre-failover ProxySQL update.

### Post-Failover

After a new master is promoted:

1. Old master is removed from the writer hostgroup
2. New master is added to the writer hostgroup
3. If reader hostgroup is configured: new master is removed from readers
4. If reader hostgroup is configured: old master is added to the reader hostgroup as `OFFLINE_SOFT`
5. `LOAD MYSQL SERVERS TO RUNTIME` is executed to apply changes immediately
6. `SAVE MYSQL SERVERS TO DISK` is executed to persist changes

### Failover Timeline

```text
Dead master detected
→ OnFailureDetectionProcesses (scripts)
→ PreFailoverProcesses (scripts)
→ ProxySQL pre-failover: drain old master ← NEW
→ [topology manipulation: elect new master]
→ KV store updates (Consul/ZK)
→ ProxySQL post-failover: promote new master ← NEW
→ PostMasterFailoverProcesses (scripts)
→ PostFailoverProcesses (scripts)
```

ProxySQL hooks run **alongside** existing script-based hooks — they don't replace `PreFailoverProcesses` or `PostFailoverProcesses`.

## CLI Commands

### Test connectivity

```bash
orchestrator-client -c proxysql-test
```

### Show ProxySQL server list

```bash
orchestrator-client -c proxysql-servers
```

## Multiple ProxySQL Instances

For ProxySQL Cluster deployments, configure orchestrator to connect to **one** ProxySQL node. Changes propagate automatically across the cluster via ProxySQL's built-in cluster synchronization (`proxysql_servers` table).

If not using ProxySQL Cluster, you can run multiple orchestrator hook configurations by setting up a ProxySQL load balancer, or by using the existing `PostMasterFailoverProcesses` script hooks for additional ProxySQL instances.

## Interaction with Existing Hooks

ProxySQL hooks are **non-blocking** during pre-failover: if ProxySQL is unreachable, the failover proceeds normally. Post-failover errors are logged but do not mark the recovery as failed.

This ensures that a ProxySQL outage never prevents MySQL failover from completing.
Loading
Loading