forked from percona/orchestrator
-
Notifications
You must be signed in to change notification settings - Fork 0
Add built-in ProxySQL failover hooks #36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
330a358
Add ProxySQL configuration fields to Config struct
renecannao dfaaf13
Add ProxySQL Admin client library
renecannao 6987922
Implement ProxySQL failover hook logic
renecannao 83d2510
Initialize ProxySQL hook at application startup
renecannao 3ef1a1b
Integrate ProxySQL hooks into master failover and graceful takeover
renecannao 253e8b8
Add proxysql-test and proxysql-servers CLI commands
renecannao 7194d16
Add ProxySQL settings to sample configuration
renecannao 97b7eda
Add ProxySQL hooks documentation
renecannao db05f43
Address review feedback: fix data race, add timeouts, fix imports, im…
renecannao File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Submodule agent-a11dcc9a
added at
516d66
Submodule agent-a2c352e9
added at
536a07
Submodule agent-a3ae8388
added at
2b0532
Submodule agent-a6ed0246
added at
58cb36
Submodule agent-a82614b2
added at
783200
Submodule agent-a8a6fb28
added at
678b63
Empty file.
Submodule agent-ab100d9d
added at
87a851
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| # ProxySQL Failover Hooks | ||
|
|
||
| Orchestrator has built-in support for updating [ProxySQL](https://proxysql.com) hostgroups during failover. When configured, orchestrator will automatically: | ||
|
|
||
| 1. **Before failover:** Drain the old master in ProxySQL (set `OFFLINE_SOFT` or `weight=0`) | ||
| 2. **After failover:** Update ProxySQL hostgroups to route traffic to the new master | ||
|
|
||
| No custom scripts needed — orchestrator + ProxySQL works out of the box. | ||
|
|
||
| ## Configuration | ||
|
|
||
| Add these settings to your `orchestrator.conf.json`: | ||
|
|
||
| ```json | ||
| { | ||
| "ProxySQLAdminAddress": "127.0.0.1", | ||
| "ProxySQLAdminPort": 6032, | ||
| "ProxySQLAdminUser": "admin", | ||
| "ProxySQLAdminPassword": "admin", | ||
| "ProxySQLWriterHostgroup": 10, | ||
| "ProxySQLReaderHostgroup": 20, | ||
| "ProxySQLPreFailoverAction": "offline_soft" | ||
| } | ||
| ``` | ||
|
|
||
| ### Settings | ||
|
|
||
| | Setting | Default | Description | | ||
| |---------|---------|-------------| | ||
| | `ProxySQLAdminAddress` | (empty) | ProxySQL Admin host. Leave empty to disable hooks. | | ||
| | `ProxySQLAdminPort` | 6032 | ProxySQL Admin port | | ||
| | `ProxySQLAdminUser` | admin | Admin interface username | | ||
| | `ProxySQLAdminPassword` | (empty) | Admin interface password | | ||
| | `ProxySQLAdminUseTLS` | false | Use TLS for Admin connection | | ||
| | `ProxySQLWriterHostgroup` | 0 | Writer hostgroup ID. Must be > 0 to enable hooks. | | ||
| | `ProxySQLReaderHostgroup` | 0 | Reader hostgroup ID. Optional. | | ||
| | `ProxySQLPreFailoverAction` | offline_soft | Pre-failover action: `offline_soft`, `weight_zero`, or `none` | | ||
|
|
||
| ## How It Works | ||
|
|
||
| ### Pre-Failover | ||
|
|
||
| When orchestrator detects a dead master and begins recovery: | ||
|
|
||
| - **`offline_soft`**: Sets the old master's status to `OFFLINE_SOFT` in ProxySQL. Existing connections are allowed to complete, but no new connections are routed to it. | ||
| - **`weight_zero`**: Sets the old master's weight to 0. Similar effect but preserves the server entry's status. | ||
| - **`none`**: Skips pre-failover ProxySQL update. | ||
|
|
||
| ### Post-Failover | ||
|
|
||
| After a new master is promoted: | ||
|
|
||
| 1. Old master is removed from the writer hostgroup | ||
| 2. New master is added to the writer hostgroup | ||
| 3. If reader hostgroup is configured: new master is removed from readers | ||
| 4. If reader hostgroup is configured: old master is added to the reader hostgroup as `OFFLINE_SOFT` | ||
| 5. `LOAD MYSQL SERVERS TO RUNTIME` is executed to apply changes immediately | ||
| 6. `SAVE MYSQL SERVERS TO DISK` is executed to persist changes | ||
|
|
||
| ### Failover Timeline | ||
|
|
||
| ```text | ||
| Dead master detected | ||
| → OnFailureDetectionProcesses (scripts) | ||
| → PreFailoverProcesses (scripts) | ||
| → ProxySQL pre-failover: drain old master ← NEW | ||
| → [topology manipulation: elect new master] | ||
| → KV store updates (Consul/ZK) | ||
| → ProxySQL post-failover: promote new master ← NEW | ||
| → PostMasterFailoverProcesses (scripts) | ||
| → PostFailoverProcesses (scripts) | ||
| ``` | ||
|
|
||
| ProxySQL hooks run **alongside** existing script-based hooks — they don't replace `PreFailoverProcesses` or `PostFailoverProcesses`. | ||
|
|
||
| ## CLI Commands | ||
|
|
||
| ### Test connectivity | ||
|
|
||
| ```bash | ||
| orchestrator-client -c proxysql-test | ||
| ``` | ||
|
|
||
| ### Show ProxySQL server list | ||
|
|
||
| ```bash | ||
| orchestrator-client -c proxysql-servers | ||
| ``` | ||
|
|
||
| ## Multiple ProxySQL Instances | ||
|
|
||
| For ProxySQL Cluster deployments, configure orchestrator to connect to **one** ProxySQL node. Changes propagate automatically across the cluster via ProxySQL's built-in cluster synchronization (`proxysql_servers` table). | ||
|
|
||
| If not using ProxySQL Cluster, you can run multiple orchestrator hook configurations by setting up a ProxySQL load balancer, or by using the existing `PostMasterFailoverProcesses` script hooks for additional ProxySQL instances. | ||
|
|
||
| ## Interaction with Existing Hooks | ||
|
|
||
| ProxySQL hooks are **non-blocking** during pre-failover: if ProxySQL is unreachable, the failover proceeds normally. Post-failover errors are logged but do not mark the recovery as failed. | ||
|
|
||
| This ensures that a ProxySQL outage never prevents MySQL failover from completing. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The sample configuration uses "admin" as the
ProxySQLAdminPassword. While this is a sample, it's generally a good practice to avoid suggesting default, insecure passwords even in examples, as users might copy-paste without changing. Consider using an empty string or a placeholder like "your_password" to encourage secure practices.