Files
tasks/tools/sql/conn.go
Tim Deeb-SwihartandGitHub be7b5514a5 Reconnect to SQL databases when connections fail (#5926)
## What changed?

Both our PostgreSQL and MySQL database backends will now automatically
reconnect to the database when certain errors occur: all errors chosen
have been experienced when testing this behavior through an AWS Aurora
RDS failover of either MySQL or PostgreSQL.

For both backends we will reconnect when we see:
- `ECONNRESET`
- `ECONNABORTED`
- `ECONNREFUSED`
- `io.EOF`
- `io.ErrUnexpectedEOF`
- `database/sql/driver.ErrBadConn`

for postgres we will also reconnect on the following SQLStates:
- `25006` read-only transaction
- `57P03` cannot connect now
- `0A000` feature not supported, but ONLY when the message is `cannot
set transaction read-write mode during recovery`

for mysql we will also reconnect when we see the following error codes:
- `1040` too many connections 
- `1792` read-only transaction (SQLstate `25006`)
- `1836` running in read-only mode

This logic is easily extensible should we discover more failure modes
over time

## Why?

We've had multiple community reports of Temporal problems during RDS
failover. One part of this is the fact that we wouldn't necessarily
reconnect; we were at the whims of our chosen SQL abstraction's
connection pooling logic.

## How did you test it?

I manually tested this functionality in the presence of repeated RDS
failovers: 

- [x] postgres12 plugin with pq driver
- [x] postgres12 plugin with pgx driver
- [x] mysql plugin

Automated testing will be added to our regular testing pipelines once
our infrastructure friends have added the support I need (it's in
progress)

## Potential risks

We're concerned there's a correctness issue in our PostgreSQL backend
that's related to our behavior during an RDS failover. If we merge this
before I figure out what's going on, we could hide the issue and make it
harder to reproduce.

## Documentation
N/A

## Is hotfix candidate?
Yes?
2024-05-17 16:59:15 -07:00

135 lines
4.1 KiB
Go

// The MIT License
//
// Copyright (c) 2020 Temporal Technologies Inc. All rights reserved.
//
// Copyright (c) 2020 Uber Technologies, Inc.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
package sql
import (
"go.temporal.io/server/common/config"
"go.temporal.io/server/common/log"
"go.temporal.io/server/common/metrics"
"go.temporal.io/server/common/persistence/sql"
"go.temporal.io/server/common/persistence/sql/sqlplugin"
"go.temporal.io/server/common/resolver"
"go.temporal.io/server/tools/common/schema"
)
type (
// Connection is the connection to database
Connection struct {
dbName string
adminDb sqlplugin.AdminDB
}
)
const dbType = "sql"
var _ schema.DB = (*Connection)(nil)
// NewConnection creates a new connection to database
func NewConnection(cfg *config.SQL, logger log.Logger) (*Connection, error) {
db, err := sql.NewSQLAdminDB(sqlplugin.DbKindUnknown, cfg, resolver.NewNoopResolver(), logger, metrics.NoopMetricsHandler)
if err != nil {
return nil, err
}
return &Connection{
adminDb: db,
dbName: cfg.DatabaseName,
}, nil
}
// CreateSchemaVersionTables sets up the schema version tables
func (c *Connection) CreateSchemaVersionTables() error {
return c.adminDb.CreateSchemaVersionTables()
}
// ReadSchemaVersion returns the current schema version for the keyspace
func (c *Connection) ReadSchemaVersion() (string, error) {
return c.adminDb.ReadSchemaVersion(c.dbName)
}
// UpdateSchemaVersion updates the schema version for the keyspace
func (c *Connection) UpdateSchemaVersion(newVersion string, minCompatibleVersion string) error {
return c.adminDb.UpdateSchemaVersion(c.dbName, newVersion, minCompatibleVersion)
}
// WriteSchemaUpdateLog adds an entry to the schema update history table
func (c *Connection) WriteSchemaUpdateLog(oldVersion string, newVersion string, manifestMD5 string, desc string) error {
return c.adminDb.WriteSchemaUpdateLog(oldVersion, newVersion, manifestMD5, desc)
}
// Exec executes a sql statement
func (c *Connection) Exec(stmt string, args ...interface{}) error {
return c.adminDb.Exec(stmt, args...)
}
// ListTables returns a list of tables in this database
func (c *Connection) ListTables() ([]string, error) {
return c.adminDb.ListTables(c.dbName)
}
// DropTable drops a given table from the database
func (c *Connection) DropTable(name string) error {
return c.adminDb.DropTable(name)
}
// DropAllTables drops all tables from this database
func (c *Connection) DropAllTables() error {
tables, err := c.ListTables()
if err != nil {
return err
}
for _, tab := range tables {
if err := c.DropTable(tab); err != nil {
return err
}
}
return nil
}
// CreateDatabase creates a database if it doesn't exist
func (c *Connection) CreateDatabase(name string) error {
return c.adminDb.CreateDatabase(name)
}
// DropDatabase drops a database
func (c *Connection) DropDatabase(name string) error {
return c.adminDb.DropDatabase(name)
}
// Close closes the sql client
func (c *Connection) Close() {
if c.adminDb != nil {
err := c.adminDb.Close()
if err != nil {
panic("cannot close connection")
}
}
}
// Type gives the type of db
func (c *Connection) Type() string {
return dbType
}