replicate/docs/PROVIDER_COMPATIBILITY.md

Storage Provider Compatibility Guide

This document details S3-compatible storage provider compatibility with Replicate, including known limitations, required configuration, and tested configurations.

Overview

Replicate uses the AWS SDK v2 for S3-compatible storage backends. While most providers implement the S3 API, there are important differences in behavior that can affect Replicate's operation.

Provider-Specific Configuration

AWS S3 (Default)

Status: Fully supported (primary target)

replicas:
  - url: s3://bucket-name/path
    region: us-east-1

Notes:

• No special configuration required
• All features fully supported
• Checksum validation enabled by default

Cloudflare R2

Status: Supported with default configuration

Known Limitations:

• Strict concurrent upload limit (2-3 concurrent uploads max)
• Does not support aws-chunked content encoding
• Does not support request/response checksums

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com
    access-key-id: your-access-key-id
    secret-access-key: your-secret-access-key

Automatic Defaults (applied when R2 endpoint detected):

• sign-payload=true - Signed payloads required
• concurrency=2 - Limits concurrent multipart upload parts
• Checksums disabled automatically

Important: The endpoint must use https:// scheme for R2 detection to work.

Related issues: #948, #947, #940, #941

Backblaze B2 (S3-Compatible API)

Status: Supported with configuration

Known Limitations:

• Requires signed payloads for all requests
• Specific authentication endpoint required

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.REGION.backblazeb2.com&sign-payload=true&force-path-style=true
    access-key-id: your-key-id
    secret-access-key: your-application-key

Required Settings:

• sign-payload=true - Required for B2 authentication
• force-path-style=true - Required for bucket access
• Endpoint format: https://s3.REGION.backblazeb2.com

Related issues: #918, #894

DigitalOcean Spaces

Status: Supported with configuration

Known Limitations:

• Does not support aws-chunked content encoding
• Signature requirements differ from AWS

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://REGION.digitaloceanspaces.com&force-path-style=false
    access-key-id: your-spaces-key
    secret-access-key: your-spaces-secret

Notes:

• Use virtual-hosted style paths (force-path-style=false)
• Checksum features disabled automatically for custom endpoints

Related issues: #943

Hanzo S3

Status: Fully supported

Hanzo S3 (ghcr.io/hanzoai/s3) is the self-hosted S3 target used by the integration suite. It serves the S3 API on port 9000 and reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for its admin identity.

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=http://localhost:9000&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key

Notes:

• Works with default settings; signed payloads and path-style addressing are applied automatically for any endpoint with an explicit port.
• Virtual-hosted style addressing requires starting the server with -s3.domainName=<suffix>.

Scaleway Object Storage

Status: Supported with configuration

Known Limitations:

• MissingContentLength errors with streaming uploads
• Requires Content-Length header

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.REGION.scw.cloud&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key

Related issues: #912

Hetzner Object Storage

Status: Supported with configuration

Known Limitations:

• InvalidArgument errors with default AWS SDK settings
• Does not support aws-chunked content encoding
• Requires signed payloads for all requests

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://REGION.your-objectstorage.com&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key

Filebase

Status: Supported with configuration

Known Limitations:

• Authentication failures with default SDK settings after SDK v2 migration

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.filebase.com&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key

Tigris

Status: Supported with configuration

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://fly.storage.tigris.dev&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key

Supabase Storage (S3-Compatible API)

Status: Supported (auto-detected)

Known Limitations:

• Requires path-style URLs (force-path-style=true)
• No S3 versioning support

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://PROJECT_REF.supabase.co/storage/v1/s3
    access-key-id: your-s3-access-key
    secret-access-key: your-s3-secret-key

Automatic Defaults (applied when Supabase endpoint detected):

• sign-payload=true - Signed payloads required
• force-path-style=true - Path-style URLs required

Notes:

• S3 access keys are generated in the Supabase dashboard under Storage > S3 Access Keys
• Endpoint format: https://<PROJECT_REF>.supabase.co/storage/v1/s3

Related issues: #1133

Wasabi

Status: Supported

Configuration:

replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.REGION.wasabisys.com
    access-key-id: your-access-key
    secret-access-key: your-secret-key

SFTP

Status: Supported

replicas:
  - url: sftp://hostname/path
    user: username
    password: password  # or use key-path

Configuration Reference

S3 Query Parameters

Parameters with an alias accept both camelCase and hyphenated forms (e.g., forcePathStyle or force-path-style).

Parameter	Alias	Description	Default
endpoint		Custom S3 endpoint URL	AWS S3
region		AWS region	Auto-detected
forcePathStyle	force-path-style	Use path-style URLs	false (auto for custom endpoints)
skipVerify	skip-verify	Skip TLS verification	false
signPayload	sign-payload	Sign request payloads	true
requireContentMD5	require-content-md5	Require Content-MD5 header	true
concurrency		Multipart upload concurrency	5
partSize	part-size	Multipart upload part size	5MB
sseCustomerAlgorithm	sse-customer-algorithm	SSE-C encryption algorithm	None
sseCustomerKey	sse-customer-key	SSE-C encryption key	None
sseCustomerKeyMD5	sse-customer-key-md5	SSE-C key MD5 checksum	None
sseKmsKeyId	sse-kms-key-id	KMS key for encryption	None

Provider Detection

Replicate automatically detects certain providers and applies appropriate defaults:

Provider	Detection Pattern	Applied Settings
Hetzner	*.your-objectstorage.com	sign-payload=true
Cloudflare R2	*.r2.cloudflarestorage.com	sign-payload=true, concurrency=2
Backblaze B2	*.backblazeb2.com	sign-payload=true, force-path-style=true
DigitalOcean	*.digitaloceanspaces.com	sign-payload=true
Scaleway	*.scw.cloud	sign-payload=true
Filebase	s3.filebase.com	sign-payload=true, force-path-style=true
Tigris	*.tigris.dev	sign-payload=true, require-content-md5=false
Self-hosted S3 (e.g. Hanzo S3)	host with an explicit port that is not a known cloud provider	sign-payload=true, force-path-style=true
Supabase	*.supabase.co	sign-payload=true, force-path-style=true

Troubleshooting

Common Errors

InvalidArgument: Unsupported content encoding: aws-chunked

• Provider doesn't support AWS SDK v2 chunked encoding
• Use a custom endpoint with automatic checksum disabling
• Or explicitly disable checksums

SignatureDoesNotMatch

• Try sign-payload=true in the URL
• Verify credentials are correct
• Check endpoint URL format

MissingContentLength

• Provider requires Content-Length header
• This is handled automatically for known providers

Too many concurrent uploads or timeout errors

• Reduce concurrency: ?concurrency=2
• Particularly important for Cloudflare R2

AccessDenied or authentication failures

• Verify credentials
• Check IAM/bucket permissions
• For B2, ensure sign-payload=true

Debug Mode

Enable verbose logging to diagnose issues:

REPLICATE_DEBUG=1 replicate replicate ...

Or in configuration:

logging:
  level: debug

Testing Your Configuration

Test connectivity without starting replication:

# List any existing backups
replicate snapshots s3://bucket/path?endpoint=...

# Perform a test restore (requires existing backup)
replicate restore -o /tmp/test.db s3://bucket/path?endpoint=...

Version Compatibility

• Replicate v0.5.x: AWS SDK v2, improved provider compatibility
• Replicate v0.4.x: AWS SDK v1, different authentication handling
• Replicate v0.3.x: Legacy format — v0.5.x can restore from v0.3.x backups via ReplicaClientV3 interface

When upgrading from v0.3.x, v0.5.x can automatically restore from v0.3.x backups if no v0.4.x+ backup exists. The S3 backend implements ReplicaClientV3 to read the v0.3.x generations/{id}/snapshots/ and generations/{id}/wal/ directory structure. See REPLICA_CLIENT_GUIDE.md for details.

Validation Interval

Replicate supports periodic validation of replica integrity:

validation:
  interval: "5m"  # How often to validate; 0 disables

This runs Store.Validate() on a ticker, comparing local and remote positions. Can also be set per-replica via the validation-interval replica config key. and cannot restore backups created by v0.3.x. See the upgrade guide for migration instructions.

Reporting Issues

When reporting provider compatibility issues, please include:

1. Provider name and region
2. Replicate version (replicate version)
3. Full error message
4. Configuration (with credentials redacted)
5. Whether the issue is with replication, restore, or both

File issues at: GitHub Issues