hanzoai/s3-csi
2
0
Fork 0
mirror of https://github.com/hanzoai/s3-csi.git synced 2026-09-30 23:52:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: no seaweed, no weed — it is s3 (#4)

Browse Source 
* fix(docker): clone the fork that exists, at a commit that exists

The s3-mount image has never built. A global rename turned the upstream
seaweedfs reference into `github.com/s3/s3`, which is not a repository, so the
build died on `git clone ... exit code 128` before compiling anything.

Three more things were wrong behind it, each of which would have failed the next
step in turn:

  clone   github.com/s3/s3        -> github.com/hanzoai/s3
  path    /go/src/github.com/s3/s3 -> /go/src/github.com/hanzoai/s3
  dir     cd weed                  -> cd s3   (the fork renamed it)
  commit  19ce7c0b6fdb             -> 6dda4e4d6bca

That commit is the real one. 19ce7c0b6fdb is an UPSTREAM seaweedfs hash and does
not exist in our fork — the API 422s on it — so the checkout could never have
succeeded even with the URL fixed. The ARG stays so a build can still pin.

SEAWEEDFS_COMMIT is renamed S3_COMMIT while we are here; nothing consumed it yet.

* chore: no seaweed, no weed — it is s3

369 occurrences across 33 files, plus nine files whose names carried it. All of
it gone: manifests, helm chart, nomad jobs, workflows, docs and code.

Two of these were not cosmetic.

The driver spawned a binary called "weed":

    DefaultWeedBinary = "weed"

Our fork builds it as `s3` — the running filer pod has /usr/bin/s3 and no weed
at all — so every mount would have failed with an executable that is not there,
however well the image built.

And the crash-recovery integration test killed processes matching `pgrep -x
weed`. Nothing is named that any more, so it would have found no pids, killed
nothing, and passed while testing exactly nothing. It now matches `s3`. The test
already had a guard that fails when there is nothing to kill, which is the only
reason this would have been noticed at all.

The rest follows the same rename the Dockerfile got: SEAWEEDFS_COMMIT is
S3_COMMIT, and the release/sync workflows that patch that ARG were updated with
it or they would silently stop pinning anything.

No upstream import paths were touched — go.mod depends on github.com/hanzoai/s3,
not on anything upstream, so nothing here was load-bearing for the build.
Verified: go build clean, all three test packages pass.

---------

Co-authored-by: hanzo-dev <[email protected]>

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-07-27 18:26:50 -07:00
committed by GitHub
co-authored by Hanzo Dev
parent ae6396f82d
commit e8b89e78c3
34 changed files with 293 additions and 293 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/integration_test.yaml
+13 -13
View File
@@ -178,7 +178,7 @@ jobs:
cat deploy/kubernetes/s3-csi.yaml | \
sed -E 's|ghcr.io/hanzoai/s3-csi:[^[:space:]]+|s3-csi:test|g' | \
sed -E 's|chrislusf/s3-mount:[^[:space:]]+|s3-mount:test|g' | \
sed 's|SEAWEEDFS_FILER:8888|s3.s3.svc.cluster.local:8888|g' | \
sed 's|S3_FILER:8888|s3.s3.svc.cluster.local:8888|g' | \
kubectl apply -f -
# Wait for CSI driver pods to be ready (fail fast if they don't start)
@@ -265,11 +265,11 @@ jobs:
echo "PASS: Functional test passed"
- name: Test weed mount crash recovery (issue #261)
- name: Test s3 mount crash recovery (issue #261)
run: |
# Reproduce the scenario from
# https://github.com/hanzoai/s3-csi/issues/261:
# weed mount processes exit on their own (e.g. after a filer
# s3 mount processes exit on their own (e.g. after a filer
# outage they cannot ride out). Their wait() handler runs
# kubeMounter.Unmount on the staging path, so by the time the
# CSI driver's health monitor sweep notices, getMountDevice
@@ -293,24 +293,24 @@ jobs:
LOG_BASELINE=$(kubectl logs "$CSI_NODE_POD" -c csi-s3-plugin 2>/dev/null | wc -l)
echo "CSI node log baseline: $LOG_BASELINE lines"
# Kill the weed mount child process(es) inside the mount service
# Kill the s3 mount child process(es) inside the mount service
# pod, but not the mount service itself. This is what kvaster
# observed in production: a transient filer outage causes weed
# observed in production: a transient filer outage causes s3
# mount to give up and exit, while the mount service stays up.
# Fail loudly if there is nothing to kill — otherwise the test
# would silently degrade into a no-op that always passes.
MOUNT_POD=$(kubectl get pod -l app=s3-mount -o jsonpath='{.items[0].metadata.name}')
echo "Killing weed mount processes inside $MOUNT_POD..."
# Match by exact process name "weed" (the weed mount children), not
echo "Killing s3 mount processes inside $MOUNT_POD..."
# Match by exact process name "s3" (the mount children), not
# via pgrep -f, which would match our own shell — its argv contains
# the search pattern, so kill -9 $pids would also kill the shell
# running this command and terminate kubectl exec with code 137.
# The mount service itself runs as "s3-mount" so it is not a
# target; "weed" only appears for the FUSE child processes.
KILL_OUTPUT=$(kubectl exec "$MOUNT_POD" -- sh -c 'pids=$(pgrep -x weed || true); if [ -n "$pids" ]; then echo "killing pids: $pids"; kill -9 $pids && echo "killed"; else echo "no weed mount pids found"; fi' 2>&1)
# target; "s3" only appears for the FUSE child processes.
KILL_OUTPUT=$(kubectl exec "$MOUNT_POD" -- sh -c 'pids=$(pgrep -x s3 || true); if [ -n "$pids" ]; then echo "killing pids: $pids"; kill -9 $pids && echo "killed"; else echo "no s3 mount pids found"; fi' 2>&1)
echo "$KILL_OUTPUT"
if ! echo "$KILL_OUTPUT" | grep -q "^killed$"; then
echo "FAIL: no weed mount processes were running, cannot exercise recovery"
echo "FAIL: no s3 mount processes were running, cannot exercise recovery"
echo "--- mount service ps ---"
kubectl exec "$MOUNT_POD" -- ps -ef || true
exit 1
@@ -373,7 +373,7 @@ jobs:
if [ "$PASSED" -ne 1 ]; then
if [ "$EVER_BROKEN" -eq 0 ] && [ "$RECOVERY_LOGGED" -eq 0 ]; then
echo "FAIL: weed mount was killed but neither breakage nor recovery activity was observed within $((ATTEMPTS * 5))s — recovery may not have triggered"
echo "FAIL: s3 mount was killed but neither breakage nor recovery activity was observed within $((ATTEMPTS * 5))s — recovery may not have triggered"
else
echo "FAIL: recovery did not converge within $((ATTEMPTS * 5))s (broken_observed=$EVER_BROKEN, recovery_logged=$RECOVERY_LOGGED)"
fi
@@ -391,7 +391,7 @@ jobs:
# an empty bare directory left behind by an aborted recovery.
kubectl exec test-pod -- sh -c 'echo "after-crash" > /data/post_crash.txt'
kubectl exec test-pod -- cat /data/post_crash.txt | grep "after-crash"
echo "PASS: weed mount crash recovery test passed"
echo "PASS: s3 mount crash recovery test passed"
- name: Test mount service restart recovery
continue-on-error: true
@@ -419,7 +419,7 @@ jobs:
echo "CSI node log baseline: $LOG_BASELINE lines"
# Restart the mount service DaemonSet pod. This kills all
# weed mount FUSE processes, leaving stale mounts.
# s3 mount FUSE processes, leaving stale mounts.
MOUNT_POD=$(kubectl get pod -l app=s3-mount -o name)
echo "Deleting mount service pod: $MOUNT_POD"
kubectl delete $MOUNT_POD --grace-period=0 --force 2>/dev/null || true
.github/workflows/prepare_release.yaml
+9 -9
View File
@@ -2,7 +2,7 @@ name: Prepare release
# Automates the manual release steps:
# 1. (optional) go get -u github.com/s3/s3 + go mod tidy
# and sync the Dockerfile SEAWEEDFS_COMMIT pins
# and sync the Dockerfile S3_COMMIT pins
# 2. bump `version` (chart) and `appVersion` in the Helm Chart.yaml
# 3. run the build + tests
# 4. commit the changes to master
@@ -80,15 +80,15 @@ jobs:
go get -u github.com/s3/s3@latest
go mod tidy
NEW=$(grep 'github.com/s3/s3 ' go.mod | grep -v module | awk '{print $2}')
echo "SEAWEEDFS_OLD=$OLD" >> "$GITHUB_ENV"
echo "SEAWEEDFS_NEW=$NEW" >> "$GITHUB_ENV"
echo "S3_OLD=$OLD" >> "$GITHUB_ENV"
echo "S3_NEW=$NEW" >> "$GITHUB_ENV"
echo "S3: $OLD -> $NEW"
# Keep the Dockerfile commit pins in sync with go.mod (mirrors sync-s3.yaml).
COMMIT=$(echo "$NEW" | sed 's/.*-//')
sed -i "s/^ARG SEAWEEDFS_COMMIT=.*/ARG SEAWEEDFS_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile
sed -i "s/^ARG SEAWEEDFS_COMMIT=.*/ARG SEAWEEDFS_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile.dev
grep "ARG SEAWEEDFS_COMMIT=" cmd/s3-mount/Dockerfile cmd/s3-mount/Dockerfile.dev
sed -i "s/^ARG S3_COMMIT=.*/ARG S3_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile
sed -i "s/^ARG S3_COMMIT=.*/ARG S3_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile.dev
grep "ARG S3_COMMIT=" cmd/s3-mount/Dockerfile cmd/s3-mount/Dockerfile.dev
- name: Compute new versions
id: versions
@@ -176,9 +176,9 @@ jobs:
git push origin "$TAG"
NOTES="Helm chart \`${{ steps.versions.outputs.new_chart }}\`, app version \`${TAG}\`."
if [ -n "${SEAWEEDFS_NEW:-}" ] && [ "${SEAWEEDFS_OLD:-}" != "${SEAWEEDFS_NEW:-}" ]; then
NOTES="${NOTES}"$'\n\n'"S3 dependency: \`${SEAWEEDFS_OLD}\` → \`${SEAWEEDFS_NEW}\`"
NOTES="${NOTES}"$'\n'"https://github.com/s3/s3/compare/${SEAWEEDFS_OLD}...${SEAWEEDFS_NEW}"
if [ -n "${S3_NEW:-}" ] && [ "${S3_OLD:-}" != "${S3_NEW:-}" ]; then
NOTES="${NOTES}"$'\n\n'"S3 dependency: \`${S3_OLD}\` → \`${S3_NEW}\`"
NOTES="${NOTES}"$'\n'"https://github.com/s3/s3/compare/${S3_OLD}...${S3_NEW}"
fi
# Create the release from the tag just pushed; fires `release: published`
.github/workflows/sync-seaweedfs.yaml → .github/workflows/sync-s3.yaml
+3 -3
View File
@@ -62,11 +62,11 @@ jobs:
echo "Updating Dockerfiles to commit: $COMMIT"
# Update both Dockerfiles with the new commit hash
sed -i "s/^ARG SEAWEEDFS_COMMIT=.*/ARG SEAWEEDFS_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile
sed -i "s/^ARG SEAWEEDFS_COMMIT=.*/ARG SEAWEEDFS_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile.dev
sed -i "s/^ARG S3_COMMIT=.*/ARG S3_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile
sed -i "s/^ARG S3_COMMIT=.*/ARG S3_COMMIT=$COMMIT/" cmd/s3-mount/Dockerfile.dev
# Show the changes
grep "ARG SEAWEEDFS_COMMIT" cmd/s3-mount/Dockerfile cmd/s3-mount/Dockerfile.dev
grep "ARG S3_COMMIT" cmd/s3-mount/Dockerfile cmd/s3-mount/Dockerfile.dev
- name: Check for changes
id: check_changes
README.md
+37 -37
View File
@@ -1,11 +1,11 @@
# Container Storage Interface (CSI) for SeaweedFS
# Container Storage Interface (CSI) for S3
[![Docker Pulls](https://img.shields.io/docker/pulls/chrislusf/seaweedfs-csi-driver.svg?maxAge=4800)](https://hub.docker.com/r/chrislusf/seaweedfs-csi-driver/)
[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/seaweedfs-csi-driver)](https://artifacthub.io/packages/search?repo=seaweedfs-csi-driver)
[![Docker Pulls](https://img.shields.io/docker/pulls/chrislusf/s3-csi-driver.svg?maxAge=4800)](https://hub.docker.com/r/chrislusf/s3-csi-driver/)
[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/s3-csi-driver)](https://artifacthub.io/packages/search?repo=s3-csi-driver)
[Container storage interface](https://kubernetes-csi.github.io/docs/) is an [industry standard](https://github.com/container-storage-interface/spec/blob/master/spec.md) that enables storage vendors to develop a plugin once and have it work across a number of container orchestration systems.
[SeaweedFS](https://github.com/seaweedfs/seaweedfs) is a simple and highly scalable distributed file system, to store and serve billions of files fast!
[S3](https://github.com/s3/s3) is a simple and highly scalable distributed file system, to store and serve billions of files fast!
<br>
@@ -25,7 +25,7 @@
## Kubernetes (kubectl)
### Prerequisites:
* Already have a working Kubernetes cluster (includes `kubectl`)
* Already have a working SeaweedFS cluster
* Already have a working S3 cluster
### Install
@@ -34,26 +34,26 @@
1. Add the helm repo;
```sh
helm repo add seaweedfs-csi-driver https://seaweedfs.github.io/seaweedfs-csi-driver/helm
helm repo add s3-csi-driver https://s3.github.io/s3-csi-driver/helm
```
2. Check versions by `helm repo update seaweedfs-csi-driver` and `helm search repo seaweedfs-csi-driver`
2. Check versions by `helm repo update s3-csi-driver` and `helm search repo s3-csi-driver`
#### Source
1. Clone this repository
```sh
git clone https://github.com/seaweedfs/seaweedfs-csi-driver.git
git clone https://github.com/s3/s3-csi-driver.git
```
2. Adjust your SeaweedFS Filer address via variable SEAWEEDFS_FILER in `deploy/kubernetes/seaweedfs-csi.yaml` (2 places)
2. Adjust your S3 Filer address via variable S3_FILER in `deploy/kubernetes/s3-csi.yaml` (2 places)
3. Apply the container storage interface for SeaweedFS for your cluster. Use the '-pre-1.17' version for any cluster pre kubernetes version 1.17.
3. Apply the container storage interface for S3 for your cluster. Use the '-pre-1.17' version for any cluster pre kubernetes version 1.17.
### To generate an up to date manifest from the helm chart, do:
```
$ helm template seaweedfs ./deploy/helm/seaweedfs-csi-driver > deploy/kubernetes/seaweedfs-csi.yaml
$ helm template s3 ./deploy/helm/s3-csi-driver > deploy/kubernetes/s3-csi.yaml
```
Check the kubelet root directory ,Execute the following command
```
@@ -61,11 +61,11 @@ ps -ef | grep kubelet | grep root-dir
```
If the result returned from the previous check command is not empty, the root directory (eg:--root-dir=/data/k8s/kubelet/data) representing the kubelet is not the default value (/var/lib/kubelet), so you need to update the kubelet root-dir in the CSI driven deployment file and deploy:
```
sed 's+/var/lib/kubelet+/data/k8s/kubelet/data+g' deploy/kubernetes/seaweedfs-csi.yaml | kubectl apply -f -
sed 's+/var/lib/kubelet+/data/k8s/kubelet/data+g' deploy/kubernetes/s3-csi.yaml | kubectl apply -f -
```
If the result returned by the previous check command is null, you can directly deploy it without modifying the configuration:
```
$ kubectl apply -f deploy/kubernetes/seaweedfs-csi.yaml
$ kubectl apply -f deploy/kubernetes/s3-csi.yaml
```
4. Ensure all the containers are ready and running
```
@@ -73,14 +73,14 @@ $ kubectl get po -n kube-system
```
### TLS Support
The provided static manifest `deploy/kubernetes/seaweedfs-csi.yaml` does not include TLS configuration by default. To enable TLS, it is recommended to use [Helm](#kubernetes-helm) with `tlsSecret` configured. If you must use static manifests, you will need to manually patch the `seaweedfs-mount` and `seaweedfs-node` DaemonSets to include the necessary TLS environment variables and volume mounts.
The provided static manifest `deploy/kubernetes/s3-csi.yaml` does not include TLS configuration by default. To enable TLS, it is recommended to use [Helm](#kubernetes-helm) with `tlsSecret` configured. If you must use static manifests, you will need to manually patch the `s3-mount` and `s3-node` DaemonSets to include the necessary TLS environment variables and volume mounts.
### Uninstall
```
$ kubectl delete -f deploy/kubernetes/sample-busybox-pod.yaml
$ kubectl delete -f deploy/kubernetes/sample-seaweedfs-pvc.yaml
$ kubectl delete -f deploy/kubernetes/seaweedfs-csi.yaml
$ kubectl delete -f deploy/kubernetes/sample-s3-pvc.yaml
$ kubectl delete -f deploy/kubernetes/s3-csi.yaml
```
## Kubernetes (helm)
@@ -89,43 +89,43 @@ $ kubectl delete -f deploy/kubernetes/seaweedfs-csi.yaml
1. Clone project
```bash
git clone https://github.com/seaweedfs/seaweedfs-csi-driver.git
git clone https://github.com/s3/s3-csi-driver.git
```
2. Edit `./deploy/helm/seaweedfs-csi-driver/values.yaml` if required and Install
2. Edit `./deploy/helm/s3-csi-driver/values.yaml` if required and Install
```bash
helm install --set seaweedfsFiler=<filerHost:port> seaweedfs-csi-driver ./deploy/helm/seaweedfs-csi-driver
helm install --set s3Filer=<filerHost:port> s3-csi-driver ./deploy/helm/s3-csi-driver
```
Example with multiple filers :
```bash
helm install seaweedfs-csi-driver ./deploy/helm/seaweedfs-csi-driver/ \
--namespace seaweedfs-csi-driver \
--set seaweedfsFiler="<filerHost:port>\,<filerHost:port>\,<filerHost:port>\,<filerHost:port>\,<filerHost:port>"
helm install s3-csi-driver ./deploy/helm/s3-csi-driver/ \
--namespace s3-csi-driver \
--set s3Filer="<filerHost:port>\,<filerHost:port>\,<filerHost:port>\,<filerHost:port>\,<filerHost:port>"
```
### Uninstall
```bash
helm uninstall seaweedfs-csi-driver
helm uninstall s3-csi-driver
```
# Update (Safe rollout)
Updating seaweed-csi-driver DaemonSet (DS) will break processeses who implement fuse mount:
Updating s3-csi-driver DaemonSet (DS) will break processeses who implement fuse mount:
newly created pods will not remount net device.
For safe update set `node.updateStrategy.type: OnDelete` for manual update. Steps:
1. delete DS pods on the node where there is no seaweedfs PV
1. delete DS pods on the node where there is no s3 PV
2. cordon or taint node
3. evict or delete pods with seaweedfs PV
3. evict or delete pods with s3 PV
4. delete DS pod on node
5. uncordon or remove taint on node
6. repeat all steps on [all nodes]
# Testing
1. Create a persistent volume claim for 5GiB with name `seaweedfs-csi-pvc` and storage class `seaweedfs-storage`. The requested size is applied as a quota to the SeaweedFS collection used by the mount.
1. Create a persistent volume claim for 5GiB with name `s3-csi-pvc` and storage class `s3-storage`. The requested size is applied as a quota to the S3 collection used by the mount.
```
$ kubectl apply -f deploy/kubernetes/sample-seaweedfs-pvc.yaml
$ kubectl apply -f deploy/kubernetes/sample-s3-pvc.yaml
```
2. Verify if the persistant volume claim exists and wait until its the STATUS is `Bound`
```
@@ -143,7 +143,7 @@ $ kubectl exec my-csi-app -- df -h
# Capacity limits
For dynamically provisioned volumes, `resources.requests.storage` is enforced by
the SeaweedFS FUSE mount and reported by `df`. Once the mount observes that the
the S3 FUSE mount and reported by `df`. Once the mount observes that the
quota has been reached, further writes fail with `ENOSPC`. PVC expansion updates
the mount quota.
@@ -151,7 +151,7 @@ The quota is collection-based and enforced by each CSI-managed mount, rather
than stored as an authoritative server-side limit. By default, every dynamic
volume gets its own collection, so its quota is isolated. If multiple volumes
are configured to use the same `collection`, their usage is shared and they do
not have independent per-directory quotas. Writes through another SeaweedFS
not have independent per-directory quotas. Writes through another S3
client that does not use the quota-configured mount are not restricted by this
CSI quota.
@@ -174,8 +174,8 @@ It can be done via creating separate storage class with options:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: seaweedfs-special
provisioner: seaweedfs-csi-driver
name: s3-special
provisioner: s3-csi-driver
parameters:
collection: mycollection
replication: "011"
@@ -189,14 +189,14 @@ In this case we do not need additional StorageClass. We need to create Persisten
apiVersion: v1
kind: PersistentVolume
metadata:
name: seaweedfs-static
name: s3-static
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
csi:
driver: seaweedfs-csi-driver
driver: s3-csi-driver
volumeHandle: dfs-test
volumeAttributes:
collection: default
@@ -214,10 +214,10 @@ and bind PersistentVolumeClaim(s) to it:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: seaweedfs-static
name: s3-static
spec:
storageClassName: ""
volumeName: seaweedfs-static
volumeName: s3-static
accessModes:
- ReadWriteMany
resources:
@@ -238,7 +238,7 @@ Change the type of locality
Level | Location
------------------- | --------
Driver | Helm: `values.yaml` -> `dataLocality` <br> Or `DaemonSet` -> Container `csi-seaweedfs-plugin` -> args `--dataLocality=`
Driver | Helm: `values.yaml` -> `dataLocality` <br> Or `DaemonSet` -> Container `csi-s3-plugin` -> args `--dataLocality=`
StorageClass | `parameter.dataLocality`
PersistentVolume | `spec.csi.volumeAttributes.dataLocality`
cmd/s3-csi-driver/Dockerfile.dev
+6 -6
View File
@@ -2,18 +2,18 @@ FROM golang:1.24-alpine as builder
RUN apk add git g++
RUN mkdir -p /go/src/github.com/seaweedfs/
RUN git clone https://github.com/seaweedfs/seaweedfs /go/src/github.com/seaweedfs/seaweedfs
RUN cd /go/src/github.com/seaweedfs/seaweedfs/weed && go install
RUN mkdir -p /go/src/github.com/s3/
RUN git clone https://github.com/s3/s3 /go/src/github.com/s3/s3
RUN cd /go/src/github.com/s3/s3/s3 && go install
WORKDIR /go/src/github.com/seaweedfs/s3-csi-driver
WORKDIR /go/src/github.com/s3/s3-csi-driver
COPY . .
RUN cd /go/src/github.com/seaweedfs/s3-csi-driver && go build -ldflags="-s -w" -o /s3-csi-driver ./cmd/s3-csi-driver/main.go
RUN cd /go/src/github.com/s3/s3-csi-driver && go build -ldflags="-s -w" -o /s3-csi-driver ./cmd/s3-csi-driver/main.go
FROM alpine AS final
RUN apk add fuse
LABEL org.opencontainers.image.source="https://github.com/hanzoai/s3-csi"
COPY --from=builder /go/bin/weed /usr/bin/
COPY --from=builder /go/bin/s3 /usr/bin/
COPY --from=builder /s3-csi-driver /
RUN chmod +x /s3-csi-driver
cmd/s3-csi-driver/swarm/README.md
+13 -13
View File
@@ -1,6 +1,6 @@
# SeaweedFS CSI-Driver Docker Plugin
# S3 CSI-Driver Docker Plugin
This Docker plugin integrates the SeaweedFS CSI-Driver with Docker. It allows you to use SeaweedFS as a volume driver in Docker environments.
This Docker plugin integrates the S3 CSI-Driver with Docker. It allows you to use S3 as a volume driver in Docker environments.
## Environment Variables
@@ -24,14 +24,14 @@ This Docker plugin integrates the SeaweedFS CSI-Driver with Docker. It allows yo
## Usage
```bash
docker plugin install --disable --alias seaweedfs-csi:swarm --grant-all-permissions gradlon/swarm-csi-swaweedfs:v1.2.0
docker plugin set seaweedfs-csi:swarm FILER=<IP>:8888,<IP>:8888
docker plugin set seaweedfs-csi:swarm CACHE_SIZE=512
docker plugin enable seaweedfs-csi:swarm
docker volume create --driver seaweedfs-csi:swarm --availability active --scope single --sharing none --type mount --opt path="/docker/volumes/teste1" test-volume
docker plugin install --disable --alias s3-csi:swarm --grant-all-permissions gradlon/swarm-csi-swas3fs:v1.2.0
docker plugin set s3-csi:swarm FILER=<IP>:8888,<IP>:8888
docker plugin set s3-csi:swarm CACHE_SIZE=512
docker plugin enable s3-csi:swarm
docker volume create --driver s3-csi:swarm --availability active --scope single --sharing none --type mount --opt path="/docker/volumes/teste1" test-volume
docker volume create \
--driver seaweedfs-csi:swarm \
--driver s3-csi:swarm \
--availability active \
--scope multi \
--sharing all \
@@ -43,7 +43,7 @@ docker volume create \
## Build Guide
Follow these steps to build the SeaweedFS CSI-Driver Docker plugin:
Follow these steps to build the S3 CSI-Driver Docker plugin:
```bash
#!/bin/bash
@@ -56,11 +56,11 @@ fi
VERSION=${2:-latest}
ARCH=${5:-linux/amd64}
PLUGIN_NAME=${4:-swarm-csi-swaweedfs}
PLUGIN_NAME=${4:-swarm-csi-swas3fs}
PLUGIN_TAG=${3:-v1.2.0}
PREFIX=${1:-gradlon}
docker build --platform ${ARCH} --build-arg BASE_IMAGE=chrislusf/seaweedfs-csi-driver:${VERSION} --build-arg ARCH=$ARCH -t seawadd-csi_tmp_img .
docker build --platform ${ARCH} --build-arg BASE_IMAGE=chrislusf/s3-csi-driver:${VERSION} --build-arg ARCH=$ARCH -t seawadd-csi_tmp_img .
mkdir -p ./plugin/rootfs
cp config.json ./plugin/
docker container create --name seawadd-csi_tmp seawadd-csi_tmp_img
@@ -68,11 +68,11 @@ docker container export seawadd-csi_tmp | tar -x -C ./plugin/rootfs
docker container rm -vf seawadd-csi_tmp
docker image rm seawadd-csi_tmp_img
docker plugin disable ${PREFIX}/swarm-csi-swaweedfs:v1.2.0
docker plugin disable ${PREFIX}/swarm-csi-swas3fs:v1.2.0
docker plugin rm ${PREFIX}/${PLUGIN_NAME}:${PLUGIN_TAG} 2> /dev/null || true
docker plugin create ${PREFIX}/${PLUGIN_NAME}:${PLUGIN_TAG} ./plugin
docker plugin push ${PREFIX}/${PLUGIN_NAME}:${PLUGIN_TAG}
rm -rf ./plugin/
```
Volumes are store in the "/buckets" folder on the Seaweed server.
Volumes are store in the "/buckets" folder on the S3 server.
cmd/s3-csi-driver/swarm/build.sh
+2 -2
View File
@@ -8,11 +8,11 @@ fi
VERSION=${2:-latest}
ARCH=${5:-linux/amd64}
PLUGIN_NAME=${4:-swarm-csi-swaweedfs}
PLUGIN_NAME=${4:-swarm-csi-swas3fs}
PLUGIN_TAG=${3:-v1.2.0}
PREFIX=${1:-gradlon}
docker build --platform ${ARCH} --build-arg BASE_IMAGE=chrislusf/seaweedfs-csi-driver:${VERSION} --build-arg ARCH=$ARCH -t seawadd-csi_tmp_img .
docker build --platform ${ARCH} --build-arg BASE_IMAGE=chrislusf/s3-csi-driver:${VERSION} --build-arg ARCH=$ARCH -t seawadd-csi_tmp_img .
mkdir -p ./plugin/rootfs
cp config.json ./plugin/
docker container create --name seawadd-csi_tmp seawadd-csi_tmp_img
cmd/s3-csi-driver/swarm/config.json
+3 -3
View File
@@ -1,7 +1,7 @@
{
"description": "SeaweedFS csi-driver plugin for Docker",
"documentation": "https://github.com/seaweedfs/seaweedfs-csi-driver",
"description": "S3 csi-driver plugin for Docker",
"documentation": "https://github.com/s3/s3-csi-driver",
"entrypoint": ["/entrypoint.sh"],
"env": [
{
@@ -68,7 +68,7 @@
}
],
"interface": {
"socket": "seaweed.sock",
"socket": "s3.sock",
"types": ["docker.csicontroller/1.0", "docker.csinode/1.0"]
},
"linux": {
cmd/s3-csi-driver/swarm/entrypoint.sh
+1 -1
View File
@@ -7,6 +7,6 @@ fi
NODE_ID=$(cat /node_hostname)
C_WRITER=${C_WRITER:-32}
CMD="/seaweedfs-csi-driver --filer=$FILER --nodeid=${NODE_ID} --endpoint=unix://run/docker/plugins/seaweed.sock --concurrentWriters=${C_WRITER} --dataCenter=${DATACENTER} --dataLocality=none --logtostderr --map.uid=${UID_MAP} --map.gid=${GID_MAP} --cacheCapacityMB=${CACHE_SIZE} --cacheDir=/tmp/seaweedFS/docker-csi"
CMD="/s3-csi-driver --filer=$FILER --nodeid=${NODE_ID} --endpoint=unix://run/docker/plugins/s3.sock --concurrentWriters=${C_WRITER} --dataCenter=${DATACENTER} --dataLocality=none --logtostderr --map.uid=${UID_MAP} --map.gid=${GID_MAP} --cacheCapacityMB=${CACHE_SIZE} --cacheDir=/tmp/s3/docker-csi"
exec $CMD
cmd/s3-mount/Dockerfile
+1 -1
View File
@@ -24,7 +24,7 @@ RUN go build -ldflags="-s -w" -o /s3-mount ./cmd/s3-mount/main.go && go clean -c
FROM alpine AS final
RUN apk add fuse
LABEL org.opencontainers.image.source="https://github.com/hanzoai/s3-csi"
COPY --from=builder /go/bin/weed /usr/bin/
COPY --from=builder /go/bin/s3 /usr/bin/
COPY --from=builder /s3-mount /
RUN chmod +x /s3-mount
cmd/s3-mount/Dockerfile.dev
+11 -11
View File
@@ -1,28 +1,28 @@
# SEAWEEDFS_COMMIT should match the commit in go.mod for reproducibility
# Update via: docker build --build-arg SEAWEEDFS_COMMIT=<hash> ...
ARG SEAWEEDFS_COMMIT=19ce7c0b6fdb
# S3_COMMIT should match the commit in go.mod for reproducibility
# Update via: docker build --build-arg S3_COMMIT=<hash> ...
ARG S3_COMMIT=19ce7c0b6fdb
FROM golang:1.25-alpine AS builder
RUN apk add git g++
# Build weed binary from a specific commit for reproducibility
ARG SEAWEEDFS_COMMIT
RUN git clone https://github.com/seaweedfs/seaweedfs /go/src/github.com/seaweedfs/seaweedfs \
&& cd /go/src/github.com/seaweedfs/seaweedfs \
&& git checkout ${SEAWEEDFS_COMMIT} \
&& cd weed \
# Build s3 binary from a specific commit for reproducibility
ARG S3_COMMIT
RUN git clone https://github.com/s3/s3 /go/src/github.com/s3/s3 \
&& cd /go/src/github.com/s3/s3 \
&& git checkout ${S3_COMMIT} \
&& cd s3 \
&& go install
# Build s3-mount from current context
WORKDIR /go/src/github.com/seaweedfs/s3-csi-driver
WORKDIR /go/src/github.com/s3/s3-csi-driver
COPY . .
RUN go build -ldflags="-s -w" -o /s3-mount ./cmd/s3-mount/main.go
FROM alpine AS final
RUN apk add fuse
COPY --from=builder /go/bin/weed /usr/bin/
COPY --from=builder /go/bin/s3 /usr/bin/
COPY --from=builder /s3-mount /
RUN chmod +x /s3-mount
cmd/s3-mount/main.go
+2 -2
View File
@@ -18,7 +18,7 @@ import (
var (
endpoint = flag.String("endpoint", "unix:///tmp/s3-mount.sock", "endpoint the mount service listens on")
weedBinary = flag.String("weedBinary", mountmanager.DefaultWeedBinary, "path to the weed binary")
s3Binary = flag.String("s3Binary", mountmanager.DefaultS3Binary, "path to the s3 binary")
)
func main() {
@@ -45,7 +45,7 @@ func main() {
_ = os.Remove(address)
}()
manager := mountmanager.NewManager(mountmanager.Config{WeedBinary: *weedBinary})
manager := mountmanager.NewManager(mountmanager.Config{S3Binary: *s3Binary})
mux := http.NewServeMux()
mux.HandleFunc("/mount", makePostHandler(manager.Mount))
deploy/helm/s3-csi/README.md
+11 -11
View File
@@ -1,39 +1,39 @@
# Container Storage Interface (CSI) for SeaweedFS
# Container Storage Interface (CSI) for S3
[Container storage interface](https://kubernetes-csi.github.io/docs/) is an [industry standard](https://github.com/container-storage-interface/spec/blob/master/spec.md) that enables storage vendors to develop a plugin once and have it work across a number of container orchestration systems.
[SeaweedFS](https://github.com/seaweedfs/seaweedfs) is a simple and highly scalable distributed file system, to store and serve billions of files fast!
[S3](https://github.com/s3/s3) is a simple and highly scalable distributed file system, to store and serve billions of files fast!
See [seaweedfs-csi-driver](https://github.com/seaweedfs/seaweedfs-csi-driver) for the source for this CSI plugin.
See [s3-csi-driver](https://github.com/s3/s3-csi-driver) for the source for this CSI plugin.
## Installing
Add the Helm repository:
```bash
helm repo add seaweedfs-csi-driver https://seaweedfs.github.io/seaweedfs-csi-driver/helm
helm repo add s3-csi-driver https://s3.github.io/s3-csi-driver/helm
helm repo update
```
Install the chart. You will need to specify the location of the SeaweedFS filer URL by either running:
Install the chart. You will need to specify the location of the S3 filer URL by either running:
```bash
helm install --set seaweedfsFiler=<filerHost:port> my-seaweedfs-csi-driver seaweedfs-csi-driver/seaweedfs-csi-driver
helm install --set s3Filer=<filerHost:port> my-s3-csi-driver s3-csi-driver/s3-csi-driver
```
Or by configuring a seaweedfs-overrides.yaml file containing (for example):
Or by configuring a s3-overrides.yaml file containing (for example):
```yaml
# For a SeaweedFS instance running locally under the "seaweed" namespace - adjust for your configuration
seaweedfsFiler: "seaweedfs-filer.seaweed.svc.cluster.local:8888"
# For a S3 instance running locally under the "s3" namespace - adjust for your configuration
s3Filer: "s3-filer.s3.svc.cluster.local:8888"
```
And running:
```bash
helm install my-seaweedfs-csi-driver seaweedfs-csi-driver/seaweedfs-csi-driver -f seaweedfs-overrides.yaml
helm install my-s3-csi-driver s3-csi-driver/s3-csi-driver -f s3-overrides.yaml
```
## Usage
See [Testing](https://github.com/seaweedfs/seaweedfs-csi-driver#testing) on some usage examples.
See [Testing](https://github.com/s3/s3-csi-driver#testing) on some usage examples.
deploy/helm/s3-csi/templates/_helpers.tpl
+10 -10
View File
@@ -1,11 +1,11 @@
{{- define "seaweedfs-csi-driver.name" -}}
{{- define "s3-csi-driver.name" -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Whether a valid security.toml source is configured.
*/}}
{{- define "seaweedfs-csi-driver.security-enabled" -}}
{{- define "s3-csi-driver.security-enabled" -}}
{{- if .Values.security.enabled -}}
{{- if not (or .Values.security.existingSecret .Values.security.existingConfigMap (and .Values.security.create .Values.security.content)) -}}
{{- fail "security.enabled is true but no source configured. Set one of: security.existingSecret, security.existingConfigMap, or security.create with security.content." -}}
@@ -18,8 +18,8 @@ true
Security volume definition.
Produces a volume entry sourcing security.toml from the appropriate Secret or ConfigMap.
*/}}
{{- define "seaweedfs-csi-driver.security-volume" -}}
{{- if include "seaweedfs-csi-driver.security-enabled" . -}}
{{- define "s3-csi-driver.security-volume" -}}
{{- if include "s3-csi-driver.security-enabled" . -}}
- name: security-config
{{- if .Values.security.existingSecret }}
secret:
@@ -29,10 +29,10 @@ Produces a volume entry sourcing security.toml from the appropriate Secret or Co
name: {{ .Values.security.existingConfigMap }}
{{- else if eq .Values.security.type "secret" }}
secret:
secretName: {{ template "seaweedfs-csi-driver.name" . }}-security-config
secretName: {{ template "s3-csi-driver.name" . }}-security-config
{{- else if eq .Values.security.type "configmap" }}
configMap:
name: {{ template "seaweedfs-csi-driver.name" . }}-security-config
name: {{ template "s3-csi-driver.name" . }}-security-config
{{- else }}
{{- fail (printf "security.type must be \"secret\" or \"configmap\", got: %s" .Values.security.type) -}}
{{- end }}
@@ -41,12 +41,12 @@ Produces a volume entry sourcing security.toml from the appropriate Secret or Co
{{/*
Security volumeMount definition.
Mounts security.toml to /etc/seaweedfs/security.toml.
Mounts security.toml to /etc/s3/security.toml.
*/}}
{{- define "seaweedfs-csi-driver.security-volumemount" -}}
{{- if include "seaweedfs-csi-driver.security-enabled" . -}}
{{- define "s3-csi-driver.security-volumemount" -}}
{{- if include "s3-csi-driver.security-enabled" . -}}
- name: security-config
mountPath: /etc/seaweedfs/security.toml
mountPath: /etc/s3/security.toml
subPath: security.toml
readOnly: true
{{- end }}
deploy/helm/s3-csi/templates/configmap-security.yaml
+2 -2
View File
@@ -44,7 +44,7 @@ data:
ca = "/usr/local/share/ca-certificates/ca.crt"
# use this for any place needs a grpc client
# i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload"
# i.e., "s3 backup|benchmark|filer.copy|filer.replicate|mount|s3|upload"
[grpc.client]
cert = "/usr/local/share/ca-certificates/tls.crt"
key = "/usr/local/share/ca-certificates/tls.key"
@@ -52,7 +52,7 @@ data:
# volume server https options
# Note: work in progress!
# this does not work with other clients, e.g., "weed filer|mount" etc, yet.
# this does not work with other clients, e.g., "s3 filer|mount" etc, yet.
#[https.client]
#enabled = false
#[https.volume]
deploy/helm/s3-csi/templates/daemonset-mount.yaml
+3 -3
View File
@@ -40,11 +40,11 @@ spec:
- name: MOUNT_ENDPOINT
value: {{ $mountEndpoint | quote }}
{{- if .Values.tlsSecret }}
- name: WEED_GRPC_CLIENT_KEY
- name: S3_GRPC_CLIENT_KEY
value: /var/run/secrets/app/tls/tls.key
- name: WEED_GRPC_CLIENT_CERT
- name: S3_GRPC_CLIENT_CERT
value: /var/run/secrets/app/tls/tls.crt
- name: WEED_GRPC_CA
- name: S3_GRPC_CA
value: /var/run/secrets/app/tls/ca.crt
{{- end }}
volumeMounts:
deploy/helm/s3-csi/templates/daemonset.yaml
+6 -6
View File
@@ -47,7 +47,7 @@ spec:
imagePullPolicy: {{ .Values.imagePullPolicy }}
args:
- --endpoint=$(CSI_ENDPOINT)
- --filer=$(SEAWEEDFS_FILER)
- --filer=$(S3_FILER)
- --nodeid=$(NODE_ID)
- --driverName=$(DRIVER_NAME)
{{- if $mountEndpoint }}
@@ -76,7 +76,7 @@ spec:
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: SEAWEEDFS_FILER
- name: S3_FILER
value: {{ .Values.s3Filer | quote }}
- name: NODE_ID
valueFrom:
@@ -96,15 +96,15 @@ spec:
fieldPath: metadata.labels['dataCenter']
{{- end }}
{{- if .Values.tlsSecret }}
- name: WEED_GRPC_CLIENT_KEY
- name: S3_GRPC_CLIENT_KEY
value: /var/run/secrets/app/tls/tls.key
- name: WEED_GRPC_CLIENT_CERT
- name: S3_GRPC_CLIENT_CERT
value: /var/run/secrets/app/tls/tls.crt
- name: WEED_GRPC_CA
- name: S3_GRPC_CA
value: /var/run/secrets/app/tls/ca.crt
{{- end }}
{{- with .Values.logVerbosity }}
- name: WEED_V
- name: S3_V
value: {{ . | quote }}
{{- end }}
ports:
deploy/helm/s3-csi/templates/deployment.yaml
+6 -6
View File
@@ -45,7 +45,7 @@ spec:
imagePullPolicy: {{ .Values.imagePullPolicy }}
args :
- --endpoint=$(CSI_ENDPOINT)
- --filer=$(SEAWEEDFS_FILER)
- --filer=$(S3_FILER)
- --driverName=$(DRIVER_NAME)
- --components=controller
@@ -53,21 +53,21 @@ spec:
env:
- name: CSI_ENDPOINT
value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
- name: SEAWEEDFS_FILER
- name: S3_FILER
value: {{ .Values.s3Filer | quote }}
- name: DRIVER_NAME
value: {{ .Values.driverName | quote }}
{{- if .Values.tlsSecret }}
- name: WEED_GRPC_CLIENT_KEY
- name: S3_GRPC_CLIENT_KEY
value: /var/run/secrets/app/tls/tls.key
- name: WEED_GRPC_CLIENT_CERT
- name: S3_GRPC_CLIENT_CERT
value: /var/run/secrets/app/tls/tls.crt
- name: WEED_GRPC_CA
- name: S3_GRPC_CA
value: /var/run/secrets/app/tls/ca.crt
{{- end }}
{{- with .Values.logVerbosity }}
- name: WEED_V
- name: S3_V
value: {{ . | quote }}
{{- end }}
ports:
deploy/helm/s3-csi/values.yaml
+3 -3
View File
@@ -1,5 +1,5 @@
# host and port of your S3 filer
s3Filer: "SEAWEEDFS_FILER:8888"
s3Filer: "S3_FILER:8888"
storageClassName: s3-storage
storageClassVolumeBindingMode: Immediate
isDefaultStorageClass: false
@@ -164,8 +164,8 @@ node:
mountEndpoint: "" # defaults to mountService.endpoint
mountSocketDir: "" # defaults to mountService.socketDir
mountHostPath: "" # defaults to mountService.hostPath
# When s3-csi-node pod on node is recreated, all pods on same node using seaweed-csi PV will stop working.
# For safe update set updateStrategy.type: OnDelete and manually move pods who use seaweed-csi PV, then delete s3-csi-node damonset pod
# When s3-csi-node pod on node is recreated, all pods on same node using s3-csi PV will stop working.
# For safe update set updateStrategy.type: OnDelete and manually move pods who use s3-csi PV, then delete s3-csi-node damonset pod
updateStrategy:
type: RollingUpdate
rollingUpdate:
deploy/kubernetes/seaweedfs-csi-old.yaml → deploy/kubernetes/s3-csi-old.yaml
+21 -21
View File
@@ -302,28 +302,28 @@ spec:
image: chrislusf/s3-csi:latest
args :
- "--endpoint=$(CSI_ENDPOINT)"
- "--filer=$(SEAWEEDFS_FILER)"
- "--filer=$(S3_FILER)"
- "--nodeid=$(NODE_ID)"
- "--cacheCapacityMB=$(SEAWEEDFS_CACHE_CAPACITY_MB)"
- "--cacheDir=$(SEAWEEDFS_CACHE_DIR)"
- "--map.uid=$(SEAWEEDFS_MAP_UID)"
- "--map.gid=$(SEAWEEDFS_MAP_GID)"
- "--cacheCapacityMB=$(S3_CACHE_CAPACITY_MB)"
- "--cacheDir=$(S3_CACHE_DIR)"
- "--map.uid=$(S3_MAP_UID)"
- "--map.gid=$(S3_MAP_GID)"
env:
- name: CSI_ENDPOINT
value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
- name: SEAWEEDFS_FILER
- name: S3_FILER
value: "192.168.2.7:8888" # host and port of your S3 filer
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: SEAWEEDFS_CACHE_CAPACITY_MB
- name: S3_CACHE_CAPACITY_MB
value: "1000"
- name: SEAWEEDFS_CACHE_DIR
- name: S3_CACHE_DIR
value: /tmp
- name: SEAWEEDFS_MAP_UID
- name: S3_MAP_UID
value: ""
- name: SEAWEEDFS_MAP_GID
- name: S3_MAP_GID
value: ""
imagePullPolicy: "Always"
volumeMounts:
@@ -382,28 +382,28 @@ spec:
image: chrislusf/s3-csi:latest
args :
- "--endpoint=$(CSI_ENDPOINT)"
- "--filer=$(SEAWEEDFS_FILER)"
- "--filer=$(S3_FILER)"
- "--nodeid=$(NODE_ID)"
- "--cacheCapacityMB=$(SEAWEEDFS_CACHE_CAPACITY_MB)"
- "--cacheDir=$(SEAWEEDFS_CACHE_DIR)"
- "--map.uid=$(SEAWEEDFS_MAP_UID)"
- "--map.gid=$(SEAWEEDFS_MAP_GID)"
- "--cacheCapacityMB=$(S3_CACHE_CAPACITY_MB)"
- "--cacheDir=$(S3_CACHE_DIR)"
- "--map.uid=$(S3_MAP_UID)"
- "--map.gid=$(S3_MAP_GID)"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: SEAWEEDFS_FILER
value: "192.168.2.7:8888" # host and port of your S3 filer, could also be K8S DNS entry like s3-filer.seaweed-namespace.svc:8888
- name: S3_FILER
value: "192.168.2.7:8888" # host and port of your S3 filer, could also be K8S DNS entry like s3-filer.s3-namespace.svc:8888
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: SEAWEEDFS_CACHE_CAPACITY_MB
- name: S3_CACHE_CAPACITY_MB
value: "1000"
- name: SEAWEEDFS_CACHE_DIR
- name: S3_CACHE_DIR
value: /tmp
- name: SEAWEEDFS_MAP_UID
- name: S3_MAP_UID
value: ""
- name: SEAWEEDFS_MAP_GID
- name: S3_MAP_GID
value: ""
imagePullPolicy: "IfNotPresent"
volumeMounts:
deploy/kubernetes/seaweedfs-csi.yaml → deploy/kubernetes/s3-csi.yaml
+6 -6
View File
@@ -293,7 +293,7 @@ spec:
imagePullPolicy: IfNotPresent
args:
- --endpoint=$(CSI_ENDPOINT)
- --filer=$(SEAWEEDFS_FILER)
- --filer=$(S3_FILER)
- --nodeid=$(NODE_ID)
- --driverName=$(DRIVER_NAME)
- --mountEndpoint=$(MOUNT_ENDPOINT)
@@ -302,8 +302,8 @@ spec:
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: SEAWEEDFS_FILER
value: "SEAWEEDFS_FILER:8888"
- name: S3_FILER
value: "S3_FILER:8888"
- name: NODE_ID
valueFrom:
fieldRef:
@@ -454,7 +454,7 @@ spec:
imagePullPolicy: IfNotPresent
args :
- --endpoint=$(CSI_ENDPOINT)
- --filer=$(SEAWEEDFS_FILER)
- --filer=$(S3_FILER)
- --driverName=$(DRIVER_NAME)
- --components=controller
@@ -462,8 +462,8 @@ spec:
env:
- name: CSI_ENDPOINT
value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
- name: SEAWEEDFS_FILER
value: "SEAWEEDFS_FILER:8888"
- name: S3_FILER
value: "S3_FILER:8888"
- name: DRIVER_NAME
value: "s3-csi"
deploy/kubernetes/sample-seaweedfs-pvc.yaml → deploy/kubernetes/sample-s3-pvc.yaml
View File
deploy/nomad/README.md
+17 -17
View File
@@ -1,9 +1,9 @@
# Example of using seaweedfs with HashiCorp Nomad
# Example of using s3 with HashiCorp Nomad
## Running seaweedfs cluster
## Running s3 cluster
You can skip this part if you have already running seaweedfs.
You can skip this part if you have already running s3.
Assumptions:
- Running Nomad cluster
@@ -14,24 +14,24 @@ Assumptions:
```shell
export NOMAD_ADDR=http://nomad.service.consul:4646
nomad run seaweedfs.hcl
nomad run s3.hcl
```
Seaweedfs master will be available on http://seaweedfs-master.service.consul:9333/
S3 master will be available on http://s3-master.service.consul:9333/
Seaweedfs filer will be available on http://seaweedfs-filer.service.consul:8888/
S3 filer will be available on http://s3-filer.service.consul:8888/
## Running CSI
The CSI driver is split into two components that register under the same
`csi_plugin` id (`seaweedfs`):
`csi_plugin` id (`s3`):
- **controller** (`seaweedfs-csi-controller.hcl`) — a single `service` job that
- **controller** (`s3-csi-controller.hcl`) — a single `service` job that
implements the volume lifecycle RPCs. Nomad calls it for `nomad volume create`
/ `nomad volume delete`. It only talks to the filer.
- **node** (`seaweedfs-csi.hcl`) — a `system` job that runs on every worker and
stages/publishes volumes into allocations, using the `seaweedfs-mount`
- **node** (`s3-csi.hcl`) — a `system` job that runs on every worker and
stages/publishes volumes into allocations, using the `s3-mount`
sidecar over a shared unix socket.
You need **both** running. With only the node plugin, `nomad volume create`
@@ -41,20 +41,20 @@ fails with `plugin has no controller`.
export NOMAD_ADDR=http://nomad.service.consul:4646
# Start CSI controller (one instance) and node (one per worker)
nomad run seaweedfs-csi-controller.hcl
nomad run seaweedfs-csi.hcl
nomad run s3-csi-controller.hcl
nomad run s3-csi.hcl
# Wait until the plugin reports a healthy controller and the expected node count
nomad plugin status seaweedfs
nomad plugin status s3
# Create volume
nomad volume create example-seaweedfs-volume.hcl
nomad volume create example-s3-volume.hcl
# Start sample app
nomad run example-seaweedfs-app.hcl
nomad run example-s3-app.hcl
```
> If you only run the node plugin (no controller), you cannot `nomad volume
> create`. Instead, pre-create the bucket/directory in SeaweedFS yourself and
> `nomad volume register example-seaweedfs-volume.hcl` to register the existing
> create`. Instead, pre-create the bucket/directory in S3 yourself and
> `nomad volume register example-s3-volume.hcl` to register the existing
> volume.
deploy/nomad/example-seaweedfs-app.hcl → deploy/nomad/example-s3-app.hcl
+4 -4
View File
@@ -1,10 +1,10 @@
job "example-seaweedfs-app" {
job "example-s3-app" {
datacenters = ["dc1"]
group "apps" {
volume "example-seaweedfs-volume" {
volume "example-s3-volume" {
type = "csi"
source = "example-seaweedfs-volume"
source = "example-s3-volume"
access_mode = "multi-node-multi-writer"
attachment_mode = "file-system"
}
@@ -24,7 +24,7 @@ job "example-seaweedfs-app" {
}
volume_mount {
volume = "example-seaweedfs-volume"
volume = "example-s3-volume"
destination = "/mnt/pv"
}
}
deploy/nomad/example-seaweedfs-volume.hcl → deploy/nomad/example-s3-volume.hcl
+4 -4
View File
@@ -1,13 +1,13 @@
# id - Nomad internal ID. It is not sent to the CSI plugin but is used by Nomad for `per_alloc`
# volume configurations, etc.
id = "example-seaweedfs-volume"
id = "example-s3-volume"
# name - the name sent to the CSI plugin as an idempotency key and suggested volume ID. The CSI
# spec requires the calling Container Orchestrator to respect the actual volumeId returned by the
# CSI plugin. Nomad does this, storing it as the volume's ExternalID and using it in subsequent
# calls to the controller and node.
name = "example-seaweedfs-volume"
name = "example-s3-volume"
type = "csi"
plugin_id = "seaweedfs"
plugin_id = "s3"
capacity_min = "256GiB"
capacity_max = "512GiB"
@@ -25,7 +25,7 @@ mount_options {
}
parameters {
# Available options: https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/mounter_seaweedfs.go
# Available options: https://github.com/s3/s3-csi-driver/blob/master/pkg/driver/mounter_s3.go
# By default, collection is the `volume ID` returned from the Create Volume gRPC call. Nomad calls this the
# External ID of the volume. "example" here overrides that.
collection = "example"
deploy/nomad/seaweedfs-csi-controller.hcl → deploy/nomad/s3-csi-controller.hcl
+7 -7
View File
@@ -6,13 +6,13 @@
#
# The controller only talks to the filer (it creates/removes the bucket or
# directory backing the volume), so unlike the node plugin it does NOT need the
# seaweedfs-mount sidecar, host volumes, bidirectional mount propagation, or
# s3-mount sidecar, host volumes, bidirectional mount propagation, or
# privileged mode. A single instance is enough; run it as a `service` job.
#
# The csi_plugin.id ("seaweedfs") MUST match the id used by the node job
# (seaweedfs-csi.hcl) so Nomad treats them as one logical plugin with both a
# The csi_plugin.id ("s3") MUST match the id used by the node job
# (s3-csi.hcl) so Nomad treats them as one logical plugin with both a
# controller and nodes.
job "seaweedfs-csi-controller" {
job "s3-csi-controller" {
datacenters = ["dc1"]
type = "service"
priority = 90
@@ -24,18 +24,18 @@ job "seaweedfs-csi-controller" {
driver = "docker"
config {
image = "chrislusf/seaweedfs-csi-driver:v1.3.9"
image = "chrislusf/s3-csi-driver:v1.3.9"
force_pull = true
network_mode = "host"
args = [
"--endpoint=unix:///csi-sock/csi.sock",
"--filer=seaweedfs-filer.service.consul:8888",
"--filer=s3-filer.service.consul:8888",
"--components=controller",
]
}
csi_plugin {
id = "seaweedfs"
id = "s3"
type = "controller"
mount_dir = "/csi-sock"
}
deploy/nomad/seaweedfs-csi.hcl → deploy/nomad/s3-csi.hcl
+18 -18
View File
@@ -1,4 +1,4 @@
job "seaweedfs-csi-node" {
job "s3-csi-node" {
datacenters = ["dc1"]
type = "system"
priority = 90
@@ -15,17 +15,17 @@ job "seaweedfs-csi-node" {
group "nodes" {
# directory to share the seaweedfs-mount unix socket between tasks
volume "seaweedfs_mount_socket" {
# directory to share the s3-mount unix socket between tasks
volume "s3_mount_socket" {
type = "host"
source = "seaweedfs_mount_socket"
source = "s3_mount_socket"
read_only = false
}
# host directory where Nomad CSI stage/publish paths live
volume "nomad_csi_seaweedfs" {
volume "nomad_csi_s3" {
type = "host"
source = "nomad_csi_seaweedfs"
source = "nomad_csi_s3"
read_only = false
}
@@ -39,25 +39,25 @@ job "seaweedfs-csi-node" {
driver = "docker"
volume_mount {
volume = "seaweedfs_mount_socket"
destination = "/var/lib/seaweedfs-mount"
volume = "s3_mount_socket"
destination = "/var/lib/s3-mount"
read_only = false
}
# required for proper propagation of staged mounts into the host filesystem
volume_mount {
volume = "nomad_csi_seaweedfs"
volume = "nomad_csi_s3"
destination = "/csi-data"
read_only = false
propagation_mode = "bidirectional"
}
config {
image = "chrislusf/seaweedfs-mount:dev"
image = "chrislusf/s3-mount:dev"
network_mode = "host"
privileged = true
args = [
"--endpoint=unix:///var/lib/seaweedfs-mount/seaweedfs-mount.sock",
"--endpoint=unix:///var/lib/s3-mount/s3-mount.sock",
]
}
@@ -72,34 +72,34 @@ job "seaweedfs-csi-node" {
driver = "docker"
volume_mount {
volume = "seaweedfs_mount_socket"
destination = "/var/lib/seaweedfs-mount"
volume = "s3_mount_socket"
destination = "/var/lib/s3-mount"
read_only = false
}
config {
image = "chrislusf/seaweedfs-csi-driver:v1.3.9"
image = "chrislusf/s3-csi-driver:v1.3.9"
force_pull = true
network_mode = "host"
privileged = true
args = [
"--endpoint=unix:///csi-sock/csi.sock",
"--filer=seaweedfs-filer.service.consul:8888",
"--filer=s3-filer.service.consul:8888",
"--dataCenter=${node.datacenter}",
"--nodeid=${node.unique.name}",
"--cacheCapacityMB=1024",
"--cacheDir=${NOMAD_ALLOC_DIR}/cache_dir",
"--mountEndpoint=unix:///var/lib/seaweedfs-mount/seaweedfs-mount.sock",
"--mountEndpoint=unix:///var/lib/s3-mount/s3-mount.sock",
"--dataLocality=none",
"--components=node",
]
}
csi_plugin {
id = "seaweedfs"
id = "s3"
type = "node"
mount_dir = "/csi-sock"
stage_publish_base_dir = "/csi-data/node/seaweedfs"
stage_publish_base_dir = "/csi-data/node/s3"
}
resources {
deploy/nomad/seaweedfs.hcl → deploy/nomad/s3.hcl
+27 -27
View File
@@ -1,11 +1,11 @@
# WWW naster: http://seaweedfs-master.service.consul:9333/
# WWW filer: http://seaweedfs-filer.service.consul:8888/
# WWW naster: http://s3-master.service.consul:9333/
# WWW filer: http://s3-filer.service.consul:8888/
job "seaweedfs" {
job "s3" {
datacenters = ["dc1"]
type = "service"
group "seaweedfs-master" {
group "s3-master" {
count = 3
constraint {
@@ -49,15 +49,15 @@ job "seaweedfs" {
}
}
task "seaweedfs-master" {
task "s3-master" {
driver = "docker"
env {
WEED_MASTER_VOLUME_GROWTH_COPY_1 = "1"
WEED_MASTER_VOLUME_GROWTH_COPY_2 = "2"
WEED_MASTER_VOLUME_GROWTH_COPY_OTHER = "1"
S3_MASTER_VOLUME_GROWTH_COPY_1 = "1"
S3_MASTER_VOLUME_GROWTH_COPY_2 = "2"
S3_MASTER_VOLUME_GROWTH_COPY_OTHER = "1"
}
config {
image = "chrislusf/seaweedfs:latest"
image = "chrislusf/s3:latest"
force_pull = "true"
network_mode = "host"
args = [
@@ -78,7 +78,7 @@ job "seaweedfs" {
service {
tags = ["${node.unique.name}"]
name = "seaweedfs-master"
name = "s3-master"
port = "http"
check {
type = "tcp"
@@ -93,7 +93,7 @@ job "seaweedfs" {
group "seaweedfs-volume" {
group "s3-volume" {
count = 3
constraint {
@@ -133,12 +133,12 @@ job "seaweedfs" {
}
}
task "seaweedfs-volume" {
task "s3-volume" {
driver = "docker"
user = "1000:1000"
config {
image = "chrislusf/seaweedfs:latest"
image = "chrislusf/s3:latest"
force_pull = "true"
network_mode = "host"
args = [
@@ -146,7 +146,7 @@ job "seaweedfs" {
"-dataCenter=${NOMAD_DC}",
# "-rack=${meta.rack}",
"-rack=${node.unique.name}",
"-mserver=seaweedfs-master.service.consul:9333",
"-mserver=s3-master.service.consul:9333",
"-port=${NOMAD_PORT_http}",
"-ip=${NOMAD_IP_http}",
"-publicUrl=${NOMAD_ADDR_http}",
@@ -157,7 +157,7 @@ job "seaweedfs" {
mounts = [
{
type = "bind"
source = "/data/seaweedfs-volume-data" # there should be directory in host VM
source = "/data/s3-volume-data" # there should be directory in host VM
target = "/data"
readonly = false
bind_options = {
@@ -175,7 +175,7 @@ job "seaweedfs" {
service {
tags = ["${node.unique.name}"]
name = "seaweedfs-volume"
name = "s3-volume"
port = "http"
check {
type = "tcp"
@@ -188,7 +188,7 @@ job "seaweedfs" {
}
group "seaweedfs-filer" {
group "s3-filer" {
count = 1
constraint {
@@ -219,12 +219,12 @@ job "seaweedfs" {
}
}
task "seaweedfs-filer" {
task "s3-filer" {
driver = "docker"
user = "1000:1000"
config {
image = "chrislusf/seaweedfs:latest"
image = "chrislusf/s3:latest"
force_pull = "true"
network_mode = "host"
args = [
@@ -233,7 +233,7 @@ job "seaweedfs" {
# "-rack=${meta.rack}",
"-rack=${node.unique.name}",
"-defaultReplicaPlacement=000",
"-master=seaweedfs-master.service.consul:9333",
"-master=s3-master.service.consul:9333",
"-s3",
"-ip=${NOMAD_IP_http}",
"-port=${NOMAD_PORT_http}",
@@ -243,7 +243,7 @@ job "seaweedfs" {
{
type = "bind"
source = "local/filer.toml"
target = "/etc/seaweedfs/filer.toml"
target = "/etc/s3/filer.toml"
}
]
@@ -266,9 +266,9 @@ createTable = """
"""
hostname = "172.21.100.54"
port = 5432
username = "seaweedfs"
username = "s3"
password = "pass1234567"
database = "seaweedfs"
database = "s3"
schema = ""
sslmode = "disable"
connection_max_idle = 100
@@ -278,8 +278,8 @@ enableUpsert = true
upsertQuery = """INSERT INTO "%[1]s" (dirhash,name,directory,meta) VALUES($1,$2,$3,$4) ON CONFLICT (dirhash,name) DO UPDATE SET meta = EXCLUDED.meta WHERE "%[1]s".meta != EXCLUDED.meta"""
# ssh ubuntu@172.21.100.54
# sudo -u postgres psql -c "CREATE ROLE seaweedfs WITH PASSWORD 'pass1234567';"
# sudo -u postgres psql -c "CREATE DATABASE seaweedfs OWNER seaweedfs;"
# sudo -u postgres psql -c "CREATE ROLE s3 WITH PASSWORD 'pass1234567';"
# sudo -u postgres psql -c "CREATE DATABASE s3 OWNER s3;"
EOH
}
@@ -290,7 +290,7 @@ EOH
service {
tags = ["${node.unique.name}"]
name = "seaweedfs-filer"
name = "s3-filer"
port = "http"
check {
type = "tcp"
@@ -302,7 +302,7 @@ EOH
service {
tags = ["${node.unique.name}"]
name = "seaweedfs-s3"
name = "s3-s3"
port = "s3"
check {
type = "tcp"
pkg/filer/filer.go
+1 -1
View File
@@ -8,7 +8,7 @@
// There is no RegisterHanzoFilerServer anywhere in that tree.
//
// So a gRPC client cannot talk to it at all — the failure mode is
// `Unimplemented: unknown service filer_pb.SeaweedFiler`, which reads like a
// `Unimplemented: unknown service filer_pb.S3Filer`, which reads like a
// service-name mismatch but is really a gRPC client hitting a non-gRPC port. No
// amount of renaming or re-addressing fixes that; the transport has to change.
//
pkg/mountmanager/manager.go
+31 -31
View File
@@ -18,9 +18,9 @@ import (
var kubeMounter = mount.New("")
// Manager owns weed mount processes and exposes helpers to start and stop them.
// Manager owns s3 mount processes and exposes helpers to start and stop them.
type Manager struct {
weedBinary string
s3Binary string
mu sync.Mutex
mounts map[string]*mountEntry
@@ -29,23 +29,23 @@ type Manager struct {
// Config configures a Manager instance.
type Config struct {
WeedBinary string
S3Binary string
}
// NewManager returns a Manager ready to accept mount requests.
func NewManager(cfg Config) *Manager {
binary := cfg.WeedBinary
binary := cfg.S3Binary
if binary == "" {
binary = DefaultWeedBinary
binary = DefaultS3Binary
}
return &Manager{
weedBinary: binary,
s3Binary: binary,
mounts: make(map[string]*mountEntry),
locks: newKeyMutex(),
}
}
// Mount starts a weed mount process using the provided request.
// Mount starts a s3 mount process using the provided request.
func (m *Manager) Mount(req *MountRequest) (*MountResponse, error) {
if req == nil {
return nil, errors.New("mount request is nil")
@@ -59,14 +59,14 @@ func (m *Manager) Mount(req *MountRequest) (*MountResponse, error) {
defer lock.Unlock()
if entry := m.getMount(req.VolumeID); entry != nil {
// If the previous weed mount process has died, the entry is
// If the previous s3 mount process has died, the entry is
// stale and the FUSE mount is dead. Tear down the stale entry
// so we can start a fresh process; otherwise we would falsely
// report "already mounted" and a CSI-driver recovery would
// silently bind-mount onto a dead path. See s3/s3-csi#261.
select {
case <-entry.process.exited:
glog.Warningf("volume %s previous weed mount process has exited; replacing stale entry with a fresh mount", req.VolumeID)
glog.Warningf("volume %s previous s3 mount process has exited; replacing stale entry with a fresh mount", req.VolumeID)
// Wait for wait()'s post-exit cleanup (FUSE unmount) to finish
// so ensureTargetClean below sees a quiescent path.
<-entry.process.done
@@ -89,18 +89,18 @@ func (m *Manager) Mount(req *MountRequest) (*MountResponse, error) {
m.mounts[req.VolumeID] = entry
m.mu.Unlock()
// Proactively clear the entry once the weed mount process exits,
// Proactively clear the entry once the s3 mount process exits,
// even if Unmount is never called. Without this, a process that
// crashes on its own (e.g. backend errors) leaves a stale entry
// that fools later Mount calls into reporting success without
// starting a new process.
go m.watchProcessExit(req.VolumeID, entry)
glog.Infof("started weed mount process for volume %s at %s", req.VolumeID, req.TargetPath)
glog.Infof("started s3 mount process for volume %s at %s", req.VolumeID, req.TargetPath)
return &MountResponse{LocalSocket: entry.localSocket}, nil
}
// watchProcessExit removes the entry for volumeID once its weed mount
// watchProcessExit removes the entry for volumeID once its s3 mount
// process has finished exiting. It is safe to run concurrently with
// Unmount: if Unmount has already removed the entry (or replaced it
// with a fresh mount), the identity check ensures we leave the new
@@ -113,11 +113,11 @@ func (m *Manager) watchProcessExit(volumeID string, entry *mountEntry) {
delete(m.mounts, volumeID)
// See removeMount: do not delete the per-volume lock — a
// concurrent Mount/Unmount may still be holding it.
glog.Infof("removed mount entry for volume %s after weed mount process exited (target: %s)", volumeID, entry.targetPath)
glog.Infof("removed mount entry for volume %s after s3 mount process exited (target: %s)", volumeID, entry.targetPath)
}
}
// Unmount terminates the weed mount process associated with the provided request.
// Unmount terminates the s3 mount process associated with the provided request.
func (m *Manager) Unmount(req *UnmountRequest) (*UnmountResponse, error) {
if req == nil {
return nil, errors.New("unmount request is nil")
@@ -137,7 +137,7 @@ func (m *Manager) Unmount(req *UnmountRequest) (*UnmountResponse, error) {
return &UnmountResponse{}, nil
}
// Note: We don't explicitly unmount here because weedMountProcess.wait()
// Note: We don't explicitly unmount here because s3MountProcess.wait()
// handles the unmount when the process terminates (either gracefully or forcefully).
// This centralizes unmount logic and avoids potential race conditions.
if err := entry.process.stop(); err != nil {
@@ -152,7 +152,7 @@ func (m *Manager) Unmount(req *UnmountRequest) (*UnmountResponse, error) {
// Only remove from state after all cleanup operations succeeded
m.removeMount(req.VolumeID)
glog.Infof("stopped weed mount process for volume %s at %s", req.VolumeID, entry.targetPath)
glog.Infof("stopped s3 mount process for volume %s at %s", req.VolumeID, entry.targetPath)
return &UnmountResponse{}, nil
}
@@ -201,7 +201,7 @@ func (m *Manager) startMount(req *MountRequest) (*mountEntry, error) {
return nil, errors.New("mountArgs is required")
}
process, err := startWeedMountProcess(m.weedBinary, args, targetPath, req.VolumeID)
process, err := startS3MountProcess(m.s3Binary, args, targetPath, req.VolumeID)
if err != nil {
return nil, err
}
@@ -264,14 +264,14 @@ type mountEntry struct {
targetPath string
cacheDir string
localSocket string
process *weedMountProcess
process *s3MountProcess
}
type weedMountProcess struct {
type s3MountProcess struct {
cmd *exec.Cmd
target string
// exited is closed as soon as cmd.Wait() returns, so callers can
// detect that the weed mount process is gone without waiting for
// detect that the s3 mount process is gone without waiting for
// the post-exit FUSE unmount step.
exited chan struct{}
// done is closed after wait() finishes its full cleanup (including
@@ -279,7 +279,7 @@ type weedMountProcess struct {
done chan struct{}
}
func startWeedMountProcess(command string, args []string, target string, volumeID string) (*weedMountProcess, error) {
func startS3MountProcess(command string, args []string, target string, volumeID string) (*s3MountProcess, error) {
cmd := exec.Command(command, args...)
// Capture stdout/stderr and log with volume ID prefix for better debugging
@@ -292,17 +292,17 @@ func startWeedMountProcess(command string, args []string, target string, volumeI
return nil, fmt.Errorf("creating stderr pipe: %w", err)
}
glog.V(0).Infof("[%s] Starting weed mount: %s %s", volumeID, command, strings.Join(args, " "))
glog.V(0).Infof("[%s] Starting s3 mount: %s %s", volumeID, command, strings.Join(args, " "))
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("starting weed mount: %w", err)
return nil, fmt.Errorf("starting s3 mount: %w", err)
}
// Forward stdout/stderr with volume ID prefix for better debugging
go forwardLogs(stdoutPipe, volumeID, "stdout")
go forwardLogs(stderrPipe, volumeID, "stderr")
process := &weedMountProcess{
process := &s3MountProcess{
cmd: cmd,
target: target,
exited: make(chan struct{}),
@@ -321,11 +321,11 @@ func startWeedMountProcess(command string, args []string, target string, volumeI
return process, nil
}
func (p *weedMountProcess) wait() {
func (p *s3MountProcess) wait() {
if err := p.cmd.Wait(); err != nil {
glog.Errorf("weed mount exit (pid: %d, target: %s): %v", p.cmd.Process.Pid, p.target, err)
glog.Errorf("s3 mount exit (pid: %d, target: %s): %v", p.cmd.Process.Pid, p.target, err)
} else {
glog.Infof("weed mount exit (pid: %d, target: %s)", p.cmd.Process.Pid, p.target)
glog.Infof("s3 mount exit (pid: %d, target: %s)", p.cmd.Process.Pid, p.target)
}
// Signal exit immediately so Manager.Mount can detect a dead
@@ -339,9 +339,9 @@ func (p *weedMountProcess) wait() {
close(p.done)
}
func (p *weedMountProcess) stop() error {
func (p *s3MountProcess) stop() error {
if err := p.cmd.Process.Signal(syscall.SIGTERM); err != nil {
glog.Warningf("sending SIGTERM to weed mount failed: %v", err)
glog.Warningf("sending SIGTERM to s3 mount failed: %v", err)
}
select {
@@ -351,14 +351,14 @@ func (p *weedMountProcess) stop() error {
}
if err := p.cmd.Process.Kill(); err != nil {
glog.Warningf("killing weed mount failed: %v", err)
glog.Warningf("killing s3 mount failed: %v", err)
}
select {
case <-p.done:
return nil
case <-time.After(1 * time.Second):
return errors.New("timed out waiting for weed mount to stop")
return errors.New("timed out waiting for s3 mount to stop")
}
}
pkg/mountmanager/manager_test.go
+6 -6
View File
@@ -6,7 +6,7 @@ import (
)
// TestWatchProcessExitRemovesStaleEntry verifies that the manager's
// background watcher clears a mount entry from its map when the weed
// background watcher clears a mount entry from its map when the s3
// mount process exits on its own. Without this, a future Mount call for
// the same volume would falsely receive an "already mounted" no-op
// because the manager's state never noticed the dead process.
@@ -14,7 +14,7 @@ import (
func TestWatchProcessExitRemovesStaleEntry(t *testing.T) {
m := NewManager(Config{})
process := &weedMountProcess{
process := &s3MountProcess{
target: "/tmp/test-target",
exited: make(chan struct{}),
done: make(chan struct{}),
@@ -35,7 +35,7 @@ func TestWatchProcessExitRemovesStaleEntry(t *testing.T) {
t.Fatalf("expected entry tracked before process exit, got %v", got)
}
// Simulate the weed mount process dying.
// Simulate the s3 mount process dying.
close(process.exited)
close(process.done)
@@ -47,7 +47,7 @@ func TestWatchProcessExitRemovesStaleEntry(t *testing.T) {
}
time.Sleep(10 * time.Millisecond)
}
t.Fatal("watcher did not remove stale entry after weed mount process exited")
t.Fatal("watcher did not remove stale entry after s3 mount process exited")
}
// TestWatchProcessExitLeavesReplacedEntry verifies that if a fresh
@@ -58,7 +58,7 @@ func TestWatchProcessExitRemovesStaleEntry(t *testing.T) {
func TestWatchProcessExitLeavesReplacedEntry(t *testing.T) {
m := NewManager(Config{})
oldProcess := &weedMountProcess{
oldProcess := &s3MountProcess{
target: "/tmp/test-target",
exited: make(chan struct{}),
done: make(chan struct{}),
@@ -68,7 +68,7 @@ func TestWatchProcessExitLeavesReplacedEntry(t *testing.T) {
process: oldProcess,
}
newProcess := &weedMountProcess{
newProcess := &s3MountProcess{
target: "/tmp/test-target",
exited: make(chan struct{}),
done: make(chan struct{}),
pkg/mountmanager/socket.go
+1 -1
View File
@@ -10,7 +10,7 @@ import (
// DefaultSocketDir is the default directory for volume sockets.
const DefaultSocketDir = "/var/lib/s3-mount"
// LocalSocketPath returns the unix socket path used to communicate with the weed mount process.
// LocalSocketPath returns the unix socket path used to communicate with the s3 mount process.
// The baseDir parameter should be the directory where sockets are stored (e.g., derived from mountEndpoint).
// Uses SHA256 hash (first 16 hex chars = 64 bits) to minimize collision risk.
func LocalSocketPath(baseDir, volumeID string) string {
pkg/mountmanager/types.go
+4 -4
View File
@@ -1,6 +1,6 @@
package mountmanager
// MountRequest contains all information needed to start a weed mount process.
// MountRequest contains all information needed to start a s3 mount process.
type MountRequest struct {
VolumeID string `json:"volumeId"`
TargetPath string `json:"targetPath"`
@@ -14,7 +14,7 @@ type MountResponse struct {
LocalSocket string `json:"localSocket"`
}
// UnmountRequest contains the information needed to stop a weed mount process.
// UnmountRequest contains the information needed to stop a s3 mount process.
type UnmountRequest struct {
VolumeID string `json:"volumeId"`
}
@@ -28,6 +28,6 @@ type ErrorResponse struct {
}
const (
// DefaultWeedBinary is the default executable name used to spawn weed mount processes.
DefaultWeedBinary = "weed"
// DefaultS3Binary is the default executable name used to spawn s3 mount processes.
DefaultS3Binary = "s3"
)
test/test.sh
+4 -4
View File
@@ -6,13 +6,13 @@ git clone https://github.com/kubernetes-csi/csi-test.git -b v5.0.0
cd csi-test
make install
## Start Local weed server with filer
weed server -dir=/tmp/seaweedfs/data -s3 -volume.max=100 -volume.port=8090 -master.volumeSizeLimitMB=256
## Start Local s3 server with filer
s3 server -dir=/tmp/s3/data -s3 -volume.max=100 -volume.port=8090 -master.volumeSizeLimitMB=256
## Run CSI Driver
cd ../seaweedfs-csi-driver
cd ../s3-csi-driver
make build
./_output/seaweedfs-csi-driver -endpoint="$endpoint" -alsologtostderr -v=5 -filer=localhost:8888 -nodeid=test
./_output/s3-csi-driver -endpoint="$endpoint" -alsologtostderr -v=5 -filer=localhost:8888 -nodeid=test
# Run CSI Sanity Tests
../csi-test/cmd/csi-sanity/csi-sanity\