mirror of
https://github.com/hanzoai/flow.git
synced 2026-10-17 01:29:18 +00:00
static analysis runs again; the GitHub-only scanners do not
The forge reads the first WORKFLOW_DIRS entry that exists. .hanzo/workflows has
existed here since the image lane landed, so all 43 workflow files under
.github/workflows ran nowhere. Nothing went red, because the checks that would
go red were the ones not running: the forge listed 2 workflows for this repo
against 45 files on disk. This repo has had no test gate of any kind since
actions-runner-controller was decommissioned.
security.yml replaces the two dead scanners rather than restoring them:
- codeql.yml ended in github/codeql-action/analyze under `security-events:
write`, which names an upload to GitHub's code-scanning store. There is no
such store here, and a scanner whose findings have nowhere to go is not a
scanner. Semgrep covers the same two languages CodeQL was configured for.
- mend.yml ran the Mend unified agent against WS_APIKEY / WS_USERKEY. That is
a commercial subscription this org does not hold. Dependency and licence
scanning is GONE, not replaced — semgrep reads code, not a dependency graph,
and claiming otherwise would be the more expensive mistake.
Measured before it was written: 98 rules over 3558 git-tracked files in 20s.
The other 42 files are upstream's CI and cannot run here: PR labelers, autofix
bots that push to pull requests, IBM Globalization Pipeline, Codeflash, macOS
and Windows release trains, PyPI publishes for packages this fork does not own,
and a `ci_success` gate that passes only when upstream's flow-nightly was
published to PyPI that same day. They are deleted rather than moved — a dead
file cannot accumulate where there is no directory.
Still ungated and needing their own change: mypy, which lint-py.yml ran and
which nothing ever called even upstream, and `ruff check`, which the tree fails
1204 times against its own declared config.
This commit is contained in:
@@ -1,65 +0,0 @@
|
||||
name: Manage Review Labels
|
||||
|
||||
on:
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
|
||||
jobs:
|
||||
label-on-review:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
pull-requests: write
|
||||
issues: write
|
||||
steps:
|
||||
- name: Manage LGTM Review Label
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const LGTM_LABEL = 'lgtm';
|
||||
|
||||
// Extract review details
|
||||
const { state: reviewState } = context.payload.review;
|
||||
const pullRequestNumber = context.payload.pull_request.number;
|
||||
const repoDetails = {
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: pullRequestNumber
|
||||
};
|
||||
|
||||
// Log review information
|
||||
console.log(`Processing review for PR #${pullRequestNumber}`);
|
||||
console.log(`Review state: ${reviewState}`);
|
||||
|
||||
// Helper function to check for LGTM label
|
||||
async function hasLgtmLabel() {
|
||||
const { data: labels } = await github.rest.issues.listLabelsOnIssue(repoDetails);
|
||||
return labels.some(label => label.name === LGTM_LABEL);
|
||||
}
|
||||
|
||||
if (reviewState === 'approved') {
|
||||
const lgtmExists = await hasLgtmLabel();
|
||||
|
||||
if (!lgtmExists) {
|
||||
console.log(`Adding ${LGTM_LABEL} label to PR #${pullRequestNumber}`);
|
||||
await github.rest.issues.addLabels({
|
||||
...repoDetails,
|
||||
labels: [LGTM_LABEL]
|
||||
});
|
||||
console.log('Label added successfully');
|
||||
} else {
|
||||
console.log(`${LGTM_LABEL} label already exists`);
|
||||
}
|
||||
} else if (reviewState === 'changes_requested') {
|
||||
const lgtmExists = await hasLgtmLabel();
|
||||
|
||||
if (lgtmExists) {
|
||||
console.log(`Removing ${LGTM_LABEL} label from PR #${pullRequestNumber}`);
|
||||
await github.rest.issues.removeLabel({
|
||||
...repoDetails,
|
||||
name: LGTM_LABEL
|
||||
});
|
||||
console.log('Label removed successfully');
|
||||
} else {
|
||||
console.log(`No ${LGTM_LABEL} label to remove`);
|
||||
}
|
||||
}
|
||||
@@ -1,13 +0,0 @@
|
||||
name: Auto-update
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
|
||||
jobs:
|
||||
Auto:
|
||||
name: Auto-update
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: tibdex/auto-update@v2
|
||||
@@ -1,27 +0,0 @@
|
||||
name: CI Scripts Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "scripts/ci/**"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
name: Test CI release scripts
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: pip install pytest requests packaging
|
||||
|
||||
- name: Run CI script tests
|
||||
run: python -m pytest scripts/ci/ -v
|
||||
@@ -1,564 +0,0 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
ref:
|
||||
description: "(Optional) Ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
python-versions:
|
||||
description: "Python Versions"
|
||||
required: false
|
||||
type: string
|
||||
default: "['3.10', '3.14']"
|
||||
frontend-tests-folder:
|
||||
description: "Frontend Tests Folder"
|
||||
required: false
|
||||
type: string
|
||||
default: "tests/core"
|
||||
release:
|
||||
description: "Release"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
run-all-tests:
|
||||
description: "Run all tests regardless of file changes (skips path filtering)"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
runs-on:
|
||||
description: "Runner to use for the tests"
|
||||
required: false
|
||||
type: string
|
||||
default: "ubuntu-latest"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
ref:
|
||||
description: "(Optional) Ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
openai_api_key:
|
||||
description: "OpenAI API Key"
|
||||
required: false
|
||||
type: string
|
||||
store_api_key:
|
||||
description: "Store API Key"
|
||||
required: false
|
||||
type: string
|
||||
python-versions:
|
||||
description: "Python Versions"
|
||||
required: false
|
||||
type: string
|
||||
default: "['3.10', '3.14']"
|
||||
runs-on:
|
||||
description: "Runner to use for the tests"
|
||||
required: false
|
||||
type: choice
|
||||
options:
|
||||
- ubuntu-latest
|
||||
- self-hosted
|
||||
- '["self-hosted", "linux", "ARM64", "flow-ai-arm64-40gb-ephemeral"]'
|
||||
default: ubuntu-latest
|
||||
pull_request:
|
||||
types: [opened, synchronize, labeled]
|
||||
merge_group:
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
STORE_API_KEY: ${{ secrets.STORE_API_KEY }}
|
||||
TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }}
|
||||
|
||||
jobs:
|
||||
echo-inputs:
|
||||
name: Echo Inputs
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Echo inputs
|
||||
run: |
|
||||
echo "Inputs:"
|
||||
echo " ref: ${{ inputs.ref }}"
|
||||
echo " python-versions: ${{ inputs.python-versions }}"
|
||||
echo " frontend-tests-folder: ${{ inputs.frontend-tests-folder }}"
|
||||
echo " release: ${{ inputs.release }}"
|
||||
echo " run-all-tests: ${{ inputs.run-all-tests }}"
|
||||
echo " runs-on: ${{ inputs.runs-on }}"
|
||||
|
||||
validate-pr-title:
|
||||
name: Validate PR Title
|
||||
if: ${{ github.event_name == 'pull_request' }}
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
PR_TITLE: ${{ github.event.pull_request.title }}
|
||||
steps:
|
||||
- name: Reject titles ending in ellipsis
|
||||
run: |
|
||||
echo "PR title: $PR_TITLE"
|
||||
if [[ "$PR_TITLE" == *"..." || "$PR_TITLE" == *"…" ]]; then
|
||||
echo "❌ PR title ends with ellipsis. Please write a complete, descriptive title."
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ PR title is complete."
|
||||
|
||||
check-nightly-status:
|
||||
name: Check PyPI Version Update
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
should-proceed: ${{ steps.check-pypi.outputs.success }}
|
||||
steps:
|
||||
- name: Check PyPI package update
|
||||
id: check-pypi
|
||||
run: |
|
||||
# Get today's date in ISO format for comparison
|
||||
TODAY=$(date -u +"%Y-%m-%d")
|
||||
echo "Today's date: $TODAY"
|
||||
|
||||
# Query PyPI API for the flow package
|
||||
HTTP_STATUS=$(curl -s -o response.json -w "%{http_code}" https://pypi.org/pypi/flow-nightly/json)
|
||||
|
||||
# Check HTTP status code first
|
||||
if [ "$HTTP_STATUS" -ne 200 ]; then
|
||||
echo "Error: PyPI API returned HTTP status $HTTP_STATUS"
|
||||
echo "success=false" >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check if response is valid JSON before proceeding
|
||||
if ! jq -e . response.json >/dev/null 2>&1; then
|
||||
echo "Error: Invalid JSON response from PyPI API"
|
||||
echo "Response preview:"
|
||||
head -n 10 response.json
|
||||
echo "success=false" >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Extract the latest version
|
||||
LATEST_VERSION=$(jq -r '.info.version // empty' response.json)
|
||||
|
||||
if [ -z "$LATEST_VERSION" ]; then
|
||||
echo "Could not extract latest version"
|
||||
echo "success=false" >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Extract the release date of the latest version
|
||||
RELEASE_DATE=$(jq -r --arg ver "$LATEST_VERSION" '.releases[$ver][0].upload_time_iso_8601 // empty' response.json | cut -d'T' -f1)
|
||||
|
||||
if [ -z "$RELEASE_DATE" ]; then
|
||||
echo "Could not extract release date"
|
||||
echo "success=false" >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Latest version: $LATEST_VERSION"
|
||||
echo "Release date: $RELEASE_DATE"
|
||||
|
||||
# Check if the release date is today
|
||||
if [[ "$RELEASE_DATE" == "$TODAY" ]]; then
|
||||
echo "Package was updated today"
|
||||
echo "success=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "Package was not updated today"
|
||||
echo "success=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
# Clean up
|
||||
rm -f response.json
|
||||
|
||||
set-ci-condition:
|
||||
name: Should Run CI
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
should-run-ci: ${{ github.event.pull_request.draft == false || (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call' || github.event_name == 'merge_group') }}
|
||||
should-run-tests: ${{ !contains(github.event.pull_request.labels.*.name, 'fast-track') || github.event_name == 'workflow_call' || github.event_name == 'workflow_dispatch' || github.event_name == 'merge_group' }}
|
||||
steps:
|
||||
# Do anything just to make the job run
|
||||
- run: echo "Debug CI Condition"
|
||||
- run: echo "Labels -> ${{ join(github.event.pull_request.labels.*.name, ',') }}"
|
||||
- run: echo "IsDraft -> ${{ github.event.pull_request.draft }}"
|
||||
- run: echo "Event name -> ${{ github.event_name }}"
|
||||
- run: echo "Should run ci -> ${{ (github.event.pull_request.draft == false) || (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call' || github.event_name == 'merge_group') }}"
|
||||
- run: echo "Should run tests -> ${{ !contains(github.event.pull_request.labels.*.name, 'fast-track') || github.event_name == 'workflow_call' || github.event_name == 'workflow_dispatch' || github.event_name == 'merge_group' }}"
|
||||
|
||||
path-filter:
|
||||
needs: set-ci-condition
|
||||
if: ${{ needs.set-ci-condition.outputs.should-run-ci == 'true' && !inputs.run-all-tests }}
|
||||
name: Filter Paths
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
python: ${{ steps.filter.outputs.python }}
|
||||
frontend: ${{ steps.filter.outputs.frontend }}
|
||||
docs: ${{ steps.filter.outputs.docs }}
|
||||
frontend-tests: ${{ steps.filter.outputs.frontend-tests }}
|
||||
components-changes: ${{ steps.filter.outputs.components-changes }}
|
||||
starter-projects-changes: ${{ steps.filter.outputs.starter-projects-changes }}
|
||||
starter-projects: ${{ steps.filter.outputs.starter-projects }}
|
||||
components: ${{ steps.filter.outputs.components }}
|
||||
workspace: ${{ steps.filter.outputs.workspace }}
|
||||
api: ${{ steps.filter.outputs.api }}
|
||||
database: ${{ steps.filter.outputs.database }}
|
||||
docker: ${{ steps.filter.outputs.docker }}
|
||||
design-md: ${{ steps.filter.outputs.design-md }}
|
||||
docs-only: ${{
|
||||
steps.filter.outputs.docs == 'true' &&
|
||||
steps.filter.outputs.python != 'true' &&
|
||||
steps.filter.outputs.frontend != 'true' &&
|
||||
steps.filter.outputs['frontend-tests'] != 'true' &&
|
||||
steps.filter.outputs['components-changes'] != 'true' &&
|
||||
steps.filter.outputs['starter-projects-changes'] != 'true' &&
|
||||
steps.filter.outputs['starter-projects'] != 'true' &&
|
||||
steps.filter.outputs.components != 'true' &&
|
||||
steps.filter.outputs.workspace != 'true' &&
|
||||
steps.filter.outputs.api != 'true' &&
|
||||
steps.filter.outputs.database != 'true' &&
|
||||
steps.filter.outputs.docker != 'true'
|
||||
}}
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
- name: Filter Paths
|
||||
id: filter
|
||||
uses: dorny/paths-filter@v3
|
||||
with:
|
||||
filters: ./.github/changes-filter.yaml
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install deps for coverage check
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install --disable-pip-version-check --no-cache-dir pyyaml
|
||||
- name: Validate Filter Coverage
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
run: |
|
||||
BASE_SHA="${{ github.event.pull_request.base.sha }}"
|
||||
BASE_REF="${{ github.event.pull_request.base.ref || 'main' }}"
|
||||
# Ensure base commit is present locally
|
||||
git fetch --no-tags --depth=1 origin "$BASE_REF" || true
|
||||
git fetch --no-tags --depth=1 origin "$BASE_SHA" || true
|
||||
git diff --name-only "${BASE_SHA:-origin/$BASE_REF}"...HEAD | python scripts/check_changes_filter.py
|
||||
|
||||
test-backend:
|
||||
needs: [path-filter, set-ci-condition]
|
||||
name: Run Backend Tests
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
needs.set-ci-condition.outputs.should-run-tests == 'true'
|
||||
uses: ./.github/workflows/python_test.yml
|
||||
with:
|
||||
python-versions: ${{ inputs.python-versions || '["3.10", "3.14"]' }}
|
||||
runs-on: ${{ inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
secrets:
|
||||
OPENAI_API_KEY: "${{ secrets.OPENAI_API_KEY }}"
|
||||
ANTHROPIC_API_KEY: "${{ secrets.ANTHROPIC_API_KEY }}"
|
||||
CODECOV_TOKEN: "${{ secrets.CODECOV_TOKEN }}"
|
||||
|
||||
lint-frontend:
|
||||
needs: [path-filter, set-ci-condition]
|
||||
name: Lint Frontend
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
needs.set-ci-condition.outputs.should-run-ci == 'true' &&
|
||||
(
|
||||
inputs.run-all-tests ||
|
||||
(needs.path-filter.result != 'skipped' &&
|
||||
needs.path-filter.outputs.frontend == 'true')
|
||||
)
|
||||
uses: ./.github/workflows/lint-js.yml
|
||||
|
||||
test-frontend-unit:
|
||||
needs: [path-filter, set-ci-condition]
|
||||
name: Run Frontend Unit Tests
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
needs.set-ci-condition.outputs.should-run-tests == 'true'
|
||||
uses: ./.github/workflows/jest_test.yml
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
secrets:
|
||||
CODECOV_TOKEN: "${{ secrets.CODECOV_TOKEN }}"
|
||||
|
||||
test-frontend:
|
||||
needs: [path-filter, set-ci-condition]
|
||||
name: Run Frontend Tests
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
needs.set-ci-condition.outputs.should-run-tests == 'true'
|
||||
uses: ./.github/workflows/typescript_test.yml
|
||||
with:
|
||||
tests_folder: ${{ inputs.frontend-tests-folder }}
|
||||
release: ${{ inputs.release || false }}
|
||||
runs-on: ${{ inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
secrets:
|
||||
OPENAI_API_KEY: "${{ secrets.OPENAI_API_KEY }}"
|
||||
STORE_API_KEY: "${{ secrets.STORE_API_KEY }}"
|
||||
ANTHROPIC_API_KEY: "${{ secrets.ANTHROPIC_API_KEY }}"
|
||||
TAVILY_API_KEY: "${{ secrets.TAVILY_API_KEY }}"
|
||||
|
||||
test-docs-build:
|
||||
needs: path-filter
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
(inputs.run-all-tests || needs.path-filter.outputs.docs == 'true')
|
||||
name: Test Docs Build
|
||||
uses: ./.github/workflows/docs_test.yml
|
||||
|
||||
test-templates:
|
||||
needs: [path-filter, set-ci-condition]
|
||||
name: Test Starter Templates
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
needs.set-ci-condition.outputs.should-run-tests == 'true' &&
|
||||
(inputs.run-all-tests || needs.path-filter.outputs.python == 'true' || needs.path-filter.outputs.frontend == 'true')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: 3.12
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv sync --dev
|
||||
|
||||
- name: Test all starter project templates
|
||||
run: |
|
||||
uv run pytest src/backend/tests/unit/template/test_starter_projects.py -v -n auto
|
||||
|
||||
test-docker:
|
||||
needs: [path-filter, set-ci-condition]
|
||||
name: Test Docker Images
|
||||
if: ${{ needs.path-filter.outputs.docker == 'true' && needs.set-ci-condition.outputs.should-run-tests == 'true' }}
|
||||
uses: ./.github/workflows/docker_test.yml
|
||||
secrets:
|
||||
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
lint-design-md:
|
||||
needs: path-filter
|
||||
name: Lint DESIGN.md
|
||||
if: ${{ inputs.run-all-tests || needs.path-filter.outputs.design-md == 'true' }}
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
DESIGN_MD_CLI_VERSION: "0.1.1"
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: "22"
|
||||
|
||||
- name: Lint DESIGN.md
|
||||
run: npx --yes @google/design.md@${{ env.DESIGN_MD_CLI_VERSION }} lint DESIGN.md
|
||||
|
||||
merge-frontend-coverage:
|
||||
name: Merge Frontend Jest + Playwright Coverage Reports
|
||||
needs: [path-filter, set-ci-condition, test-frontend-unit, test-frontend]
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
needs.set-ci-condition.outputs.should-run-tests == 'true' &&
|
||||
(
|
||||
inputs.run-all-tests ||
|
||||
(needs.path-filter.result != 'skipped' &&
|
||||
needs.path-filter.outputs.docs-only != 'true')
|
||||
)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Download Jest Coverage
|
||||
uses: actions/download-artifact@v8
|
||||
continue-on-error: true
|
||||
with:
|
||||
name: jest-coverage
|
||||
path: coverage/jest
|
||||
|
||||
- name: Download Playwright Coverage
|
||||
uses: actions/download-artifact@v8
|
||||
continue-on-error: true
|
||||
with:
|
||||
pattern: playwright-coverage-*
|
||||
path: coverage/playwright
|
||||
merge-multiple: true
|
||||
|
||||
- name: Merge Coverage
|
||||
run: |
|
||||
mkdir -p coverage/combined
|
||||
npx nyc merge coverage/playwright/individual-test coverage/combined/playwright.json || echo "Failed to merge Playwright coverage"
|
||||
npx nyc merge coverage/jest coverage/combined/jest.json || echo "Failed to merge Jest coverage"
|
||||
npx nyc merge coverage/combined coverage/combined/total-frontend-coverage.json || echo "Failed to combine Playwright + Jest coverage"
|
||||
|
||||
- name: Generate Combined Report
|
||||
run: |
|
||||
npx nyc report \
|
||||
--reporter=html \
|
||||
--reporter=text \
|
||||
--temp-dir coverage/combined \
|
||||
--report-dir coverage/final
|
||||
|
||||
- name: Upload Merged Frontend Coverage to Codecov
|
||||
if: env.CODECOV_TOKEN != ''
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
uses: codecov/codecov-action@v5
|
||||
with:
|
||||
token: ${{ env.CODECOV_TOKEN }}
|
||||
files: coverage/combined/total-frontend-coverage.json
|
||||
flags: frontend
|
||||
fail_ci_if_error: false
|
||||
|
||||
# https://github.com/langchain-ai/langchain/blob/master/.github/workflows/check_diffs.yml
|
||||
ci_success:
|
||||
name: "CI Success"
|
||||
# Note: `lint-frontend` is intentionally NOT in this list — the Biome
|
||||
# check runs on every PR and shows up in the checks list, but its failure
|
||||
# does NOT block merge (informational only).
|
||||
needs:
|
||||
[
|
||||
validate-pr-title,
|
||||
test-backend,
|
||||
test-frontend-unit,
|
||||
test-frontend,
|
||||
test-docs-build,
|
||||
test-templates,
|
||||
test-docker,
|
||||
lint-design-md,
|
||||
set-ci-condition,
|
||||
path-filter,
|
||||
check-nightly-status,
|
||||
merge-frontend-coverage,
|
||||
]
|
||||
|
||||
if: ${{ always() }}
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
JOBS_JSON: ${{ toJSON(needs) }}
|
||||
RESULTS_JSON: ${{ toJSON(needs.*.result) }}
|
||||
# Skip nightly build check if only docs files changed
|
||||
DOCS_ONLY: ${{ needs.path-filter.outputs.docs-only }}
|
||||
EXIT_CODE: ${{ (contains(needs.*.result, 'failure') ||
|
||||
contains(needs.*.result, 'cancelled') ||
|
||||
(needs.check-nightly-status.outputs.should-proceed != 'true' && github.event_name != 'workflow_dispatch' && github.event_name != 'merge_group' && needs.path-filter.outputs.docs-only != 'true' && !contains(github.event.pull_request.labels.*.name, 'skip-nightly-check')))
|
||||
&& '1' || '0' }}
|
||||
steps:
|
||||
- name: "CI Success"
|
||||
run: |
|
||||
echo "=== CI Status Summary ==="
|
||||
echo "Should run tests: ${{ needs.set-ci-condition.outputs.should-run-tests }}"
|
||||
echo "Should run CI: ${{ needs.set-ci-condition.outputs.should-run-ci }}"
|
||||
echo "Nightly build status: ${{ needs.check-nightly-status.outputs.should-proceed }}"
|
||||
echo "Event type: ${{ github.event_name }}"
|
||||
echo "Docs only changes: $DOCS_ONLY"
|
||||
echo "Python changes: ${{ needs.path-filter.outputs.python }}"
|
||||
echo "Frontend changes: ${{ needs.path-filter.outputs.frontend }}"
|
||||
echo "Docs changes: ${{ needs.path-filter.outputs.docs }}"
|
||||
echo ""
|
||||
|
||||
# Check for job failures
|
||||
if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then
|
||||
echo "❌ CI FAILED: One or more jobs failed"
|
||||
echo ""
|
||||
echo "Failed jobs:"
|
||||
|
||||
# Dynamically list failed jobs with helpful descriptions
|
||||
echo "$JOBS_JSON" | jq -r '
|
||||
to_entries[]
|
||||
| select(.value.result=="failure")
|
||||
| .key as $job
|
||||
| if $job == "validate-pr-title" then
|
||||
" - PR Title Validation: Title ends with ellipsis (...). Use a complete, descriptive title."
|
||||
elif $job == "test-backend" then
|
||||
" - Backend Tests: Check Python code, tests, and dependencies"
|
||||
elif $job == "test-frontend-unit" then
|
||||
" - Frontend Unit Tests: Check React components and unit test logic"
|
||||
elif $job == "test-frontend" then
|
||||
" - Frontend E2E Tests: Check integration tests and UI functionality"
|
||||
elif $job == "test-docs-build" then
|
||||
" - Documentation Build: Check documentation syntax and build process"
|
||||
elif $job == "test-templates" then
|
||||
" - Template Tests: Check starter project templates"
|
||||
elif $job == "test-docker" then
|
||||
" - Docker Tests: Check Docker image builds and version verification"
|
||||
elif $job == "lint-design-md" then
|
||||
" - DESIGN.md Lint: Check token references, WCAG contrast, and spec compliance"
|
||||
elif $job == "path-filter" then
|
||||
" - Path Filter: File path filtering failed"
|
||||
elif $job == "set-ci-condition" then
|
||||
" - CI Condition Check: CI condition evaluation failed"
|
||||
elif $job == "check-nightly-status" then
|
||||
" - Nightly Status Check: PyPI package status check failed"
|
||||
else
|
||||
" - \($job): See job log for details"
|
||||
end
|
||||
'
|
||||
|
||||
echo ""
|
||||
echo "🔧 Next steps:"
|
||||
echo " 1. Review the failed job logs above"
|
||||
echo " 2. Fix the identified issues in your code"
|
||||
echo " 3. Push your changes to re-run the tests"
|
||||
|
||||
elif [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then
|
||||
echo "⚠️ CI CANCELLED: One or more jobs were cancelled"
|
||||
echo ""
|
||||
echo "🔧 Next steps:"
|
||||
echo " 1. Check if the cancellation was intentional"
|
||||
echo " 2. Re-run the workflow if needed"
|
||||
|
||||
elif [[ "${{ needs.check-nightly-status.outputs.should-proceed }}" != "true" && "${{ github.event_name }}" != "workflow_dispatch" && "$DOCS_ONLY" != "true" ]]; then
|
||||
echo "🚫 CI BLOCKED: Nightly build is broken"
|
||||
echo ""
|
||||
echo "The nightly PyPI package was not updated today, indicating the nightly build failed."
|
||||
echo ""
|
||||
echo "🔧 Next steps:"
|
||||
echo " 1. Work with the team to investigate and fix the nightly build"
|
||||
echo " 2. Check the nightly build logs for errors"
|
||||
echo " 3. Once the nightly build is fixed and publishes successfully, re-run this workflow"
|
||||
echo " 4. Alternatively, use 'workflow_dispatch' to manually override this check if needed"
|
||||
echo " 5. Note: PRs with only documentation changes can bypass this check"
|
||||
|
||||
else
|
||||
echo "✅ CI SUCCESS: All checks passed!"
|
||||
echo ""
|
||||
echo "🎉 Your changes are ready:"
|
||||
echo " - All tests passed"
|
||||
echo " - Code quality checks passed"
|
||||
echo " - Nightly build is healthy"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Exit code: $EXIT_CODE"
|
||||
exit $EXIT_CODE
|
||||
@@ -1,37 +0,0 @@
|
||||
name: Codeflash
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "src/backend/base/flow/**"
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
optimize:
|
||||
name: Optimize new Python code in this PR
|
||||
if: ${{ github.actor != 'codeflash-ai[bot]' }}
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
CODEFLASH_API_KEY: ${{ secrets.CODEFLASH_API_KEY }}
|
||||
CODEFLASH_PR_NUMBER: ${{ github.event.number }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
- run: uv sync
|
||||
- name: Run Codeflash Optimizer
|
||||
working-directory: ./src/backend/base
|
||||
continue-on-error: true
|
||||
run: uv run codeflash
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
name: "CodeQL"
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'dev', 'main' ]
|
||||
pull_request:
|
||||
# The branches below must be a subset of the branches above
|
||||
branches: [ 'dev' ]
|
||||
schedule:
|
||||
- cron: '17 2 * * 1'
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
name: Analyze
|
||||
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
|
||||
timeout-minutes: ${{ (matrix.language == 'swift' && 120) || 360 }}
|
||||
permissions:
|
||||
actions: read
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
language: [ 'python', 'javascript' ]
|
||||
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
|
||||
# Use only 'java' to analyze code written in Java, Kotlin or both
|
||||
# Use only 'javascript' to analyze code written in JavaScript, TypeScript or both
|
||||
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
|
||||
# Initializes the CodeQL tools for scanning.
|
||||
- name: Initialize CodeQL
|
||||
uses: github/codeql-action/init@v3
|
||||
with:
|
||||
languages: ${{ matrix.language }}
|
||||
# If you wish to specify custom queries, you can do so here or in a config file.
|
||||
# By default, queries listed here will override any specified in a config file.
|
||||
# Prefix the list here with "+" to use these queries and those in the config file.
|
||||
|
||||
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
|
||||
# queries: security-extended,security-and-quality
|
||||
|
||||
|
||||
# Autobuild attempts to build any compiled languages (C/C++, C#, Go, Java, or Swift).
|
||||
# If this step fails, then you should remove it and run the build manually (see below)
|
||||
- name: Autobuild
|
||||
uses: github/codeql-action/autobuild@v3
|
||||
|
||||
# ℹ️ Command-line programs to run using the OS shell.
|
||||
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
|
||||
|
||||
# If the Autobuild fails above, remove it and uncomment the following three lines.
|
||||
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
|
||||
|
||||
# - run: |
|
||||
# echo "Run, Build Application using script"
|
||||
# ./location_of_script_within_repo/buildscript.sh
|
||||
|
||||
- name: Perform CodeQL Analysis
|
||||
uses: github/codeql-action/analyze@v3
|
||||
with:
|
||||
category: "/language:${{matrix.language}}"
|
||||
@@ -1,27 +0,0 @@
|
||||
name: Add Community Label
|
||||
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [opened]
|
||||
|
||||
jobs:
|
||||
add-label:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Add community label
|
||||
if: github.event.pull_request.author_association != 'MEMBER' && github.event.pull_request.author_association != 'OWNER' && github.event.pull_request.author_association != 'COLLABORATOR'
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const pullRequestNumber = context.payload.pull_request.number;
|
||||
const repoDetails = {
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: pullRequestNumber
|
||||
};
|
||||
await github.rest.issues.addLabels({
|
||||
...repoDetails,
|
||||
labels: ['community']
|
||||
});
|
||||
@@ -1,29 +0,0 @@
|
||||
# Warning, do not check out untrusted code with
|
||||
# the pull_request_target event.
|
||||
name: Label PRs with Conventional Commits
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [opened, edited, synchronize]
|
||||
merge_group:
|
||||
|
||||
jobs:
|
||||
validate-pr:
|
||||
name: Validate PR
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Validate the pull request
|
||||
id: validate
|
||||
uses: Namchee/[email protected]
|
||||
with:
|
||||
access_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
issue: false
|
||||
|
||||
label:
|
||||
needs: validate-pr
|
||||
name: Label PR
|
||||
runs-on: ubuntu-latest
|
||||
if: ${{ github.event.pull_request.user.type != 'Bot'}}
|
||||
steps:
|
||||
- uses: bcoe/conventional-release-labels@v1
|
||||
with:
|
||||
type_labels: '{"feat": "enhancement","fix": "bug","docs": "documentation","style": "style","refactor": "refactor","perf": "performance","test": "test","chore": "chore","build": "build"}'
|
||||
@@ -1,36 +0,0 @@
|
||||
name: Create Release
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version:
|
||||
description: "Version to release"
|
||||
required: true
|
||||
type: string
|
||||
ref:
|
||||
description: "Commit to tag the release"
|
||||
required: true
|
||||
type: string
|
||||
pre_release:
|
||||
description: "Pre-release tag"
|
||||
required: true
|
||||
type: boolean
|
||||
|
||||
jobs:
|
||||
create_release:
|
||||
name: Create Release Job
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-main
|
||||
path: dist
|
||||
- name: Create Release Notes
|
||||
uses: ncipollo/release-action@v1
|
||||
with:
|
||||
artifacts: "dist/*"
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
draft: false
|
||||
generateReleaseNotes: true
|
||||
prerelease: ${{ inputs.pre_release }}
|
||||
tag: v${{ inputs.version }}
|
||||
commit: ${{ inputs.ref }}
|
||||
@@ -1,205 +0,0 @@
|
||||
# Cross-Platform Install Tests
|
||||
|
||||
Unified workflow for testing flow installation across multiple platforms, supporting both manual and programmatic execution.
|
||||
|
||||
## Manual Testing
|
||||
|
||||
### 1. Test from PyPI
|
||||
Tests published flow packages from PyPI across all platforms.
|
||||
|
||||
**Via GitHub UI:**
|
||||
1. Go to **Actions** → **Cross-Platform Installation Test**
|
||||
2. Check **"Test from PyPI instead of building from source"**
|
||||
3. Optionally specify a version (leave empty for latest)
|
||||
4. Click **"Run workflow"**
|
||||
|
||||
**Via CLI:**
|
||||
```bash
|
||||
# Test latest version
|
||||
gh workflow run cross-platform-test.yml -f test-from-pypi=true
|
||||
|
||||
# Test specific version
|
||||
gh workflow run cross-platform-test.yml \
|
||||
-f test-from-pypi=true \
|
||||
-f flow-version="1.0.18"
|
||||
```
|
||||
|
||||
### 2. Test from Source
|
||||
Builds and tests flow from current branch source code using release-like dependency resolution (transforms workspace dependencies to published packages for testing parity).
|
||||
|
||||
**Via GitHub UI:**
|
||||
1. Go to **Actions** → **Cross-Platform Installation Test**
|
||||
2. Leave **"Test from PyPI instead of building from source"** unchecked
|
||||
3. Click **"Run workflow"**
|
||||
|
||||
**Via CLI:**
|
||||
```bash
|
||||
# Test current branch with release-like dependencies
|
||||
gh workflow run cross-platform-test.yml -f test-from-pypi=false
|
||||
```
|
||||
|
||||
## Programmatic Testing
|
||||
|
||||
For CI, releases, and other automated workflows that test wheel installation:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
test-cross-platform:
|
||||
uses: ./.github/workflows/cross-platform-test.yml
|
||||
with:
|
||||
base-artifact-name: "dist-base"
|
||||
main-artifact-name: "dist-main"
|
||||
test-timeout: 120 # optional, defaults to 5
|
||||
```
|
||||
|
||||
## Platforms Tested
|
||||
|
||||
- **Linux**: AMD64
|
||||
- **macOS**: Intel (AMD64), Apple Silicon (ARM64)
|
||||
- **Windows**: AMD64
|
||||
- **Python versions**:
|
||||
- **All platforms**: 3.10, 3.12, and 3.13
|
||||
- **Stability**: 3.10 and 3.12 are required to pass (blocking)
|
||||
- **Preview**: 3.13 testing is optional (non-blocking) to monitor ecosystem readiness
|
||||
|
||||
## What Gets Tested
|
||||
|
||||
1. **Package Installation**: `uv pip install flow` (PyPI) or local wheel installation
|
||||
2. **Dependencies**: Additional packages like `openai` for full functionality
|
||||
3. **CLI Help**: `flow --help`
|
||||
4. **Server Startup**: `flow run --backend-only` with `/health_check` endpoint validation
|
||||
5. **Python Import**: `import flow`
|
||||
|
||||
## Common Options
|
||||
|
||||
```bash
|
||||
# Extended timeout (10 minutes instead of default 5)
|
||||
gh workflow run cross-platform-test.yml \
|
||||
-f test-timeout=10
|
||||
|
||||
# Test specific PyPI version
|
||||
gh workflow run cross-platform-test.yml \
|
||||
-f test-from-pypi=true \
|
||||
-f flow-version="1.0.18"
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Before releases**: Verify current branch works on all platforms
|
||||
- **After PyPI publish**: Confirm published packages install correctly
|
||||
- **Debugging issues**: Test specific versions when users report problems
|
||||
- **Development**: Quick cross-platform validation during feature work
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Installation Methods
|
||||
- **PyPI testing**: Uses `uv pip install` with official PyPI packages
|
||||
- **Source testing**: Transforms workspace dependencies to published packages (like nightly builds), then builds wheels from source and installs locally
|
||||
- **Dependencies**: Automatically installs additional packages (`openai`) for full functionality
|
||||
|
||||
### Health Checking
|
||||
- **Endpoint**: Uses `/health_check` for reliable server readiness validation
|
||||
- **Validation**: Checks database connectivity and chat service functionality
|
||||
- **Timeout**: Configurable timeout with proper cross-platform handling
|
||||
|
||||
### Platform-Specific Optimizations
|
||||
- **Stable versions**: Python 3.10 and 3.12 provide reliable package ecosystem support
|
||||
- **Preview testing**: Python 3.13 runs as non-blocking to monitor when it becomes viable
|
||||
- **Virtual Environments**: Uses `uv venv --seed` for consistent pip availability
|
||||
|
||||
### Workflow Architecture
|
||||
|
||||
**Unified Single-File Design:**
|
||||
|
||||
```
|
||||
cross-platform-test.yml
|
||||
├── workflow_dispatch (Manual UI)
|
||||
│ ├── PyPI Testing (test-from-pypi=true)
|
||||
│ └── Source Testing (test-from-pypi=false)
|
||||
└── workflow_call (Programmatic API)
|
||||
└── Wheel Testing (always uses wheel method)
|
||||
```
|
||||
|
||||
**Key Benefits:**
|
||||
- **Single File**: No complex workflow chains or parameter passing issues
|
||||
- **Unified Logic**: Same test matrix for all use cases
|
||||
- **Smart Routing**: Automatically determines install method based on trigger type
|
||||
- **Context-Aware**: Summary messages adapt to manual vs programmatic usage
|
||||
|
||||
### Trigger Types
|
||||
|
||||
**Manual (`workflow_dispatch`):**
|
||||
- Simple boolean toggle: "Test from PyPI" vs "Test from Source"
|
||||
- Source testing always uses release-like dependency resolution for testing parity
|
||||
- User-friendly parameter names
|
||||
- Context-specific success/failure messages
|
||||
|
||||
**Programmatic (`workflow_call`):**
|
||||
- Full parameter control for CI/releases
|
||||
- Backward compatible with existing workflows
|
||||
- Always uses wheel installation method (tests built artifacts)
|
||||
|
||||
### Implementation Details
|
||||
|
||||
The workflow uses dynamic job conditions to route execution:
|
||||
|
||||
```yaml
|
||||
# Build only runs for source testing or when no artifacts provided
|
||||
build-if-needed:
|
||||
if: |
|
||||
(github.event_name == 'workflow_dispatch' && inputs.test-from-pypi == false) ||
|
||||
(github.event_name == 'workflow_call' && (inputs.base-artifact-name == '' || inputs.main-artifact-name == ''))
|
||||
|
||||
# Test matrix adapts install method based on trigger
|
||||
test-installation:
|
||||
steps:
|
||||
- name: Determine install method
|
||||
# workflow_dispatch: maps boolean to install method
|
||||
# workflow_call: always uses wheel method
|
||||
- name: Install from PyPI
|
||||
if: steps.install-method.outputs.method == 'pypi'
|
||||
- name: Install from wheels
|
||||
if: steps.install-method.outputs.method == 'wheel'
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
|
||||
### macOS Compilation Issues (Historical)
|
||||
|
||||
**Issue**: Previously, nightly/release builds could fail on macOS with Python 3.13 due to `chroma-hnswlib` compilation errors:
|
||||
```
|
||||
clang: error: unsupported argument 'native' to option '-march='
|
||||
error: command '/usr/bin/clang++' failed with exit code 1
|
||||
```
|
||||
|
||||
**Root Cause**: Systematic difference in dependency resolution between workspace builds vs published packages:
|
||||
|
||||
| Build Type | Package Source | Dependencies | chromadb | Result |
|
||||
|------------|----------------|--------------|----------|---------|
|
||||
| **Manual/Source** | Workspace (`flow-base = { workspace = true }`) | 162 packages | ❌ Not included | ✅ Success |
|
||||
| **Nightly/Release** | Published (`flow-base-nightly==0.5.0.dev21`) | 420 packages | ✅ Included | ❌ Compilation fails |
|
||||
|
||||
**Technical Details**:
|
||||
1. **Workspace builds** use local `src/backend/base/pyproject.toml` which excludes `chromadb`
|
||||
2. **Nightly builds** modify dependencies via `scripts/ci/update_uv_dependency.py`:
|
||||
- Changes: `flow-base~=0.5.0` → `flow-base-nightly==0.5.0.dev21`
|
||||
- Uses published PyPI package with full dependency tree including `chromadb==0.5.23`
|
||||
3. **macOS clang** doesn't support `-march=native` flag used by `chroma-hnswlib` compilation
|
||||
|
||||
**Current Status**:
|
||||
- **Stable testing**: Python 3.10 and 3.12 are required to pass (blocking jobs)
|
||||
- **Preview testing**: Python 3.13 runs as non-blocking to monitor ecosystem readiness
|
||||
- **Compilation issues**: Python 3.13 may still fail due to `chroma-hnswlib` but won't block releases
|
||||
- **Manual testing**: Source builds now use the same dependency transformation as nightly builds for testing parity
|
||||
|
||||
**Files Involved**:
|
||||
- `scripts/ci/update_uv_dependency.py` - Modifies dependency resolution
|
||||
- `scripts/ci/update_pyproject_combined.py` - Orchestrates nightly build changes
|
||||
- `pyproject.toml` vs `src/backend/base/pyproject.toml` - Different dependency trees
|
||||
|
||||
## Results
|
||||
|
||||
- ✅ **Success**: All platforms pass installation and basic functionality
|
||||
- ❌ **Failure**: One or more platforms fail (check logs for details)
|
||||
- Each platform/Python combination runs independently
|
||||
- **Parallel execution**: All platforms tested simultaneously for faster feedback
|
||||
@@ -1,848 +0,0 @@
|
||||
name: Cross-Platform Installation Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
flow-version:
|
||||
description: "Flow version to test from PyPI (leave empty to test from source)"
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
workflow_call:
|
||||
inputs:
|
||||
base-artifact-name:
|
||||
description: "Name of the base package artifact"
|
||||
required: true
|
||||
type: string
|
||||
main-artifact-name:
|
||||
description: "Name of the main package artifact"
|
||||
required: true
|
||||
type: string
|
||||
lfx-artifact-name:
|
||||
description: "Name of the LFX package artifact"
|
||||
required: false
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
build-if-needed:
|
||||
name: Build Packages (if no artifacts provided)
|
||||
runs-on: ubuntu-latest
|
||||
if: inputs.flow-version == '' && contains(github.workflow_ref, 'cross-platform-test.yml')
|
||||
outputs:
|
||||
base-artifact-name: ${{ steps.set-names.outputs.base-artifact-name }}
|
||||
main-artifact-name: ${{ steps.set-names.outputs.main-artifact-name }}
|
||||
lfx-artifact-name: ${{ steps.set-names.outputs.lfx-artifact-name }}
|
||||
sdk-artifact-name: ${{ steps.set-names.outputs.sdk-artifact-name }}
|
||||
bundles-artifact-name: ${{ steps.set-names.outputs.bundles-artifact-name }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
- name: Install frontend dependencies
|
||||
run: make install_frontendci
|
||||
- name: Build frontend
|
||||
run: make build_frontend
|
||||
- name: Build base package
|
||||
run: make build_flow_base args="--wheel"
|
||||
- name: Move base package to correct directory
|
||||
run: |
|
||||
# Base package builds to dist/ but should be in src/backend/base/dist/
|
||||
mkdir -p src/backend/base/dist
|
||||
mv dist/flow_base*.whl src/backend/base/dist/
|
||||
- name: Build LFX package
|
||||
run: |
|
||||
cd src/lfx
|
||||
uv build --wheel --out-dir dist
|
||||
- name: Build SDK package
|
||||
run: |
|
||||
cd src/sdk
|
||||
uv build --wheel --out-dir dist
|
||||
- name: Build main package
|
||||
run: make build_flow args="--wheel"
|
||||
- name: Build bundle wheels
|
||||
id: build-bundles
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
bundles=(src/bundles/*/pyproject.toml)
|
||||
if [ ${#bundles[@]} -eq 0 ]; then
|
||||
echo "No bundles found under src/bundles/*/"
|
||||
echo "bundles-found=0" >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
fi
|
||||
mkdir -p bundles-dist
|
||||
BUNDLES_DIST="$PWD/bundles-dist"
|
||||
for bundle_pyproject in "${bundles[@]}"; do
|
||||
bundle_dir=$(dirname "$bundle_pyproject")
|
||||
echo "Building wheel for $bundle_dir"
|
||||
(cd "$bundle_dir" && uv build --wheel --out-dir "$BUNDLES_DIST")
|
||||
done
|
||||
echo "bundles-found=1" >> $GITHUB_OUTPUT
|
||||
ls -la bundles-dist/
|
||||
- name: Upload lfx artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: adhoc-dist-lfx
|
||||
path: src/lfx/dist
|
||||
- name: Upload SDK artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: adhoc-dist-sdk
|
||||
path: src/sdk/dist
|
||||
- name: Upload base artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: adhoc-dist-base
|
||||
path: src/backend/base/dist
|
||||
- name: Upload main artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: adhoc-dist-main
|
||||
path: dist
|
||||
- name: Upload bundles artifact
|
||||
if: steps.build-bundles.outputs.bundles-found == '1'
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: adhoc-dist-bundles
|
||||
path: bundles-dist
|
||||
- name: Set artifact names
|
||||
id: set-names
|
||||
run: |
|
||||
echo "base-artifact-name=adhoc-dist-base" >> $GITHUB_OUTPUT
|
||||
echo "main-artifact-name=adhoc-dist-main" >> $GITHUB_OUTPUT
|
||||
echo "lfx-artifact-name=adhoc-dist-lfx" >> $GITHUB_OUTPUT
|
||||
echo "sdk-artifact-name=adhoc-dist-sdk" >> $GITHUB_OUTPUT
|
||||
if [ "${{ steps.build-bundles.outputs.bundles-found }}" = "1" ]; then
|
||||
echo "bundles-artifact-name=adhoc-dist-bundles" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "bundles-artifact-name=" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
test-installation-stable:
|
||||
name: Install & Run - ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.python-version }}
|
||||
needs: [build-if-needed]
|
||||
if: always() && (needs.build-if-needed.result == 'success' || needs.build-if-needed.result == 'skipped')
|
||||
runs-on: ${{ matrix.runner }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
# Linux AMD64
|
||||
- os: linux
|
||||
arch: amd64
|
||||
runner: ubuntu-latest
|
||||
python-version: "3.10"
|
||||
- os: linux
|
||||
arch: amd64
|
||||
runner: ubuntu-latest
|
||||
python-version: "3.12"
|
||||
# macOS AMD64
|
||||
- os: macos
|
||||
arch: amd64
|
||||
runner: macos-latest-large
|
||||
python-version: "3.10"
|
||||
- os: macos
|
||||
arch: amd64
|
||||
runner: macos-latest-large
|
||||
python-version: "3.12"
|
||||
# macOS ARM64 (Apple Silicon)
|
||||
- os: macos
|
||||
arch: arm64
|
||||
runner: macos-latest
|
||||
python-version: "3.10"
|
||||
- os: macos
|
||||
arch: arm64
|
||||
runner: macos-latest
|
||||
python-version: "3.12"
|
||||
# Windows AMD64
|
||||
- os: windows
|
||||
arch: amd64
|
||||
runner: windows-latest
|
||||
python-version: "3.10"
|
||||
- os: windows
|
||||
arch: amd64
|
||||
runner: windows-latest
|
||||
python-version: "3.12"
|
||||
|
||||
steps:
|
||||
- name: Determine install method
|
||||
id: install-method
|
||||
run: |
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
echo "method=pypi" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "method=wheel" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
else
|
||||
# workflow_call always uses wheel method (backward compatibility)
|
||||
echo "method=wheel" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
architecture: ${{ matrix.arch == 'amd64' && 'x64' || matrix.arch }}
|
||||
|
||||
- name: Setup UV
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
ignore-empty-workdir: true
|
||||
|
||||
- name: Install Protocol Buffers (macOS AMD64)
|
||||
if: matrix.os == 'macos' && matrix.arch == 'amd64'
|
||||
run: |
|
||||
if ! command -v protoc &> /dev/null; then
|
||||
brew install protobuf
|
||||
else
|
||||
echo "protoc already installed, skipping"
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
# Download artifacts for wheel installation
|
||||
- name: Download flow-sdk package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel' && (inputs.sdk-artifact-name != '' || needs.build-if-needed.outputs.sdk-artifact-name != '')
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.sdk-artifact-name || needs.build-if-needed.outputs.sdk-artifact-name || 'adhoc-dist-sdk' }}
|
||||
path: ./sdk-dist
|
||||
|
||||
- name: Download LFX package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel' && (inputs.lfx-artifact-name != '' || needs.build-if-needed.outputs.lfx-artifact-name != '')
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.lfx-artifact-name || needs.build-if-needed.outputs.lfx-artifact-name || 'adhoc-dist-lfx' }}
|
||||
path: ./lfx-dist
|
||||
|
||||
- name: Download base package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel'
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.base-artifact-name || needs.build-if-needed.outputs.base-artifact-name || 'adhoc-dist-base' }}
|
||||
path: ./base-dist
|
||||
|
||||
- name: Download main package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel'
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.main-artifact-name || needs.build-if-needed.outputs.main-artifact-name || 'adhoc-dist-main' }}
|
||||
path: ./main-dist
|
||||
|
||||
- name: Download bundles package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel' && (inputs.bundles-artifact-name != '' || needs.build-if-needed.outputs.bundles-artifact-name != '')
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.bundles-artifact-name || needs.build-if-needed.outputs.bundles-artifact-name }}
|
||||
path: ./bundles-dist
|
||||
|
||||
- name: Create fresh virtual environment
|
||||
run: |
|
||||
uv venv test-env --seed
|
||||
shell: bash
|
||||
|
||||
- name: Install packages from wheel
|
||||
if: steps.install-method.outputs.method == 'wheel'
|
||||
run: |
|
||||
set -eo pipefail
|
||||
shopt -s nullglob
|
||||
|
||||
if [ "${{ matrix.os }}" = "windows" ]; then
|
||||
PYTHON=./test-env/Scripts/python.exe
|
||||
else
|
||||
PYTHON=./test-env/bin/python
|
||||
fi
|
||||
|
||||
refresh_find_links() {
|
||||
FIND_LINKS=()
|
||||
for wheel_dir in ./sdk-dist ./lfx-dist ./base-dist ./bundles-dist; do
|
||||
if [ -d "$wheel_dir" ]; then
|
||||
FIND_LINKS+=(--find-links "$wheel_dir")
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
install_wheels() {
|
||||
label="$1"
|
||||
shift
|
||||
if [ "$#" -eq 0 ]; then
|
||||
echo "No $label wheel found"
|
||||
exit 1
|
||||
fi
|
||||
printf 'Installing %s wheel(s):\n' "$label"
|
||||
printf '%s\n' "$@"
|
||||
refresh_find_links
|
||||
# Use `if-necessary-or-explicit` (not `allow`): nightly flow* wheels
|
||||
# are .dev pre-releases so we need to allow them explicitly, but `allow`
|
||||
# leaks to transitive deps and resolves pre-release pydantic alphas
|
||||
# (e.g. 2.14.0a1) which break langchain-core's module-level
|
||||
# `RunnablePassthrough()` call at import time.
|
||||
uv pip install --prerelease=if-necessary-or-explicit --python "$PYTHON" "${FIND_LINKS[@]}" "$@"
|
||||
}
|
||||
|
||||
SDK_WHEELS=(./sdk-dist/*.whl)
|
||||
LFX_WHEELS=(./lfx-dist/*.whl)
|
||||
if [ ${#SDK_WHEELS[@]} -gt 0 ] || [ ${#LFX_WHEELS[@]} -gt 0 ]; then
|
||||
install_wheels "SDK/LFX" "${SDK_WHEELS[@]}" "${LFX_WHEELS[@]}"
|
||||
fi
|
||||
|
||||
BASE_WHEELS=(./base-dist/*.whl)
|
||||
install_wheels "base" "${BASE_WHEELS[@]}"
|
||||
|
||||
if [ -d ./bundles-dist ]; then
|
||||
BUNDLE_WHEELS=(./bundles-dist/*.whl)
|
||||
install_wheels "bundle" "${BUNDLE_WHEELS[@]}"
|
||||
fi
|
||||
|
||||
MAIN_WHEELS=()
|
||||
while IFS= read -r wheel; do
|
||||
MAIN_WHEELS+=("$wheel")
|
||||
done < <(find ./main-dist -name "flow*.whl" ! -name "flow_base*.whl" -type f | sort)
|
||||
if [ ${#MAIN_WHEELS[@]} -ne 1 ]; then
|
||||
echo "Expected exactly one flow wheel in ./main-dist, found ${#MAIN_WHEELS[@]}"
|
||||
find ./main-dist -name "*.whl" -type f | sort
|
||||
exit 1
|
||||
fi
|
||||
install_wheels "main" "${MAIN_WHEELS[@]}"
|
||||
shell: bash
|
||||
|
||||
# PyPI installation steps
|
||||
- name: Install flow from PyPI (Windows)
|
||||
if: steps.install-method.outputs.method == 'pypi' && matrix.os == 'windows'
|
||||
run: |
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
uv pip install --python ./test-env/Scripts/python.exe flow==${{ inputs.flow-version }}
|
||||
else
|
||||
uv pip install --python ./test-env/Scripts/python.exe flow
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
- name: Install flow from PyPI (Unix)
|
||||
if: steps.install-method.outputs.method == 'pypi' && matrix.os != 'windows'
|
||||
run: |
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
uv pip install --python ./test-env/bin/python flow==${{ inputs.flow-version }}
|
||||
else
|
||||
uv pip install --python ./test-env/bin/python flow
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
# Install additional dependencies
|
||||
- name: Install additional dependencies (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
run: |
|
||||
uv pip install --python ./test-env/Scripts/python.exe openai
|
||||
shell: bash
|
||||
|
||||
- name: Install additional dependencies (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
run: |
|
||||
uv pip install --python ./test-env/bin/python openai
|
||||
shell: bash
|
||||
|
||||
# Test steps
|
||||
- name: Test CLI help command
|
||||
if: matrix.os == 'windows'
|
||||
run: |
|
||||
uv run --python ./test-env/Scripts/python.exe python -m flow --help
|
||||
shell: cmd
|
||||
|
||||
- name: Test CLI help command (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
run: |
|
||||
uv run --python ./test-env/bin/python python -m flow --help
|
||||
shell: bash
|
||||
|
||||
- name: Test server startup (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
timeout-minutes: 5
|
||||
run: |
|
||||
# Start server in background
|
||||
$serverProcess = Start-Process -FilePath ".\test-env\Scripts\python.exe" -ArgumentList "-m", "flow", "run", "--host", "localhost", "--port", "7860", "--backend-only" -PassThru -WindowStyle Hidden
|
||||
|
||||
# Wait for server to be ready (GitHub Actions will timeout after 5 minutes)
|
||||
do {
|
||||
try {
|
||||
$response = Invoke-WebRequest -Uri "http://localhost:7860/health_check" -UseBasicParsing -TimeoutSec 5
|
||||
if ($response.StatusCode -eq 200) {
|
||||
Write-Host "✅ Server is ready on ${{ matrix.os }}-${{ matrix.arch }}"
|
||||
break
|
||||
}
|
||||
} catch {
|
||||
Start-Sleep -Seconds 5
|
||||
}
|
||||
} while ($true)
|
||||
|
||||
# Stop the server process
|
||||
Stop-Process -Id $serverProcess.Id -Force -ErrorAction SilentlyContinue
|
||||
shell: powershell
|
||||
|
||||
- name: Test server startup (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
timeout-minutes: 5
|
||||
env:
|
||||
OBJC_DISABLE_INITIALIZE_FORK_SAFETY: YES
|
||||
run: |
|
||||
# Start server in background
|
||||
./test-env/bin/python -m flow run --host localhost --port 7860 --backend-only &
|
||||
SERVER_PID=$!
|
||||
|
||||
# Wait for server to be ready (GitHub Actions will timeout after 5 minutes)
|
||||
while true; do
|
||||
if curl -f http://localhost:7860/health_check >/dev/null 2>&1; then
|
||||
echo "✅ Server is ready on ${{ matrix.os }}-${{ matrix.arch }}"
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# Clean shutdown
|
||||
kill $SERVER_PID 2>/dev/null || true
|
||||
sleep 5
|
||||
shell: bash
|
||||
|
||||
- name: Test import in Python (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
run: |
|
||||
test-env\Scripts\python.exe -c "
|
||||
try:
|
||||
import flow
|
||||
print('✅ flow import successful on ${{ matrix.os }}-${{ matrix.arch }}')
|
||||
except Exception as e:
|
||||
print(f'❌ flow import failed on ${{ matrix.os }}-${{ matrix.arch }}: {e}')
|
||||
exit(1)
|
||||
"
|
||||
shell: cmd
|
||||
|
||||
- name: Test import in Python (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
run: |
|
||||
./test-env/bin/python -c "
|
||||
try:
|
||||
import flow
|
||||
print('✅ flow import successful on ${{ matrix.os }}-${{ matrix.arch }}')
|
||||
except Exception as e:
|
||||
print(f'❌ flow import failed on ${{ matrix.os }}-${{ matrix.arch }}: {e}')
|
||||
exit(1)
|
||||
"
|
||||
shell: bash
|
||||
|
||||
test-installation-experimental:
|
||||
name: Install & Run - ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.python-version }} (Experimental)
|
||||
needs: [build-if-needed]
|
||||
if: always() && (needs.build-if-needed.result == 'success' || needs.build-if-needed.result == 'skipped')
|
||||
runs-on: ${{ matrix.runner }}
|
||||
continue-on-error: true
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
# Python 3.13 - Experimental
|
||||
- os: linux
|
||||
arch: amd64
|
||||
runner: ubuntu-latest
|
||||
python-version: "3.13"
|
||||
- os: windows
|
||||
arch: amd64
|
||||
runner: windows-latest
|
||||
python-version: "3.13"
|
||||
- os: macos
|
||||
arch: amd64
|
||||
runner: macos-latest-large
|
||||
python-version: "3.13"
|
||||
- os: macos
|
||||
arch: arm64
|
||||
runner: macos-latest
|
||||
python-version: "3.13"
|
||||
|
||||
steps:
|
||||
- name: Determine install method
|
||||
id: install-method
|
||||
run: |
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
echo "method=pypi" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "method=wheel" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
else
|
||||
# workflow_call always uses wheel method (backward compatibility)
|
||||
echo "method=wheel" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
architecture: ${{ matrix.arch == 'amd64' && 'x64' || matrix.arch }}
|
||||
|
||||
- name: Setup UV
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
ignore-empty-workdir: true
|
||||
|
||||
- name: Install Protocol Buffers (macOS AMD64)
|
||||
if: matrix.os == 'macos' && matrix.arch == 'amd64'
|
||||
run: |
|
||||
if ! command -v protoc &> /dev/null; then
|
||||
brew install protobuf
|
||||
else
|
||||
echo "protoc already installed, skipping"
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
# Download artifacts for wheel installation
|
||||
- name: Download flow-sdk package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel' && (inputs.sdk-artifact-name != '' || needs.build-if-needed.outputs.sdk-artifact-name != '')
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.sdk-artifact-name || needs.build-if-needed.outputs.sdk-artifact-name || 'adhoc-dist-sdk' }}
|
||||
path: ./sdk-dist
|
||||
|
||||
- name: Download LFX package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel' && (inputs.lfx-artifact-name != '' || needs.build-if-needed.outputs.lfx-artifact-name != '')
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.lfx-artifact-name || needs.build-if-needed.outputs.lfx-artifact-name || 'adhoc-dist-lfx' }}
|
||||
path: ./lfx-dist
|
||||
|
||||
- name: Download base package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel'
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.base-artifact-name || needs.build-if-needed.outputs.base-artifact-name || 'adhoc-dist-base' }}
|
||||
path: ./base-dist
|
||||
|
||||
- name: Download main package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel'
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.main-artifact-name || needs.build-if-needed.outputs.main-artifact-name || 'adhoc-dist-main' }}
|
||||
path: ./main-dist
|
||||
|
||||
- name: Download bundles package artifact
|
||||
if: steps.install-method.outputs.method == 'wheel' && (inputs.bundles-artifact-name != '' || needs.build-if-needed.outputs.bundles-artifact-name != '')
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.bundles-artifact-name || needs.build-if-needed.outputs.bundles-artifact-name }}
|
||||
path: ./bundles-dist
|
||||
|
||||
- name: Create fresh virtual environment
|
||||
run: |
|
||||
uv venv test-env --seed
|
||||
shell: bash
|
||||
|
||||
- name: Install packages from wheel
|
||||
if: steps.install-method.outputs.method == 'wheel'
|
||||
run: |
|
||||
set -eo pipefail
|
||||
shopt -s nullglob
|
||||
|
||||
if [ "${{ matrix.os }}" = "windows" ]; then
|
||||
PYTHON=./test-env/Scripts/python.exe
|
||||
else
|
||||
PYTHON=./test-env/bin/python
|
||||
fi
|
||||
|
||||
refresh_find_links() {
|
||||
FIND_LINKS=()
|
||||
for wheel_dir in ./sdk-dist ./lfx-dist ./base-dist ./bundles-dist; do
|
||||
if [ -d "$wheel_dir" ]; then
|
||||
FIND_LINKS+=(--find-links "$wheel_dir")
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
install_wheels() {
|
||||
label="$1"
|
||||
shift
|
||||
if [ "$#" -eq 0 ]; then
|
||||
echo "No $label wheel found"
|
||||
exit 1
|
||||
fi
|
||||
printf 'Installing %s wheel(s):\n' "$label"
|
||||
printf '%s\n' "$@"
|
||||
refresh_find_links
|
||||
# See stable job above for rationale on --prerelease=if-necessary-or-explicit.
|
||||
uv pip install --prerelease=if-necessary-or-explicit --python "$PYTHON" "${FIND_LINKS[@]}" "$@"
|
||||
}
|
||||
|
||||
SDK_WHEELS=(./sdk-dist/*.whl)
|
||||
LFX_WHEELS=(./lfx-dist/*.whl)
|
||||
if [ ${#SDK_WHEELS[@]} -gt 0 ] || [ ${#LFX_WHEELS[@]} -gt 0 ]; then
|
||||
install_wheels "SDK/LFX" "${SDK_WHEELS[@]}" "${LFX_WHEELS[@]}"
|
||||
fi
|
||||
|
||||
BASE_WHEELS=(./base-dist/*.whl)
|
||||
install_wheels "base" "${BASE_WHEELS[@]}"
|
||||
|
||||
if [ -d ./bundles-dist ]; then
|
||||
BUNDLE_WHEELS=(./bundles-dist/*.whl)
|
||||
install_wheels "bundle" "${BUNDLE_WHEELS[@]}"
|
||||
fi
|
||||
|
||||
MAIN_WHEELS=()
|
||||
while IFS= read -r wheel; do
|
||||
MAIN_WHEELS+=("$wheel")
|
||||
done < <(find ./main-dist -name "flow*.whl" ! -name "flow_base*.whl" -type f | sort)
|
||||
if [ ${#MAIN_WHEELS[@]} -ne 1 ]; then
|
||||
echo "Expected exactly one flow wheel in ./main-dist, found ${#MAIN_WHEELS[@]}"
|
||||
find ./main-dist -name "*.whl" -type f | sort
|
||||
exit 1
|
||||
fi
|
||||
install_wheels "main" "${MAIN_WHEELS[@]}"
|
||||
shell: bash
|
||||
|
||||
- name: Force reinstall local wheels to prevent downgrades (Windows)
|
||||
if: steps.install-method.outputs.method == 'wheel' && matrix.os == 'windows'
|
||||
run: |
|
||||
# Reinstall LFX if it was provided
|
||||
if [ -n "${{ inputs.lfx-artifact-name || needs.build-if-needed.outputs.lfx-artifact-name }}" ]; then
|
||||
LFX_WHEEL=$(find ./lfx-dist -name "*.whl" -type f | head -1)
|
||||
if [ -n "$LFX_WHEEL" ]; then
|
||||
echo "Force reinstalling LFX: $LFX_WHEEL"
|
||||
INSTALL_WHEELS=("$LFX_WHEEL")
|
||||
SDK_WHEEL=""
|
||||
if [ -d ./sdk-dist ]; then
|
||||
SDK_WHEEL=$(find ./sdk-dist -name "*.whl" -type f | head -1)
|
||||
fi
|
||||
if [ -n "$SDK_WHEEL" ]; then
|
||||
echo "Force reinstalling flow-sdk with LFX: $SDK_WHEEL"
|
||||
INSTALL_WHEELS=("$SDK_WHEEL" "$LFX_WHEEL")
|
||||
fi
|
||||
NO_DEPS="--no-deps"
|
||||
if [ "${{ inputs.pre_release }}" = "true" ]; then
|
||||
NO_DEPS=""
|
||||
fi
|
||||
uv pip install --force-reinstall $NO_DEPS --prerelease=if-necessary-or-explicit --python ./test-env/Scripts/python.exe "${INSTALL_WHEELS[@]}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Reinstall base
|
||||
BASE_WHEEL=$(find ./base-dist -name "*.whl" -type f | head -1)
|
||||
if [ -n "$BASE_WHEEL" ]; then
|
||||
echo "Force reinstalling flow-base: $BASE_WHEEL"
|
||||
NO_DEPS="--no-deps"
|
||||
if [ "${{ inputs.pre_release }}" = "true" ]; then
|
||||
NO_DEPS=""
|
||||
fi
|
||||
uv pip install --force-reinstall $NO_DEPS --prerelease=if-necessary-or-explicit --python ./test-env/Scripts/python.exe "$BASE_WHEEL"
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
- name: Force reinstall local wheels to prevent downgrades (Unix)
|
||||
if: steps.install-method.outputs.method == 'wheel' && matrix.os != 'windows'
|
||||
run: |
|
||||
# Reinstall LFX if it was provided
|
||||
if [ -n "${{ inputs.lfx-artifact-name || needs.build-if-needed.outputs.lfx-artifact-name }}" ]; then
|
||||
LFX_WHEEL=$(find ./lfx-dist -name "*.whl" -type f | head -1)
|
||||
if [ -n "$LFX_WHEEL" ]; then
|
||||
echo "Force reinstalling LFX: $LFX_WHEEL"
|
||||
INSTALL_WHEELS=("$LFX_WHEEL")
|
||||
SDK_WHEEL=""
|
||||
if [ -d ./sdk-dist ]; then
|
||||
SDK_WHEEL=$(find ./sdk-dist -name "*.whl" -type f | head -1)
|
||||
fi
|
||||
if [ -n "$SDK_WHEEL" ]; then
|
||||
echo "Force reinstalling flow-sdk with LFX: $SDK_WHEEL"
|
||||
INSTALL_WHEELS=("$SDK_WHEEL" "$LFX_WHEEL")
|
||||
fi
|
||||
NO_DEPS="--no-deps"
|
||||
if [ "${{ inputs.pre_release }}" = "true" ]; then
|
||||
NO_DEPS=""
|
||||
fi
|
||||
uv pip install --force-reinstall $NO_DEPS --prerelease=if-necessary-or-explicit --python ./test-env/bin/python "${INSTALL_WHEELS[@]}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Reinstall base
|
||||
BASE_WHEEL=$(find ./base-dist -name "*.whl" -type f | head -1)
|
||||
if [ -n "$BASE_WHEEL" ]; then
|
||||
echo "Force reinstalling flow-base: $BASE_WHEEL"
|
||||
NO_DEPS="--no-deps"
|
||||
if [ "${{ inputs.pre_release }}" = "true" ]; then
|
||||
NO_DEPS=""
|
||||
fi
|
||||
uv pip install --force-reinstall $NO_DEPS --prerelease=if-necessary-or-explicit --python ./test-env/bin/python "$BASE_WHEEL"
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
# PyPI installation steps
|
||||
- name: Install flow from PyPI (Windows)
|
||||
if: steps.install-method.outputs.method == 'pypi' && matrix.os == 'windows'
|
||||
run: |
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
uv pip install --python ./test-env/Scripts/python.exe flow==${{ inputs.flow-version }}
|
||||
else
|
||||
uv pip install --python ./test-env/Scripts/python.exe flow
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
- name: Install flow from PyPI (Unix)
|
||||
if: steps.install-method.outputs.method == 'pypi' && matrix.os != 'windows'
|
||||
run: |
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
uv pip install --python ./test-env/bin/python flow==${{ inputs.flow-version }}
|
||||
else
|
||||
uv pip install --python ./test-env/bin/python flow
|
||||
fi
|
||||
shell: bash
|
||||
|
||||
# Install additional dependencies
|
||||
- name: Install additional dependencies (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
run: |
|
||||
uv pip install --python ./test-env/Scripts/python.exe openai
|
||||
shell: bash
|
||||
|
||||
- name: Install additional dependencies (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
run: |
|
||||
uv pip install --python ./test-env/bin/python openai
|
||||
shell: bash
|
||||
|
||||
# Test steps
|
||||
- name: Test CLI help command (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
run: |
|
||||
uv run --python ./test-env/Scripts/python.exe python -m flow --help
|
||||
shell: cmd
|
||||
|
||||
- name: Test CLI help command (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
run: |
|
||||
uv run --python ./test-env/bin/python python -m flow --help
|
||||
shell: bash
|
||||
|
||||
- name: Test server startup (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
timeout-minutes: 5
|
||||
run: |
|
||||
# Start server in background
|
||||
$serverProcess = Start-Process -FilePath ".\test-env\Scripts\python.exe" -ArgumentList "-m", "flow", "run", "--host", "localhost", "--port", "7860", "--backend-only" -PassThru -WindowStyle Hidden
|
||||
|
||||
# Wait for server to be ready (GitHub Actions will timeout after 5 minutes)
|
||||
do {
|
||||
try {
|
||||
$response = Invoke-WebRequest -Uri "http://localhost:7860/health_check" -UseBasicParsing -TimeoutSec 5
|
||||
if ($response.StatusCode -eq 200) {
|
||||
Write-Host "✅ Server is ready on ${{ matrix.os }}-${{ matrix.arch }}"
|
||||
break
|
||||
}
|
||||
} catch {
|
||||
Start-Sleep -Seconds 5
|
||||
}
|
||||
} while ($true)
|
||||
|
||||
# Stop the server process
|
||||
Stop-Process -Id $serverProcess.Id -Force -ErrorAction SilentlyContinue
|
||||
shell: powershell
|
||||
|
||||
- name: Test server startup (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
timeout-minutes: 5
|
||||
env:
|
||||
OBJC_DISABLE_INITIALIZE_FORK_SAFETY: YES
|
||||
run: |
|
||||
# Start server in background
|
||||
./test-env/bin/python -m flow run --host localhost --port 7860 --backend-only &
|
||||
SERVER_PID=$!
|
||||
|
||||
# Wait for server to be ready (GitHub Actions will timeout after 5 minutes)
|
||||
while true; do
|
||||
if curl -f http://localhost:7860/health_check >/dev/null 2>&1; then
|
||||
echo "✅ Server is ready on ${{ matrix.os }}-${{ matrix.arch }}"
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# Clean shutdown
|
||||
kill $SERVER_PID 2>/dev/null || true
|
||||
sleep 5
|
||||
shell: bash
|
||||
|
||||
- name: Test import in Python (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
run: |
|
||||
test-env\Scripts\python.exe -c "
|
||||
try:
|
||||
import flow
|
||||
print('✅ flow import successful on ${{ matrix.os }}-${{ matrix.arch }}')
|
||||
except Exception as e:
|
||||
print(f'❌ flow import failed on ${{ matrix.os }}-${{ matrix.arch }}: {e}')
|
||||
exit(1)
|
||||
"
|
||||
shell: cmd
|
||||
|
||||
- name: Test import in Python (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
run: |
|
||||
./test-env/bin/python -c "
|
||||
try:
|
||||
import flow
|
||||
print('✅ flow import successful on ${{ matrix.os }}-${{ matrix.arch }}')
|
||||
except Exception as e:
|
||||
print(f'❌ flow import failed on ${{ matrix.os }}-${{ matrix.arch }}: {e}')
|
||||
exit(1)
|
||||
"
|
||||
shell: bash
|
||||
|
||||
test-summary:
|
||||
name: Cross-Platform Test Summary
|
||||
needs: [test-installation-stable, test-installation-experimental]
|
||||
runs-on: ubuntu-latest
|
||||
if: always()
|
||||
steps:
|
||||
- name: Check test results
|
||||
run: |
|
||||
stable_result="${{ needs.test-installation-stable.result }}"
|
||||
experimental_result="${{ needs.test-installation-experimental.result }}"
|
||||
|
||||
echo "Stable platforms result: $stable_result"
|
||||
echo "Experimental platforms result: $experimental_result"
|
||||
|
||||
# Stable platforms must succeed, experimental can fail
|
||||
if [ "$stable_result" = "success" ]; then
|
||||
if [ "$experimental_result" = "success" ]; then
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
echo "✅ All PyPI installation tests passed (including Python 3.13)"
|
||||
else
|
||||
echo "✅ All source build and installation tests passed (including Python 3.13)"
|
||||
fi
|
||||
else
|
||||
echo "✅ All cross-platform tests passed - PyPI upload can proceed"
|
||||
fi
|
||||
else
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
if [ -n "${{ inputs.flow-version }}" ]; then
|
||||
echo "✅ PyPI installation tests passed (Python 3.13 experimental failures are acceptable)"
|
||||
else
|
||||
echo "✅ Source build and installation tests passed (Python 3.13 experimental failures are acceptable)"
|
||||
fi
|
||||
else
|
||||
echo "✅ Cross-platform tests passed - Python 3.13 experimental failures are acceptable - PyPI upload can proceed"
|
||||
fi
|
||||
fi
|
||||
elif [ "$stable_result" = "failure" ]; then
|
||||
echo "❌ Critical platform tests failed - blocking release"
|
||||
echo "Stable platforms (Python 3.10, 3.12) must pass for release"
|
||||
exit 1
|
||||
elif [ "$stable_result" = "cancelled" ]; then
|
||||
echo "❌ Critical platform tests were cancelled"
|
||||
exit 1
|
||||
else
|
||||
echo "❌ Critical platform tests were skipped unexpectedly"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,491 +0,0 @@
|
||||
name: DB Migration Validation
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
nightly_tag:
|
||||
description: "Nightly tag to test migration to"
|
||||
required: true
|
||||
type: string
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
nightly_tag:
|
||||
description: "Nightly tag to test migration to (e.g., flowai/flow-nightly:latest)"
|
||||
required: false
|
||||
type: string
|
||||
default: "flowai/flow-nightly:latest"
|
||||
# Note: This workflow is called by nightly_build.yml after Docker images are built
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: "3.13"
|
||||
POSTGRES_DB: flow_test
|
||||
POSTGRES_USER: flow
|
||||
POSTGRES_PASSWORD: flow_test_pass # pragma: allowlist secret
|
||||
|
||||
jobs:
|
||||
migration-pip-venv:
|
||||
name: "Migration Test: pip/venv (stable → nightly)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16
|
||||
env:
|
||||
POSTGRES_DB: ${{ env.POSTGRES_DB }}
|
||||
POSTGRES_USER: ${{ env.POSTGRES_USER }}
|
||||
POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
ports:
|
||||
- 5432:5432
|
||||
|
||||
steps:
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
|
||||
- name: Create test directory
|
||||
run: |
|
||||
mkdir -p migration-test
|
||||
cd migration-test
|
||||
|
||||
- name: Install latest stable Flow with PostgreSQL
|
||||
working-directory: migration-test
|
||||
run: |
|
||||
echo "Installing latest stable flow[postgresql]..."
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install "flow[postgresql]"
|
||||
|
||||
# Verify installation
|
||||
python -c 'from importlib.metadata import version; print("Installed Flow version:", version("flow"))'
|
||||
|
||||
- name: Initialize database with stable version
|
||||
working-directory: migration-test
|
||||
env:
|
||||
FLOW_DATABASE_URL: postgresql://${{ env.POSTGRES_USER }}:${{ env.POSTGRES_PASSWORD }}@localhost:5432/${{ env.POSTGRES_DB }} # pragma: allowlist secret
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
|
||||
echo "Starting Flow to initialize database..."
|
||||
# shellcheck disable=SC2016
|
||||
timeout 120 bash -c '
|
||||
python -m flow run --host 127.0.0.1 --port 7860 --backend-only &
|
||||
FLOW_PID=$!
|
||||
until curl -f http://127.0.0.1:7860/health_check 2>/dev/null; do
|
||||
sleep 2
|
||||
done
|
||||
kill $FLOW_PID
|
||||
wait $FLOW_PID 2>/dev/null || true
|
||||
' || {
|
||||
echo "Failed to start Flow stable version"
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "Database initialized successfully with stable version"
|
||||
|
||||
- name: Create witness flow
|
||||
working-directory: migration-test
|
||||
env:
|
||||
FLOW_DATABASE_URL: postgresql://${{ env.POSTGRES_USER }}:${{ env.POSTGRES_PASSWORD }}@localhost:5432/${{ env.POSTGRES_DB }}
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
|
||||
# Start Flow briefly to create test data
|
||||
python -m flow run --host 127.0.0.1 --port 7860 --backend-only &
|
||||
FLOW_PID=$!
|
||||
|
||||
# Wait for startup
|
||||
timeout 60 bash -c 'until curl -f http://127.0.0.1:7860/health_check 2>/dev/null; do sleep 2; done'
|
||||
|
||||
# Get auth token via auto_login (works under default AUTO_LOGIN=true, no credentials needed)
|
||||
TOKEN=$(curl -fsS http://127.0.0.1:7860/api/v1/auto_login | python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])")
|
||||
if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
|
||||
echo "❌ Failed to get authentication token"
|
||||
kill $FLOW_PID
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Authentication token obtained"
|
||||
|
||||
# Create a witness flow (proves row persistence)
|
||||
curl -fsSL -X POST http://127.0.0.1:7860/api/v1/flows/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d '{
|
||||
"name": "Migration Witness Flow",
|
||||
"description": "Test flow to verify data persistence across migration",
|
||||
"data": {"nodes": [], "edges": []}
|
||||
}' > flow_response.json
|
||||
|
||||
if ! grep -q '"id"' flow_response.json; then
|
||||
echo "❌ Flow creation failed - no id in response"
|
||||
cat flow_response.json
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Witness flow created successfully"
|
||||
|
||||
# Create a witness credential (type=Credential stores encrypted value, exercises encrypted-column migrations)
|
||||
curl -fsSL -X POST http://127.0.0.1:7860/api/v1/variables/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d '{"name": "migration_witness_secret", "value": "witness_value_12345", "type": "Credential", "default_fields": []}' > variable_response.json # pragma: allowlist secret
|
||||
|
||||
if ! grep -q '"id"' variable_response.json; then
|
||||
echo "❌ Credential creation failed - no id in response"
|
||||
cat variable_response.json
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Witness credential created successfully"
|
||||
|
||||
# Stop Flow
|
||||
kill $FLOW_PID
|
||||
wait $FLOW_PID 2>/dev/null || true
|
||||
|
||||
echo "Witness data created (flow + credential)"
|
||||
|
||||
- name: Upgrade to nightly version
|
||||
working-directory: migration-test
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
|
||||
NIGHTLY_TAG="${{ inputs.nightly_tag || 'flowai/flow-nightly:latest' }}"
|
||||
echo "Upgrading to nightly version: $NIGHTLY_TAG"
|
||||
|
||||
# Remove stable distributions before installing nightly.
|
||||
# flow and flow-nightly share flow-base which writes to the
|
||||
# same site-packages/flow/ namespace. If stable stays installed,
|
||||
# python -m flow still boots the stable version and no real
|
||||
# nightly migration is exercised (false-positive test).
|
||||
echo "Removing stable flow to avoid namespace collision with nightly..."
|
||||
uv pip uninstall -y flow flow-base || true
|
||||
|
||||
# Extract version from Docker tag (format: flowai/flow-nightly:v1.10.0.dev20260522)
|
||||
if [[ "$NIGHTLY_TAG" == *":"* ]]; then
|
||||
VERSION="${NIGHTLY_TAG##*:}"
|
||||
echo "Extracted version from tag: $VERSION"
|
||||
|
||||
# Strip 'v' prefix if present (PyPI doesn't use 'v' prefix)
|
||||
VERSION="${VERSION#v}"
|
||||
echo "Version for PyPI: $VERSION"
|
||||
|
||||
if [[ "$VERSION" == "latest" ]]; then
|
||||
# Install latest nightly from PyPI
|
||||
uv pip install --upgrade 'flow-nightly[postgresql]'
|
||||
else
|
||||
# Install specific version
|
||||
uv pip install --upgrade "flow-nightly[postgresql]==$VERSION"
|
||||
fi
|
||||
else
|
||||
# Direct version string (strip 'v' prefix if present)
|
||||
VERSION="${NIGHTLY_TAG#v}"
|
||||
uv pip install --upgrade "flow-nightly[postgresql]==$VERSION"
|
||||
fi
|
||||
|
||||
# Verify upgrade
|
||||
python -c 'from importlib.metadata import version; print("Upgraded to Flow Nightly version:", version("flow-nightly"))'
|
||||
|
||||
- name: Run migration and verify startup
|
||||
working-directory: migration-test
|
||||
env:
|
||||
FLOW_DATABASE_URL: postgresql://${{ env.POSTGRES_USER }}:${{ env.POSTGRES_PASSWORD }}@localhost:5432/${{ env.POSTGRES_DB }} # pragma: allowlist secret
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
|
||||
echo "Starting Flow nightly to run migrations..."
|
||||
# shellcheck disable=SC2016
|
||||
timeout 180 bash -c '
|
||||
python -m flow run --host 127.0.0.1 --port 7860 --backend-only > flow_nightly.log 2>&1 &
|
||||
FLOW_PID=$!
|
||||
|
||||
until curl -f http://127.0.0.1:7860/health_check 2>/dev/null; do
|
||||
if ! kill -0 $FLOW_PID 2>/dev/null; then
|
||||
echo "Flow process died during startup"
|
||||
cat flow_nightly.log
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "Flow nightly started successfully"
|
||||
kill $FLOW_PID
|
||||
wait $FLOW_PID 2>/dev/null || true
|
||||
' || {
|
||||
echo "Failed to start Flow nightly version"
|
||||
cat flow_nightly.log || true
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Verify witness data persisted
|
||||
working-directory: migration-test
|
||||
env:
|
||||
FLOW_DATABASE_URL: postgresql://${{ env.POSTGRES_USER }}:${{ env.POSTGRES_PASSWORD }}@localhost:5432/${{ env.POSTGRES_DB }}
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
|
||||
# Start Flow to query data
|
||||
python -m flow run --host 127.0.0.1 --port 7860 --backend-only &
|
||||
FLOW_PID=$!
|
||||
|
||||
timeout 60 bash -c 'until curl -f http://127.0.0.1:7860/health_check 2>/dev/null; do sleep 2; done'
|
||||
|
||||
# Get auth token via auto_login (works under default AUTO_LOGIN=true, no credentials needed)
|
||||
TOKEN=$(curl -fsS http://127.0.0.1:7860/api/v1/auto_login | python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])")
|
||||
if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
|
||||
echo "❌ Failed to get authentication token"
|
||||
kill $FLOW_PID
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Authentication token obtained"
|
||||
|
||||
# Verify witness flow persisted
|
||||
curl -fsSL --compressed http://127.0.0.1:7860/api/v1/flows/ \
|
||||
-H "Authorization: Bearer $TOKEN" > flows_after_migration.json
|
||||
|
||||
if grep -q "Migration Witness Flow" flows_after_migration.json; then
|
||||
echo "✅ Witness flow found after migration"
|
||||
else
|
||||
echo "❌ Witness flow NOT found after migration"
|
||||
cat flows_after_migration.json
|
||||
kill $FLOW_PID
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Verify witness credential persisted (confirms encrypted-column migration ran without data loss)
|
||||
curl -fsSL --compressed http://127.0.0.1:7860/api/v1/variables/ \
|
||||
-H "Authorization: Bearer $TOKEN" > variables_after_migration.json
|
||||
|
||||
if grep -q "migration_witness_secret" variables_after_migration.json; then
|
||||
echo "✅ Witness credential found after migration"
|
||||
else
|
||||
echo "❌ Witness credential NOT found after migration"
|
||||
cat variables_after_migration.json
|
||||
kill $FLOW_PID
|
||||
exit 1
|
||||
fi
|
||||
|
||||
kill $FLOW_PID
|
||||
wait $FLOW_PID 2>/dev/null || true
|
||||
|
||||
- name: Upload logs on failure
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: migration-pip-venv-logs
|
||||
path: |
|
||||
migration-test/*.log
|
||||
migration-test/*.json
|
||||
retention-days: 7
|
||||
|
||||
migration-docker-compose:
|
||||
name: "Migration Test: Docker Compose (stable → nightly)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Create Docker Compose test directory
|
||||
run: |
|
||||
mkdir -p docker-migration-test
|
||||
cd docker-migration-test
|
||||
|
||||
- name: Create Docker Compose file for stable
|
||||
working-directory: docker-migration-test
|
||||
run: | # pragma: allowlist secret
|
||||
cat > docker-compose.yml <<'EOF'
|
||||
services:
|
||||
flow:
|
||||
image: flowai/flow:latest
|
||||
ports:
|
||||
- "7860:7860"
|
||||
environment: # pragma: allowlist secret
|
||||
- FLOW_DATABASE_URL=postgresql://flow:flow@postgres:5432/flow # pragma: allowlist secret
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:7860/health_check"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
postgres:
|
||||
image: postgres:16
|
||||
environment: # pragma: allowlist secret
|
||||
- POSTGRES_USER=flow
|
||||
- POSTGRES_PASSWORD=flow # pragma: allowlist secret
|
||||
- POSTGRES_DB=flow
|
||||
volumes:
|
||||
- flow_postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U flow"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
volumes:
|
||||
flow_postgres_data:
|
||||
EOF
|
||||
|
||||
- name: Start stable Flow with Docker Compose
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
echo "Starting Flow stable version..."
|
||||
docker compose up -d
|
||||
|
||||
echo "Waiting for Flow to be healthy..."
|
||||
timeout 180 bash -c 'until docker compose exec -T flow curl -f http://localhost:7860/health_check 2>/dev/null; do sleep 5; done' || {
|
||||
echo "Flow stable failed to start"
|
||||
docker compose logs
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "Flow stable is running"
|
||||
|
||||
- name: Create witness flow via API
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
# Get auth token via auto_login (works under default AUTO_LOGIN=true, no credentials needed)
|
||||
TOKEN=$(curl -fsS http://localhost:7860/api/v1/auto_login | python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])")
|
||||
if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
|
||||
echo "❌ Failed to get authentication token"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Authentication token obtained"
|
||||
|
||||
echo "Creating witness flow..."
|
||||
curl -fsSL -X POST http://localhost:7860/api/v1/flows/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d '{
|
||||
"name": "Docker Migration Witness Flow",
|
||||
"description": "Test flow for Docker Compose migration",
|
||||
"data": {"nodes": [], "edges": []}
|
||||
}' > flow_response.json
|
||||
|
||||
if ! grep -q '"id"' flow_response.json; then
|
||||
echo "❌ Flow creation failed - no id in response"
|
||||
cat flow_response.json
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Witness flow created successfully"
|
||||
|
||||
echo "Creating witness credential (exercises encrypted-column migrations)..."
|
||||
curl -fsSL -X POST http://localhost:7860/api/v1/variables/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d '{"name": "migration_witness_secret", "value": "witness_value_12345", "type": "Credential", "default_fields": []}' > variable_response.json # pragma: allowlist secret
|
||||
|
||||
if ! grep -q '"id"' variable_response.json; then
|
||||
echo "❌ Credential creation failed - no id in response"
|
||||
cat variable_response.json
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Witness credential created successfully"
|
||||
|
||||
- name: Stop stable Flow (keep PostgreSQL volume)
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
echo "Stopping Flow stable..."
|
||||
docker compose stop flow
|
||||
docker compose rm -f flow
|
||||
|
||||
- name: Update to nightly image
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
NIGHTLY_TAG="${{ inputs.nightly_tag || 'flowai/flow-nightly:latest' }}"
|
||||
# Strip 'v' prefix from version part — Docker images are published without it
|
||||
# e.g. flowai/flow-nightly:v1.10.0.dev20260522 → flowai/flow-nightly:1.10.0.dev20260522
|
||||
VERSION_PART="${NIGHTLY_TAG##*:}"
|
||||
IMAGE_NAME="${NIGHTLY_TAG%%:*}"
|
||||
DOCKER_TAG="${IMAGE_NAME}:${VERSION_PART#v}"
|
||||
echo "Updating to nightly: $DOCKER_TAG"
|
||||
|
||||
# Update docker-compose.yml to use nightly image
|
||||
sed -i "s|image: flowai/flow:latest|image: $DOCKER_TAG|" docker-compose.yml
|
||||
|
||||
cat docker-compose.yml
|
||||
|
||||
- name: Start nightly Flow with same PostgreSQL volume
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
echo "Starting Flow nightly version..."
|
||||
docker compose up -d flow
|
||||
|
||||
echo "Waiting for Flow nightly to be healthy..."
|
||||
timeout 180 bash -c 'until docker compose exec -T flow curl -f http://localhost:7860/health_check 2>/dev/null; do sleep 5; done' || {
|
||||
echo "Flow nightly failed to start"
|
||||
docker compose logs flow
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "Flow nightly started successfully"
|
||||
|
||||
- name: Verify witness data persisted
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
# Get auth token via auto_login (works under default AUTO_LOGIN=true, no credentials needed)
|
||||
TOKEN=$(curl -fsS http://localhost:7860/api/v1/auto_login | python3 -c "import json,sys; print(json.load(sys.stdin)['access_token'])")
|
||||
if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
|
||||
echo "❌ Failed to get authentication token"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Authentication token obtained"
|
||||
|
||||
echo "Verifying witness flow persisted..."
|
||||
curl -fsSL --compressed http://localhost:7860/api/v1/flows/ \
|
||||
-H "Authorization: Bearer $TOKEN" > flows_after_migration.json
|
||||
|
||||
if grep -q "Docker Migration Witness Flow" flows_after_migration.json; then
|
||||
echo "✅ Witness flow found after Docker migration"
|
||||
else
|
||||
echo "❌ Witness flow NOT found after Docker migration"
|
||||
cat flows_after_migration.json
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Verifying witness credential persisted (confirms encrypted-column migration ran without data loss)..."
|
||||
curl -fsSL --compressed http://localhost:7860/api/v1/variables/ \
|
||||
-H "Authorization: Bearer $TOKEN" > variables_after_migration.json
|
||||
|
||||
if grep -q "migration_witness_secret" variables_after_migration.json; then
|
||||
echo "✅ Witness credential found after Docker migration"
|
||||
else
|
||||
echo "❌ Witness credential NOT found after Docker migration"
|
||||
cat variables_after_migration.json
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Collect logs on failure
|
||||
if: failure()
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
docker compose logs > docker-compose-logs.txt
|
||||
|
||||
- name: Upload logs on failure
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: migration-docker-compose-logs
|
||||
path: |
|
||||
docker-migration-test/*.log
|
||||
docker-migration-test/*.json
|
||||
docker-migration-test/*.txt
|
||||
retention-days: 7
|
||||
|
||||
- name: Cleanup
|
||||
if: always()
|
||||
working-directory: docker-migration-test
|
||||
run: |
|
||||
docker compose down -v
|
||||
@@ -1,711 +0,0 @@
|
||||
name: Docker Build and Push v2
|
||||
run-name: Docker Build and Push @${{ inputs.release_type }} by @${{ github.actor }}
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
release_type:
|
||||
required: true
|
||||
type: string
|
||||
description: "Release type. One of 'main', 'main-backend', 'main-frontend', 'main-ep', 'base', 'main-all'."
|
||||
pre_release:
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
ref:
|
||||
required: true
|
||||
type: string
|
||||
description: "Ref to check out (branch, tag, or commit). This is required -- it specifies where the source code for the release is located."
|
||||
push_to_registry:
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
description: "Whether to push images to registries. Set to false for testing builds without publishing."
|
||||
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_type:
|
||||
description: "Type of release. One of 'main', 'main-backend', 'main-frontend', 'main-ep', 'base', 'main-all'."
|
||||
required: true
|
||||
type: choice
|
||||
options:
|
||||
- main
|
||||
- main-backend
|
||||
- main-frontend
|
||||
- main-ep
|
||||
- base
|
||||
- main-all
|
||||
pre_release:
|
||||
description: "Whether this is a pre-release."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
ref:
|
||||
required: true
|
||||
type: string
|
||||
description: "Ref to check out (branch, tag, or commit). This is required -- it specifies where the source code for the release is located."
|
||||
push_to_registry:
|
||||
description: "Whether to push images to registries. Set to false for testing builds without publishing."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: "3.13"
|
||||
|
||||
jobs:
|
||||
determine-base-version:
|
||||
name: Determine Base Version
|
||||
if: ${{ inputs.release_type == 'base' }}
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.version.outputs.version }}
|
||||
skipped: ${{ steps.version.outputs.skipped }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
prune-cache: false
|
||||
|
||||
- name: Determine version
|
||||
id: version
|
||||
run: |
|
||||
version=$(uv tree 2>/dev/null | grep '^flow-base' | cut -d' ' -f2 | sed 's/^v//')
|
||||
echo "Base version from pyproject.toml: $version"
|
||||
|
||||
if [ ${{inputs.pre_release}} == "true" ]; then
|
||||
last_released_version=$(curl -s "https://registry.hub.docker.com/v2/repositories/ghcr.io/hanzoai/flow/tags?page_size=100" | jq -r '.results[].name' | grep -E '\.rc[0-9]+' | sort -V | tail -n 1)
|
||||
version="$(uv run ./scripts/ci/flow_pre_release_tag.py "$version" "$last_released_version")"
|
||||
echo "Latest base pre-release version: $last_released_version"
|
||||
echo "Base pre-release version to be released: $version"
|
||||
else
|
||||
last_released_version=$(curl -s "https://registry.hub.docker.com/v2/repositories/ghcr.io/hanzoai/flow/tags?page_size=100" | jq -r '.results[].name' | grep -v 'latest' | sort -V | tail -n 1)
|
||||
echo "Latest base release version: $last_released_version"
|
||||
fi
|
||||
|
||||
if [ "$version" = "$last_released_version" ]; then
|
||||
echo "Base docker version $version is already released. Skipping release."
|
||||
echo skipped=true >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
else
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
echo skipped=false >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
determine-main-version:
|
||||
name: Determine Main Version
|
||||
if: ${{ inputs.release_type != 'base' }}
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.version.outputs.version }}
|
||||
skipped: ${{ steps.version.outputs.skipped }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
prune-cache: false
|
||||
|
||||
- name: Determine version
|
||||
id: version
|
||||
run: |
|
||||
version=$(uv tree 2>/dev/null | grep -E '^flow(-nightly)?[[:space:]]' | cut -d' ' -f2 | sed 's/^v//')
|
||||
echo "Main version from pyproject.toml: $version"
|
||||
|
||||
if [ ${{inputs.pre_release}} == "true" ]; then
|
||||
last_released_version=$(curl -s "https://registry.hub.docker.com/v2/repositories/ghcr.io/hanzoai/flow/tags?page_size=100" | jq -r '.results[].name' | grep -E '\.rc[0-9]+' | sort -V | tail -n 1)
|
||||
version="$(uv run ./scripts/ci/flow_pre_release_tag.py "$version" "$last_released_version")"
|
||||
echo "Latest main pre-release version: $last_released_version"
|
||||
echo "Main pre-release version to be released: $version"
|
||||
else
|
||||
last_released_version=$(curl -s "https://registry.hub.docker.com/v2/repositories/ghcr.io/hanzoai/flow/tags?page_size=100" | jq -r '.results[].name' | grep -v 'latest' | sort -V | tail -n 1)
|
||||
echo "Latest main release version: $last_released_version"
|
||||
fi
|
||||
|
||||
# Skip check only applies to main image, not backend/frontend which are separate images
|
||||
if [ "$version" = "$last_released_version" ] && [[ "${{ inputs.release_type }}" != "main-backend" ]] && [[ "${{ inputs.release_type }}" != "main-frontend" ]]; then
|
||||
echo "Main docker version $version is already released. Skipping release."
|
||||
echo skipped=true >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
else
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
echo skipped=false >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
build-base:
|
||||
name: Build Base Package
|
||||
needs: [determine-base-version]
|
||||
if: ${{ inputs.release_type == 'base' && needs.determine-base-version.outputs.skipped == 'false' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Set tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ needs.determine-base-version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow:base-${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow:base-${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push_base.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push_base.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build-main:
|
||||
name: Build Main Package
|
||||
needs: [determine-main-version]
|
||||
if: ${{ inputs.release_type == 'main' && needs.determine-main-version.outputs.skipped == 'false' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Set tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ needs.determine-main-version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build-main-backend:
|
||||
name: Build Main Backend Package
|
||||
needs: [determine-main-version]
|
||||
if: ${{ inputs.release_type == 'main-backend' && needs.determine-main-version.outputs.skipped == 'false' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Set tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ needs.determine-main-version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-backend:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-backend:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
build-args: |
|
||||
FLOW_IMAGE=ghcr.io/hanzoai/flow:${{ steps.version.outputs.version }}-${{ matrix.arch }}
|
||||
file: ./docker/build_and_push_backend.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
build-args: |
|
||||
FLOW_IMAGE=ghcr.io/hanzoai/flow:${{ steps.version.outputs.version }}-${{ matrix.arch }}
|
||||
file: ./docker/build_and_push_backend.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build-main-frontend:
|
||||
name: Build Main Frontend Package
|
||||
needs: [determine-main-version]
|
||||
if: ${{ inputs.release_type == 'main-frontend' && needs.determine-main-version.outputs.skipped == 'false' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Set tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ needs.determine-main-version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-frontend:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-frontend:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/frontend/build_and_push_frontend.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/frontend/build_and_push_frontend.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build-main-ep:
|
||||
name: Build Main EP Package
|
||||
needs: [determine-main-version]
|
||||
if: ${{ inputs.release_type == 'main-ep' && needs.determine-main-version.outputs.skipped == 'false' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Set tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ needs.determine-main-version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-ep:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-ep:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push_ep.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push_ep.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build-main-all:
|
||||
name: Build Main All Package
|
||||
needs: [determine-main-version]
|
||||
if: ${{ inputs.release_type == 'main-all' && needs.determine-main-version.outputs.skipped == 'false' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Set tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ needs.determine-main-version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-all:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-all:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push_with_extras.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
file: ./docker/build_and_push_with_extras.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
create-manifest:
|
||||
name: Create Multi-Arch Manifest
|
||||
needs:
|
||||
[
|
||||
build-base,
|
||||
build-main,
|
||||
build-main-backend,
|
||||
build-main-frontend,
|
||||
build-main-ep,
|
||||
build-main-all,
|
||||
determine-base-version,
|
||||
determine-main-version,
|
||||
]
|
||||
runs-on: ubuntu-latest
|
||||
if: ${{ always() && inputs.push_to_registry && (needs.build-base.result == 'success' || needs.build-main.result == 'success' || needs.build-main-backend.result == 'success' || needs.build-main-frontend.result == 'success' || needs.build-main-ep.result == 'success' || needs.build-main-all.result == 'success') }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Get version
|
||||
id: version
|
||||
run: |
|
||||
if [[ "${{ inputs.release_type }}" == "base" ]]; then
|
||||
version=$(uv tree 2>/dev/null | grep 'flow-base' | awk '{print $3}' | sed 's/^v//' | head -n 1)
|
||||
else
|
||||
version=$(uv tree 2>/dev/null | grep -E '^flow(-nightly)?[[:space:]]' | cut -d' ' -f2 | sed 's/^v//')
|
||||
fi
|
||||
echo "Using version: $version"
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Set tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ needs.determine-main-version.outputs.version }}"
|
||||
case "${{ inputs.release_type }}" in
|
||||
"base")
|
||||
version="${{ needs.determine-base-version.outputs.version }}"
|
||||
if [[ "${{ inputs.pre_release }}" == "true" ]]; then
|
||||
echo "final_tags=ghcr.io/hanzoai/flow:base-${version},ghcr.io/hanzoai/flow:base-${version}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "final_tags=ghcr.io/hanzoai/flow:base-${version},ghcr.io/hanzoai/flow:base-latest,ghcr.io/hanzoai/flow:base-${version},ghcr.io/hanzoai/flow:base-latest" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
echo "arch_base=ghcr.io/hanzoai/flow:base-${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow:base-${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
"main")
|
||||
if [[ "${{ inputs.pre_release }}" == "true" ]]; then
|
||||
echo "final_tags=ghcr.io/hanzoai/flow:${version},ghcr.io/hanzoai/flow:${version}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "final_tags=ghcr.io/hanzoai/flow:${version},ghcr.io/hanzoai/flow:latest,ghcr.io/hanzoai/flow:${version},ghcr.io/hanzoai/flow:latest" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
echo "arch_base=ghcr.io/hanzoai/flow:${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow:${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
"main-backend")
|
||||
if [[ "${{ inputs.pre_release }}" == "true" ]]; then
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-backend:${version},ghcr.io/hanzoai/flow-backend:${version}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-backend:${version},ghcr.io/hanzoai/flow-backend:latest,ghcr.io/hanzoai/flow-backend:${version},ghcr.io/hanzoai/flow-backend:latest" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
echo "arch_base=ghcr.io/hanzoai/flow-backend:${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow-backend:${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
"main-frontend")
|
||||
if [[ "${{ inputs.pre_release }}" == "true" ]]; then
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-frontend:${version},ghcr.io/hanzoai/flow-frontend:${version}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-frontend:${version},ghcr.io/hanzoai/flow-frontend:latest,ghcr.io/hanzoai/flow-frontend:${version},ghcr.io/hanzoai/flow-frontend:latest" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
echo "arch_base=ghcr.io/hanzoai/flow-frontend:${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow-frontend:${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
"main-ep")
|
||||
if [[ "${{ inputs.pre_release }}" == "true" ]]; then
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-ep:${version},ghcr.io/hanzoai/flow-ep:${version}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-ep:${version},ghcr.io/hanzoai/flow-ep:latest,ghcr.io/hanzoai/flow-ep:${version},ghcr.io/hanzoai/flow-ep:latest" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
echo "arch_base=ghcr.io/hanzoai/flow-ep:${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow-ep:${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
"main-all")
|
||||
if [[ "${{ inputs.pre_release }}" == "true" ]]; then
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-all:${version},ghcr.io/hanzoai/flow-all:${version}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-all:${version},ghcr.io/hanzoai/flow-all:latest,ghcr.io/hanzoai/flow-all:${version},ghcr.io/hanzoai/flow-all:latest" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
echo "arch_base=ghcr.io/hanzoai/flow-all:${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow-all:${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
*)
|
||||
echo "Error: Invalid release_type: ${{ inputs.release_type }}" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Create and push multi-arch manifests
|
||||
run: |
|
||||
# Split tags and create manifests for each
|
||||
IFS=',' read -ra TAGS <<< "${{ steps.tags.outputs.final_tags }}"
|
||||
for tag in "${TAGS[@]}"; do
|
||||
echo "Creating manifest for $tag"
|
||||
|
||||
# Determine architecture-specific tags
|
||||
if [[ "$tag" == *"flowai"* ]]; then
|
||||
amd64_tag="${{ steps.tags.outputs.arch_base }}-amd64"
|
||||
arm64_tag="${{ steps.tags.outputs.arch_base }}-arm64"
|
||||
else
|
||||
amd64_tag="${{ steps.tags.outputs.ghcr_arch_base }}-amd64"
|
||||
arm64_tag="${{ steps.tags.outputs.ghcr_arch_base }}-arm64"
|
||||
fi
|
||||
|
||||
docker buildx imagetools create \
|
||||
--tag "$tag" \
|
||||
"$amd64_tag" \
|
||||
"$arm64_tag"
|
||||
done
|
||||
@@ -1,401 +0,0 @@
|
||||
name: Docker Build and Push
|
||||
run-name: Docker Build and Push @${{ inputs.release_type }} by @${{ github.actor }}
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
main_version:
|
||||
required: true
|
||||
type: string
|
||||
description: "Main version to tag images with. Required for both main and base releases."
|
||||
base_version:
|
||||
required: false
|
||||
type: string
|
||||
description: "Base version to tag images with. Required for base release type."
|
||||
release_type:
|
||||
required: true
|
||||
type: string
|
||||
description: "Release type. One of 'main', 'main-ep', 'base', 'nightly-main', 'nightly-base', 'main-all', 'nightly-main-all'."
|
||||
pre_release:
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
ref:
|
||||
required: false
|
||||
type: string
|
||||
description: "Ref to check out. If not specified, will default to the main version or current branch."
|
||||
warning_check:
|
||||
required: true
|
||||
type: boolean
|
||||
description: "Warning - use docker-build-v2 unless you have a very valid reason for using this deprecated workflow. By setting to True, you acknowledge all risks of using a possibly breaking workflow."
|
||||
default: false
|
||||
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
main_version:
|
||||
description: "Main version to tag images with. Required for both main and base releases."
|
||||
required: false
|
||||
type: string
|
||||
base_version:
|
||||
description: "Base version to tag images with. Required for base release type."
|
||||
required: false
|
||||
type: string
|
||||
release_type:
|
||||
description: "Type of release. One of 'main', 'main-ep', 'base', 'nightly-main', 'nightly-base', 'main-all', 'nightly-main-all'."
|
||||
required: true
|
||||
type: string
|
||||
pre_release:
|
||||
description: "Whether this is a pre-release."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
ref:
|
||||
required: false
|
||||
type: string
|
||||
description: "Ref to check out. If not specified, will default to the main version or current branch."
|
||||
warning_check:
|
||||
description: "Warning - use docker-build-v2 unless you have a very valid reason for using this deprecated workflow. By setting to True, you acknowledge all risks of using a possibly breaking workflow."
|
||||
required: true
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: "3.13"
|
||||
TEST_TAG: "ghcr.io/hanzoai/flow:test"
|
||||
|
||||
jobs:
|
||||
get-version:
|
||||
name: Get Version
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.get-version-input.outputs.version || steps.get-version-base.outputs.version || steps.get-version-main.outputs.version }}
|
||||
steps:
|
||||
- name: Verify warning check
|
||||
run: |
|
||||
if [[ "${{ inputs.warning_check }}" == "false" ]]; then
|
||||
echo "Warning - use docker-build-v2 unless you have a very valid reason for using this deprecated workflow. By setting to True, you acknowledge all risks of using a possibly breaking workflow."
|
||||
exit 1
|
||||
else
|
||||
echo "User has acknowledged the risks of using this deprecated workflow. Proceeding with build."
|
||||
fi
|
||||
|
||||
- name: Verify a main version exists
|
||||
if: ${{ inputs.main_version == '' }}
|
||||
run: |
|
||||
# due to our how we split packages, we need to have a main version to check out.
|
||||
echo "Must specify a main version to check out."
|
||||
exit 1
|
||||
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || inputs.main_version || github.ref }}
|
||||
persist-credentials: true
|
||||
|
||||
- name: Get Version to Tag
|
||||
if: ${{ inputs.main_version != '' }}
|
||||
id: get-version-input
|
||||
run: |
|
||||
# Produces the versions we will use to tag the docker images with.
|
||||
|
||||
if [[ "${{ inputs.release_type }}" == "base" && "${{ inputs.base_version }}" == '' ]]; then
|
||||
echo "Must specify a base version for base release type."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${{ inputs.release_type }}" == "nightly-base" && "${{ inputs.base_version }}" == '' ]]; then
|
||||
echo "Must specify a base version for nightly-base release type."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ("${{ inputs.release_type }}" == "main" || "${{ inputs.release_type }}" == "main-all") && "${{ inputs.main_version }}" == '' ]]; then
|
||||
echo "Must specify a main version for main release type."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${{ inputs.release_type }}" == "main-ep" && "${{ inputs.main_version }}" == '' ]]; then
|
||||
echo "Must specify a main version for main-ep release type."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ("${{ inputs.release_type }}" == "nightly-main" || "${{ inputs.release_type }}" == "nightly-main-all") && "${{ inputs.main_version }}" == '' ]]; then
|
||||
echo "Must specify a main version for nightly-main release type."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${{ inputs.release_type }}" == "base" || "${{ inputs.release_type }}" == "nightly-base" ]]; then
|
||||
version=${{ inputs.base_version }}
|
||||
echo "base version=${{ inputs.base_version }}"
|
||||
echo version=$version
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
elif [[ "${{ inputs.release_type }}" == "main" || "${{ inputs.release_type }}" == "main-ep" || "${{ inputs.release_type }}" == "nightly-main" || "${{ inputs.release_type }}" == "nightly-main-all" ]]; then
|
||||
version=${{ inputs.main_version }}
|
||||
echo version=$version
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "No version or ref specified. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
- name: Get Version Base
|
||||
if: ${{ inputs.base_version == '' && (inputs.release_type == 'base' || inputs.release_type == 'nightly-base') }}
|
||||
id: get-version-base
|
||||
run: |
|
||||
version=$(uv tree | grep 'flow-base' | awk '{print $3}' | sed 's/^v//' | head -n 1)
|
||||
if [ -z "$version" ]; then
|
||||
echo "Failed to extract version from uv tree output"
|
||||
exit 1
|
||||
fi
|
||||
echo version=$version
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
- name: Get Version Main
|
||||
if: ${{ inputs.main_version == '' && (inputs.release_type == 'main' || inputs.release_type == 'main-ep' || inputs.release_type == 'nightly-main' || inputs.release_type == 'main-all' || inputs.release_type == 'nightly-main-all') }}
|
||||
id: get-version-main
|
||||
run: |
|
||||
version=$(uv tree | grep -E '^flow(-nightly)?[[:space:]]' | awk '{print $2}' | sed 's/^v//')
|
||||
echo version=$version
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
setup:
|
||||
runs-on: ubuntu-latest
|
||||
needs: get-version
|
||||
outputs:
|
||||
docker_tags: ${{ steps.set-vars.outputs.docker_tags }}
|
||||
ghcr_tags: ${{ steps.set-vars.outputs.ghcr_tags }}
|
||||
file: ${{ steps.set-vars.outputs.file }}
|
||||
steps:
|
||||
- name: Set Dockerfile and Tags
|
||||
id: set-vars
|
||||
run: |
|
||||
nightly_suffix=''
|
||||
if [[ "${{ inputs.release_type }}" == "nightly-base" || "${{ inputs.release_type }}" == "nightly-main" || "${{ inputs.release_type }}" == "nightly-main-all" ]]; then
|
||||
nightly_suffix="-nightly"
|
||||
fi
|
||||
|
||||
if [[ "${{ inputs.release_type }}" == "base" || "${{ inputs.release_type }}" == "nightly-base" ]]; then
|
||||
# FLOW-BASE RELEASE
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow${nightly_suffix}:base-${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow${nightly_suffix}:base-latest" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow${nightly_suffix}:base-${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow${nightly_suffix}:base-latest" >> $GITHUB_OUTPUT
|
||||
echo "file=./docker/build_and_push_base.Dockerfile" >> $GITHUB_OUTPUT
|
||||
else
|
||||
if [[ "${{ inputs.pre_release }}" == "true" ]]; then
|
||||
# FLOW-MAIN PRE-RELEASE
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow${nightly_suffix}:${{ needs.get-version.outputs.version }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow${nightly_suffix}:${{ needs.get-version.outputs.version }}" >> $GITHUB_OUTPUT
|
||||
echo "file=./docker/build_and_push.Dockerfile" >> $GITHUB_OUTPUT
|
||||
elif [[ "${{ inputs.release_type }}" == "main-ep" ]]; then
|
||||
# FLOW-MAIN (ENTRYPOINT) RELEASE
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-ep${nightly_suffix}:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-ep${nightly_suffix}:latest" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-ep${nightly_suffix}:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-ep${nightly_suffix}:latest" >> $GITHUB_OUTPUT
|
||||
echo "file=./docker/build_and_push_ep.Dockerfile" >> $GITHUB_OUTPUT
|
||||
elif [[ "${{ inputs.release_type }}" == "main" || "${{ inputs.release_type }}" == "nightly-main" ]]; then
|
||||
# FLOW-MAIN RELEASE
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow${nightly_suffix}:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow${nightly_suffix}:latest" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow${nightly_suffix}:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow${nightly_suffix}:latest" >> $GITHUB_OUTPUT
|
||||
echo "file=./docker/build_and_push.Dockerfile" >> $GITHUB_OUTPUT
|
||||
elif [[ "${{ inputs.release_type }}" == "main-all" || "${{ inputs.release_type }}" == "nightly-main-all" ]]; then
|
||||
# FLOW-MAIN (ALL OPTIONAL DEPS) RELEASE
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-all${nightly_suffix}:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-all${nightly_suffix}:latest" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-all${nightly_suffix}:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-all${nightly_suffix}:latest" >> $GITHUB_OUTPUT
|
||||
echo "file=./docker/build_and_push_with_extras.Dockerfile" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "Invalid release type. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
build:
|
||||
runs-on: [self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
needs: [get-version, setup]
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || inputs.main_version || github.ref }}
|
||||
persist-credentials: true
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
- name: Install the project
|
||||
run: |
|
||||
if [[ "${{ inputs.release_type }}" == "base" || "${{ inputs.release_type }}" == "nightly-base" ]]; then
|
||||
uv sync --directory src/backend/base --no-dev --no-sources
|
||||
else
|
||||
uv sync --no-dev --no-sources
|
||||
fi
|
||||
|
||||
- name: Docker System Info and Cleanup
|
||||
run: |
|
||||
echo "=== Docker System Usage Before Cleanup ==="
|
||||
docker system df || true
|
||||
docker buildx du || true
|
||||
|
||||
echo "=== Cleaning up Docker System ==="
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
echo "=== Docker System Usage After Cleanup ==="
|
||||
docker system df || true
|
||||
docker buildx du || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
version: latest
|
||||
driver: docker-container
|
||||
driver-opts: |
|
||||
image=moby/buildkit:v0.22.0
|
||||
network=host
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Build and Push to Docker Hub
|
||||
uses: Wandalen/wretry.action@master
|
||||
with:
|
||||
action: docker/build-push-action@v6
|
||||
with: |
|
||||
context: .
|
||||
push: true
|
||||
file: ${{ needs.setup.outputs.file }}
|
||||
tags: ${{ needs.setup.outputs.docker_tags }}
|
||||
platforms: linux/amd64,linux/arm64
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Login to Github Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Github Container Registry
|
||||
uses: Wandalen/wretry.action@master
|
||||
with:
|
||||
action: docker/build-push-action@v6
|
||||
with: |
|
||||
context: .
|
||||
push: true
|
||||
file: ${{ needs.setup.outputs.file }}
|
||||
tags: ${{ needs.setup.outputs.ghcr_tags }}
|
||||
platforms: linux/amd64,linux/arm64
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build_components:
|
||||
if: ${{ inputs.release_type == 'main' }}
|
||||
runs-on: [self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
permissions:
|
||||
packages: write
|
||||
needs: [build, get-version]
|
||||
strategy:
|
||||
matrix:
|
||||
component:
|
||||
[docker-backend, docker-frontend, ghcr-backend, ghcr-frontend]
|
||||
include:
|
||||
- component: docker-backend
|
||||
dockerfile: ./docker/build_and_push_backend.Dockerfile
|
||||
tags: ghcr.io/hanzoai/flow-backend:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-backend:latest
|
||||
flow_image: ghcr.io/hanzoai/flow:${{ needs.get-version.outputs.version }}
|
||||
- component: docker-frontend
|
||||
dockerfile: ./docker/frontend/build_and_push_frontend.Dockerfile
|
||||
tags: ghcr.io/hanzoai/flow-frontend:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-frontend:latest
|
||||
flow_image: ghcr.io/hanzoai/flow:${{ needs.get-version.outputs.version }}
|
||||
- component: ghcr-backend
|
||||
dockerfile: ./docker/build_and_push_backend.Dockerfile
|
||||
tags: ghcr.io/hanzoai/flow-backend:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-backend:latest
|
||||
flow_image: ghcr.io/hanzoai/flow:${{ needs.get-version.outputs.version }}
|
||||
- component: ghcr-frontend
|
||||
dockerfile: ./docker/frontend/build_and_push_frontend.Dockerfile
|
||||
tags: ghcr.io/hanzoai/flow-frontend:${{ needs.get-version.outputs.version }},ghcr.io/hanzoai/flow-frontend:latest
|
||||
flow_image: ghcr.io/hanzoai/flow:${{ needs.get-version.outputs.version }}
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || inputs.main_version || github.ref }}
|
||||
|
||||
- name: Docker System Info and Cleanup
|
||||
run: |
|
||||
echo "=== Docker System Usage Before Cleanup ==="
|
||||
docker system df || true
|
||||
docker buildx du || true
|
||||
|
||||
echo "=== Cleaning up Docker System ==="
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
echo "=== Docker System Usage After Cleanup ==="
|
||||
docker system df || true
|
||||
docker buildx du || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
version: latest
|
||||
driver: docker-container
|
||||
driver-opts: |
|
||||
image=moby/buildkit:v0.22.0
|
||||
network=host
|
||||
|
||||
- name: Login to Docker Hub
|
||||
if: ${{ matrix.component == 'docker-backend' }} || ${{ matrix.component == 'docker-frontend' }}
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to Github Container Registry
|
||||
if: ${{ matrix.component == 'ghcr-backend' }} || ${{ matrix.component == 'ghcr-frontend' }}
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Wait for propagation (for backend)
|
||||
run: sleep 120
|
||||
|
||||
- name: Build and push ${{ matrix.component }}
|
||||
uses: Wandalen/wretry.action@master
|
||||
with:
|
||||
action: docker/build-push-action@v6
|
||||
with: |
|
||||
context: .
|
||||
push: true
|
||||
build-args: |
|
||||
FLOW_IMAGE=${{ matrix.flow_image }}
|
||||
file: ${{ matrix.dockerfile }}
|
||||
tags: ${{ matrix.tags }}
|
||||
# provenance: false will result in a single manifest for all platforms which makes the image pullable from arm64 machines via the emulation (e.g. Apple Silicon machines)
|
||||
provenance: false
|
||||
|
||||
restart-space:
|
||||
name: Restart HuggingFace Spaces
|
||||
if: ${{ inputs.release_type == 'main' }}
|
||||
runs-on: ubuntu-latest
|
||||
needs: [build, get-version]
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- "3.13"
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || inputs.main_version || github.ref }}
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Restart HuggingFace Spaces Build
|
||||
run: |
|
||||
uv run ./scripts/factory_restart_space.py --space "Flow/Flow" --token ${{ secrets.HUGGINGFACE_API_TOKEN }}
|
||||
@@ -1,420 +0,0 @@
|
||||
name: Docker Nightly Build and Push
|
||||
run-name: Docker Nightly Build @${{ inputs.release_type }} by @${{ github.actor }}
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
release_type:
|
||||
required: true
|
||||
type: string
|
||||
description: "Nightly release type. One of 'nightly-main', 'nightly-base', 'nightly-main-all'."
|
||||
ref:
|
||||
required: true
|
||||
type: string
|
||||
description: "Ref to check out (branch, tag, or commit). This is required -- it specifies where the source code for the release is located."
|
||||
push_to_registry:
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
description: "Whether to push images to registries. Set to false for testing builds without publishing."
|
||||
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_type:
|
||||
description: "Nightly release type. One of 'nightly-main', 'nightly-base', 'nightly-main-all'."
|
||||
required: true
|
||||
type: choice
|
||||
options:
|
||||
- nightly-main
|
||||
- nightly-base
|
||||
- nightly-main-all
|
||||
ref:
|
||||
required: true
|
||||
type: string
|
||||
description: "Ref to check out (branch, tag, or commit). This is required -- it specifies where the source code for the release is located. Note that if running via Github Actions, this tag (formatted as, e.g., v1.5.1.dev36) must be manually created and pushed to a branch prior to running the workflow."
|
||||
push_to_registry:
|
||||
description: "Whether to push images to registries. Set to false for testing builds without publishing."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: "3.13"
|
||||
|
||||
jobs:
|
||||
build-nightly-base:
|
||||
name: Build Nightly Base Package
|
||||
if: ${{ inputs.release_type == 'nightly-base' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
prune-cache: false
|
||||
|
||||
- name: Get version for tagging
|
||||
id: version
|
||||
run: |
|
||||
echo "Extracting base version from pyproject.toml"
|
||||
version=$(uv tree 2>/dev/null | grep 'flow-base' | awk '{print $3}' | sed 's/^v//' | head -n 1)
|
||||
|
||||
|
||||
# Verify nightly tag format
|
||||
if [[ ! "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.dev[0-9]+$ ]]; then
|
||||
echo "Base version format is incorrect. Must be in the format (e.g.) v1.5.1.dev36"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Using version: $version"
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Set nightly tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ steps.version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-nightly:base-${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-nightly:base-${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
pull: true
|
||||
file: ./docker/build_and_push_base.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
pull: true
|
||||
file: ./docker/build_and_push_base.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build-nightly-main:
|
||||
name: Build Nightly Main Package
|
||||
if: ${{ inputs.release_type == 'nightly-main' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
prune-cache: false
|
||||
|
||||
- name: Get version for tagging
|
||||
id: version
|
||||
run: |
|
||||
echo "Extracting main version from pyproject.toml"
|
||||
version=$(uv tree 2>/dev/null | grep -E '^flow(-nightly)?[[:space:]]' | cut -d' ' -f2 | sed 's/^v//')
|
||||
|
||||
# Verify nightly tag format
|
||||
if [[ ! "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.dev[0-9]+$ ]]; then
|
||||
echo "Main version format is incorrect. Must be in the format (e.g.) v1.5.1.dev36"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Using version: $version"
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Set nightly tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ steps.version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-nightly:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-nightly:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
pull: true
|
||||
file: ./docker/build_and_push.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
pull: true
|
||||
file: ./docker/build_and_push.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
build-nightly-main-all:
|
||||
name: Build Nightly Main All Package
|
||||
if: ${{ inputs.release_type == 'nightly-main-all' }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- arch: amd64
|
||||
runner: [Flow-runner]
|
||||
- arch: arm64
|
||||
runner:
|
||||
[self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
prune-cache: false
|
||||
|
||||
- name: Get version for tagging
|
||||
id: version
|
||||
run: |
|
||||
echo "Extracting main version from pyproject.toml"
|
||||
version=$(uv tree 2>/dev/null | grep -E '^flow(-nightly)?[[:space:]]' | cut -d' ' -f2 | sed 's/^v//')
|
||||
|
||||
# Verify nightly tag format
|
||||
if [[ ! "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.dev[0-9]+$ ]]; then
|
||||
echo "Main version format is incorrect. Must be in the format (e.g.) v1.5.1.dev36"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Using version: $version"
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Set nightly tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ steps.version.outputs.version }}"
|
||||
echo "docker_tags=ghcr.io/hanzoai/flow-nightly-all:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_tags=ghcr.io/hanzoai/flow-nightly-all:${version}-${{ matrix.arch }}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Build and push to Docker Hub
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
pull: true
|
||||
file: ./docker/build_and_push_with_extras.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.docker_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push to GitHub Container Registry
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
push: ${{ inputs.push_to_registry }}
|
||||
pull: true
|
||||
file: ./docker/build_and_push_with_extras.Dockerfile
|
||||
tags: ${{ steps.tags.outputs.ghcr_tags }}
|
||||
platforms: linux/${{ matrix.arch }}
|
||||
provenance: false
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
create-nightly-manifest:
|
||||
name: Create Multi-Arch Nightly Manifest
|
||||
needs: [build-nightly-base, build-nightly-main, build-nightly-main-all]
|
||||
runs-on: ubuntu-latest
|
||||
if: ${{ always() && inputs.push_to_registry && (needs.build-nightly-base.result == 'success' || needs.build-nightly-main.result == 'success' || needs.build-nightly-main-all.result == 'success') }}
|
||||
steps:
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
prune-cache: false
|
||||
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref }}
|
||||
|
||||
- name: Get version
|
||||
id: version
|
||||
run: |
|
||||
if [[ "${{ inputs.release_type }}" == "nightly-base" ]]; then
|
||||
version=$(uv tree 2>/dev/null | grep 'flow-base' | awk '{print $3}' | sed 's/^v//' | head -n 1)
|
||||
else
|
||||
version=$(uv tree 2>/dev/null | grep -E '^flow(-nightly)?[[:space:]]' | cut -d' ' -f2 | sed 's/^v//')
|
||||
fi
|
||||
echo "Using version: $version"
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Set nightly tags
|
||||
id: tags
|
||||
run: |
|
||||
version="${{ steps.version.outputs.version }}"
|
||||
case "${{ inputs.release_type }}" in
|
||||
"nightly-base")
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-nightly:base-${version},ghcr.io/hanzoai/flow-nightly:base-latest,ghcr.io/hanzoai/flow-nightly:base-${version},ghcr.io/hanzoai/flow-nightly:base-latest" >> $GITHUB_OUTPUT
|
||||
echo "arch_base=ghcr.io/hanzoai/flow-nightly:base-${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow-nightly:base-${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
"nightly-main")
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-nightly:${version},ghcr.io/hanzoai/flow-nightly:latest,ghcr.io/hanzoai/flow-nightly:${version},ghcr.io/hanzoai/flow-nightly:latest" >> $GITHUB_OUTPUT
|
||||
echo "arch_base=ghcr.io/hanzoai/flow-nightly:${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow-nightly:${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
"nightly-main-all")
|
||||
echo "final_tags=ghcr.io/hanzoai/flow-nightly-all:${version},ghcr.io/hanzoai/flow-nightly-all:latest,ghcr.io/hanzoai/flow-nightly-all:${version},ghcr.io/hanzoai/flow-nightly-all:latest" >> $GITHUB_OUTPUT
|
||||
echo "arch_base=ghcr.io/hanzoai/flow-nightly-all:${version}" >> $GITHUB_OUTPUT
|
||||
echo "ghcr_arch_base=ghcr.io/hanzoai/flow-nightly-all:${version}" >> $GITHUB_OUTPUT
|
||||
;;
|
||||
*)
|
||||
echo "Error: Invalid release_type: ${{ inputs.release_type }}" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.TEMP_GHCR_TOKEN}}
|
||||
|
||||
- name: Create and push multi-arch nightly manifests
|
||||
run: |
|
||||
# Split tags and create manifests for each
|
||||
IFS=',' read -ra TAGS <<< "${{ steps.tags.outputs.final_tags }}"
|
||||
for tag in "${TAGS[@]}"; do
|
||||
echo "Creating manifest for $tag"
|
||||
|
||||
# Determine architecture-specific tags
|
||||
if [[ "$tag" == *"flowai"* ]]; then
|
||||
amd64_tag="${{ steps.tags.outputs.arch_base }}-amd64"
|
||||
arm64_tag="${{ steps.tags.outputs.arch_base }}-arm64"
|
||||
else
|
||||
amd64_tag="${{ steps.tags.outputs.ghcr_arch_base }}-amd64"
|
||||
arm64_tag="${{ steps.tags.outputs.ghcr_arch_base }}-arm64"
|
||||
fi
|
||||
|
||||
docker buildx imagetools create \
|
||||
--tag "$tag" \
|
||||
"$amd64_tag" \
|
||||
"$arm64_tag"
|
||||
done
|
||||
@@ -1,13 +0,0 @@
|
||||
name: Build and Push Docker Image
|
||||
# Native deploy pipeline is .hanzo/workflows/deploy.yml — GitHub is a mirror.
|
||||
# This workflow is intentionally a no-op sync notice: image build + push +
|
||||
# operator deploy now happen in-cluster via Hanzo Git (act_runner + BuildKit).
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
notice:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Sync notice
|
||||
run: echo "native pipeline is .hanzo/workflows/deploy.yml; GitHub is a mirror"
|
||||
@@ -1,92 +0,0 @@
|
||||
name: Test Docker images
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
secrets:
|
||||
DOCKERHUB_USERNAME:
|
||||
required: true
|
||||
DOCKERHUB_TOKEN:
|
||||
required: true
|
||||
workflow_dispatch:
|
||||
|
||||
env:
|
||||
POETRY_VERSION: "1.8.2"
|
||||
|
||||
jobs:
|
||||
test-docker:
|
||||
runs-on: [self-hosted, linux, ARM64, flow-ai-arm64-40gb-ephemeral]
|
||||
name: Test docker images
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Set container runtime
|
||||
run: |
|
||||
if command -v docker &> /dev/null; then
|
||||
echo "CONTAINER_RUNTIME=docker" >> $GITHUB_ENV
|
||||
elif command -v podman &> /dev/null; then
|
||||
echo "CONTAINER_RUNTIME=podman" >> $GITHUB_ENV
|
||||
else
|
||||
echo "Error: Neither docker nor podman found"
|
||||
exit 1
|
||||
fi
|
||||
echo "Using container runtime: ${{ env.CONTAINER_RUNTIME }}"
|
||||
- name: Docker System Info and Cleanup
|
||||
run: |
|
||||
echo "=== Docker System Usage Before Cleanup ==="
|
||||
${{ env.CONTAINER_RUNTIME }} system df || true
|
||||
if command -v docker &> /dev/null; then
|
||||
docker buildx du || true
|
||||
fi
|
||||
|
||||
echo "=== Cleaning up Docker System ==="
|
||||
${{ env.CONTAINER_RUNTIME }} system prune -af --volumes || true
|
||||
if command -v docker &> /dev/null; then
|
||||
docker buildx prune -af || true
|
||||
fi
|
||||
|
||||
echo "=== Docker System Usage After Cleanup ==="
|
||||
${{ env.CONTAINER_RUNTIME }} system df || true
|
||||
if command -v docker &> /dev/null; then
|
||||
docker buildx du || true
|
||||
fi
|
||||
- name: Login to Docker Hub
|
||||
if: github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Build image
|
||||
run: |
|
||||
${{ env.CONTAINER_RUNTIME }} build -t ghcr.io/hanzoai/flow:latest-dev \
|
||||
-f docker/build_and_push.Dockerfile \
|
||||
.
|
||||
- name: Test image
|
||||
run: |
|
||||
expected_version=$(cat pyproject.toml | grep version | head -n 1 | cut -d '"' -f 2)
|
||||
version=$(${{ env.CONTAINER_RUNTIME }} run --rm --entrypoint bash ghcr.io/hanzoai/flow:latest-dev -c "python -c 'from flow.utils.version import get_version_info; print(get_version_info()[\"version\"])'")
|
||||
if [ "$expected_version" != "$version" ]; then
|
||||
echo "Expected version: $expected_version"
|
||||
echo "Actual version: $version"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Build backend image
|
||||
run: |
|
||||
${{ env.CONTAINER_RUNTIME }} build -t ghcr.io/hanzoai/flow-backend:latest-dev \
|
||||
--build-arg FLOW_IMAGE=ghcr.io/hanzoai/flow:latest-dev \
|
||||
-f docker/build_and_push_backend.Dockerfile \
|
||||
.
|
||||
- name: Test backend image
|
||||
run: |
|
||||
expected_version=$(cat src/backend/base/pyproject.toml | grep version | head -n 1 | cut -d '"' -f 2)
|
||||
version=$(${{ env.CONTAINER_RUNTIME }} run --rm --entrypoint bash ghcr.io/hanzoai/flow-backend:latest-dev -c "python -c 'from flow.utils.version import get_version_info; print(get_version_info()[\"version\"])'")
|
||||
if [ "$expected_version" != "$version" ]; then
|
||||
echo "Expected version: $expected_version"
|
||||
echo "Actual version: $version"
|
||||
exit 1
|
||||
fi
|
||||
- name: Build frontend image
|
||||
run: |
|
||||
${{ env.CONTAINER_RUNTIME }} build -t ghcr.io/hanzoai/flow-frontend:latest-dev \
|
||||
-f docker/frontend/build_and_push_frontend.Dockerfile \
|
||||
.
|
||||
@@ -1,38 +0,0 @@
|
||||
name: Test Docs Build
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
branch:
|
||||
description: "(Optional) Branch to checkout"
|
||||
required: false
|
||||
type: string
|
||||
|
||||
env:
|
||||
NODE_VERSION: "22"
|
||||
|
||||
jobs:
|
||||
test-docs-build:
|
||||
name: Test Docs Build
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.branch || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
id: setup-node
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: npm
|
||||
cache-dependency-path: ./docs/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
run: cd docs && npm install
|
||||
|
||||
- name: Build docs
|
||||
run: cd docs && npm run build
|
||||
@@ -1,107 +0,0 @@
|
||||
name: GP Download Translations
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 23 * * *" # 11 PM UTC — 1h before nightly build at midnight
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
download:
|
||||
runs-on: ubuntu-latest
|
||||
name: Download translations from Globalization Pipeline
|
||||
environment: GP-test
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- name: Resolve latest release branch
|
||||
id: resolve-branch
|
||||
run: |
|
||||
BRANCH=$(git ls-remote --heads https://github.com/${{ github.repository }} 'refs/heads/release-*' \
|
||||
| awk '{print $2}' \
|
||||
| sed 's|refs/heads/||' \
|
||||
| grep -E '^release-[0-9]+\.[0-9]+\.[0-9]+$' \
|
||||
| sort -V \
|
||||
| tail -n 1)
|
||||
if [ -z "$BRANCH" ]; then
|
||||
echo "No release-* branch found in ${{ github.repository }}"
|
||||
exit 1
|
||||
fi
|
||||
echo "branch=${BRANCH}" >> "$GITHUB_OUTPUT"
|
||||
echo "Using release branch: $BRANCH"
|
||||
|
||||
- name: Checkout release branch
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ steps.resolve-branch.outputs.branch }}
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install dependencies
|
||||
run: pip install -r scripts/gp/requirements.txt
|
||||
|
||||
- name: Download translations from GP
|
||||
working-directory: scripts/gp
|
||||
env:
|
||||
GP_ADMIN_USER_ID: ${{ secrets.GP_ADMIN_USER_ID }}
|
||||
GP_ADMIN_PASSWORD: ${{ secrets.GP_ADMIN_PASSWORD }}
|
||||
GP_INSTANCE: ${{ vars.GP_INSTANCE }}
|
||||
GP_BUNDLE: ${{ vars.GP_BUNDLE }}
|
||||
run: python download_translations.py --output ../../src/frontend/src/locales/
|
||||
|
||||
- name: Close stale translation PRs
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
script: |
|
||||
const { data: pulls } = await github.rest.pulls.list({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
state: 'open'
|
||||
});
|
||||
for (const pull of pulls) {
|
||||
if (pull.title === "chore: update translations from Globalization Pipeline") {
|
||||
await github.rest.pulls.update({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: pull.number,
|
||||
state: 'closed'
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
- name: Create Pull Request
|
||||
id: create-pr
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
branch-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
base: ${{ steps.resolve-branch.outputs.branch }}
|
||||
commit-message: "chore: update translations from Globalization Pipeline [skip ci]"
|
||||
title: "chore: update translations from Globalization Pipeline"
|
||||
body: |
|
||||
Automated PR to update locale files from IBM Globalization Pipeline.
|
||||
|
||||
This PR was automatically created by the `gp-download` workflow.
|
||||
If no files changed, this PR will be empty and can be closed.
|
||||
branch: translations/update-gp
|
||||
branch-suffix: timestamp
|
||||
delete-branch: true
|
||||
maintainer-can-modify: true
|
||||
add-paths: src/frontend/src/locales/*.json
|
||||
|
||||
- name: Enable auto-merge
|
||||
if: steps.create-pr.outputs.pull-request-number != ''
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
gh pr merge ${{ steps.create-pr.outputs.pull-request-number }} --auto --squash \
|
||||
--repo ${{ github.repository }}
|
||||
|
||||
- name: Report GP download failure
|
||||
if: failure()
|
||||
run: echo "::warning::GP translation download failed — translations may be out of date. Check the 'Download translations from GP' step above for details."
|
||||
@@ -1,27 +0,0 @@
|
||||
name: GP Scripts Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "scripts/gp/**"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
name: Test Globalization Pipeline scripts
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install dependencies
|
||||
run: pip install -r scripts/gp/requirements.txt pytest
|
||||
|
||||
- name: Run GP script tests
|
||||
run: python -m pytest scripts/gp/tests/ -v
|
||||
@@ -1,56 +0,0 @@
|
||||
name: GP Upload Strings
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- "release-*"
|
||||
paths:
|
||||
- "src/frontend/src/locales/en.json"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
upload:
|
||||
runs-on: ubuntu-latest
|
||||
name: Upload English strings to Globalization Pipeline
|
||||
environment: GP-test
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Check if this is the latest release branch
|
||||
id: check-branch
|
||||
run: |
|
||||
LATEST=$(git ls-remote --heads https://github.com/${{ github.repository }} 'refs/heads/release-*' \
|
||||
| awk '{print $2}' \
|
||||
| sed 's|refs/heads/||' \
|
||||
| grep -E '^release-[0-9]+\.[0-9]+\.[0-9]+$' \
|
||||
| sort -V \
|
||||
| tail -n 1)
|
||||
CURRENT="${{ github.ref_name }}"
|
||||
if [ "$CURRENT" != "$LATEST" ]; then
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
echo "::notice::Skipping upload — $CURRENT is not the latest release branch ($LATEST)"
|
||||
else
|
||||
echo "skip=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Set up Python
|
||||
if: steps.check-branch.outputs.skip != 'true'
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install dependencies
|
||||
if: steps.check-branch.outputs.skip != 'true'
|
||||
run: pip install -r scripts/gp/requirements.txt
|
||||
|
||||
- name: Upload strings to GP
|
||||
if: steps.check-branch.outputs.skip != 'true'
|
||||
working-directory: scripts/gp
|
||||
env:
|
||||
GP_ADMIN_USER_ID: ${{ secrets.GP_ADMIN_USER_ID }}
|
||||
GP_ADMIN_PASSWORD: ${{ secrets.GP_ADMIN_PASSWORD }}
|
||||
GP_INSTANCE: ${{ vars.GP_INSTANCE }}
|
||||
GP_BUNDLE: ${{ vars.GP_BUNDLE }}
|
||||
run: python upload_strings.py --source ../../src/frontend/src/locales/en.json
|
||||
@@ -1,56 +0,0 @@
|
||||
name: Integration Tests
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
ref:
|
||||
description: "(Optional) ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
workflow_call:
|
||||
inputs:
|
||||
python-versions:
|
||||
description: "(Optional) Python versions to test"
|
||||
required: true
|
||||
type: string
|
||||
default: "['3.10', '3.11', '3.12', '3.13', '3.14']"
|
||||
ref:
|
||||
description: "(Optional) ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
|
||||
env:
|
||||
POETRY_VERSION: "1.8.2"
|
||||
|
||||
jobs:
|
||||
integration-tests:
|
||||
name: Run Integration Tests
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
max-parallel: 1 # Currently, we can only run at a time for collection-per-db-constraints
|
||||
matrix:
|
||||
python-version:
|
||||
- "3.13"
|
||||
- "3.12"
|
||||
- "3.11"
|
||||
- "3.10"
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
|
||||
ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
- name: Run integration tests with api keys
|
||||
timeout-minutes: 20
|
||||
run: |
|
||||
make integration_tests_api_keys
|
||||
@@ -1,77 +0,0 @@
|
||||
name: Run Frontend Jest Unit Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
secrets:
|
||||
CODECOV_TOKEN:
|
||||
required: false
|
||||
inputs:
|
||||
ref:
|
||||
description: "(Optional) ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
ref:
|
||||
description: "(Optional) ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
|
||||
env:
|
||||
NODE_VERSION: "22"
|
||||
|
||||
jobs:
|
||||
jest-unit-tests:
|
||||
name: Frontend Jest Unit Tests
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
checks: write
|
||||
steps:
|
||||
- name: Checkout Repository
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Setup Node.js Environment
|
||||
uses: actions/setup-node@v6
|
||||
id: setup-node
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: "npm"
|
||||
cache-dependency-path: ./src/frontend/package-lock.json
|
||||
|
||||
- name: Run Frontend Unit Tests
|
||||
run: make test_frontend_ci
|
||||
|
||||
- name: Publish Jest Test Results
|
||||
uses: mikepenz/action-junit-report@v6
|
||||
if: always()
|
||||
with:
|
||||
report_paths: "src/frontend/test-results/junit.xml"
|
||||
check_name: "Frontend Unit Test Results"
|
||||
fail_on_failure: true
|
||||
require_tests: true
|
||||
|
||||
- name: Add Jest Coverage PR Comment
|
||||
uses: MishaKav/jest-coverage-comment@main
|
||||
if: github.event_name == 'pull_request'
|
||||
with:
|
||||
coverage-summary-path: src/frontend/coverage/jest/coverage-summary.json
|
||||
title: Frontend Unit Test Coverage Report
|
||||
summary-title: Coverage Summary
|
||||
badge-title: Coverage
|
||||
hide-comment: false
|
||||
create-new-comment: false
|
||||
hide-summary: false
|
||||
junitxml-title: Unit Test Results
|
||||
junitxml-path: src/frontend/test-results/junit.xml
|
||||
|
||||
- name: Upload Jest Coverage Artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: jest-coverage
|
||||
path: src/frontend/coverage/jest
|
||||
retention-days: 30
|
||||
@@ -1,46 +0,0 @@
|
||||
name: autofix.ci
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "src/frontend/**"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
env:
|
||||
NODE_VERSION: "22"
|
||||
jobs:
|
||||
autofix:
|
||||
if: ${{ github.actor != 'github-actions[bot]' }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
id: setup-node
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
- name: Cache Node.js dependencies
|
||||
uses: actions/cache@v5
|
||||
id: npm-cache
|
||||
with:
|
||||
path: ~/.npm
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('src/frontend/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-node-
|
||||
|
||||
- name: Install Node.js dependencies
|
||||
run: |
|
||||
cd src/frontend
|
||||
npm ci
|
||||
if: ${{ steps.setup-node.outputs.cache-hit != 'true' }}
|
||||
- name: Run Biome
|
||||
run: |
|
||||
cd src/frontend
|
||||
npm run format
|
||||
|
||||
- uses: autofix-ci/action@635ffb0c9798bd160680f18fd73371e355b85f27
|
||||
@@ -1,89 +0,0 @@
|
||||
name: Lint Frontend
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
branch:
|
||||
description: "(Optional) Branch to checkout"
|
||||
required: false
|
||||
type: string
|
||||
|
||||
|
||||
env:
|
||||
NODE_VERSION: "22"
|
||||
|
||||
jobs:
|
||||
run-linters:
|
||||
name: Run Biome
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.branch || github.ref }}
|
||||
# Need full history so biome can diff against the PR base
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
id: setup-node
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
- name: Cache Node.js dependencies
|
||||
uses: actions/cache@v5
|
||||
id: npm-cache
|
||||
with:
|
||||
path: ~/.npm
|
||||
key: ${{ runner.os }}-node-${{ hashFiles('src/frontend/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-node-
|
||||
|
||||
- name: Install Node.js dependencies
|
||||
run: |
|
||||
cd src/frontend
|
||||
npm install
|
||||
if: ${{ steps.setup-node.outputs.cache-hit != 'true' }}
|
||||
|
||||
- name: Run Biome on PR-changed files
|
||||
run: |
|
||||
# Determine the branch this PR targets (falls back to `main` for
|
||||
# workflow_dispatch / non-PR contexts).
|
||||
BASE_REF="${{ github.base_ref || 'main' }}"
|
||||
echo "=== Debug ==="
|
||||
echo "HEAD: $(git rev-parse HEAD)"
|
||||
echo "BASE_REF: $BASE_REF"
|
||||
echo "origin/$BASE_REF: $(git rev-parse "origin/$BASE_REF" 2>/dev/null || echo NOT_FOUND)"
|
||||
|
||||
# Merge-base = the commit where this branch forked off its target.
|
||||
# Diffing against it yields ONLY the files this PR introduced,
|
||||
# ignoring unrelated upstream changes merged into the base since.
|
||||
MERGE_BASE=$(git merge-base HEAD "origin/$BASE_REF")
|
||||
echo "MERGE_BASE: $MERGE_BASE"
|
||||
|
||||
# Collect frontend files modified in this PR. Filter to file types
|
||||
# biome can process (ts/tsx/js/jsx/json/jsonc and the like).
|
||||
CHANGED_FILES=$(git diff --name-only --diff-filter=ACMR \
|
||||
"$MERGE_BASE"...HEAD -- 'src/frontend/' \
|
||||
| grep -E '\.(tsx?|jsx?|c(js|ts)|m(js|ts)|jsonc?)$' \
|
||||
|| true)
|
||||
|
||||
if [ -z "$CHANGED_FILES" ]; then
|
||||
echo "No frontend files changed in this PR — skipping biome."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "=== Files to lint ==="
|
||||
echo "$CHANGED_FILES"
|
||||
|
||||
# Biome runs from src/frontend; strip that prefix from paths.
|
||||
RELATIVE_FILES=$(echo "$CHANGED_FILES" | sed 's|^src/frontend/||')
|
||||
|
||||
cd src/frontend
|
||||
echo "$RELATIVE_FILES" | xargs npx @biomejs/biome check \
|
||||
--files-ignore-unknown=true \
|
||||
--diagnostic-level=error
|
||||
@@ -1,47 +0,0 @@
|
||||
name: Lint Python
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
branch:
|
||||
description: "(Optional) Branch to checkout"
|
||||
required: false
|
||||
type: string
|
||||
env:
|
||||
POETRY_VERSION: "1.8.2"
|
||||
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
name: Run Mypy
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- "3.14"
|
||||
- "3.13"
|
||||
- "3.12"
|
||||
- "3.11"
|
||||
- "3.10"
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.branch || github.ref }}
|
||||
persist-credentials: true
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
- name: Run Mypy
|
||||
run: |
|
||||
uv run mypy --namespace-packages -p "flow"
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.github_token }}
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
{
|
||||
"problemMatcher": [
|
||||
{
|
||||
"owner": "ruff",
|
||||
"pattern": [
|
||||
{
|
||||
"regexp": "^(Would reformat): (.+)$",
|
||||
"message": 1,
|
||||
"file": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,56 +0,0 @@
|
||||
name: "Mend"
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["release-*", "main"]
|
||||
pull_request:
|
||||
branches: ["release-*", "main"]
|
||||
|
||||
concurrency:
|
||||
group: mend-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
NODE_VERSION: "22"
|
||||
|
||||
jobs:
|
||||
mend-scan:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Checkout Repository
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js Environment
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: "npm"
|
||||
cache-dependency-path: ./src/frontend/package-lock.json
|
||||
|
||||
- name: Install Frontend Dependencies
|
||||
working-directory: ./src/frontend
|
||||
run: npm ci
|
||||
|
||||
- name: Set up Java (Semeru)
|
||||
uses: actions/setup-java@v5
|
||||
with:
|
||||
distribution: "semeru"
|
||||
java-version: "21"
|
||||
|
||||
- name: Download Mend Unified Agent
|
||||
run: |
|
||||
curl --fail --show-error --retry 3 --location -o wss-unified-agent.jar https://unified-agent.s3.amazonaws.com/wss-unified-agent.jar
|
||||
|
||||
- name: Run Mend Scan
|
||||
#prevent forks from running mend scans
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
|
||||
env:
|
||||
WS_APIKEY: ${{ secrets.WS_APIKEY }}
|
||||
WS_WSS_URL: ${{ secrets.WS_WSS_URL }}
|
||||
WS_USERKEY: ${{ secrets.WS_USERKEY }}
|
||||
WS_PRODUCTNAME: ${{ secrets.WS_PRODUCTNAME }}
|
||||
WS_PROJECTNAME: ${{ secrets.WS_PROJECTNAME }}
|
||||
run: |
|
||||
java -jar wss-unified-agent.jar -d src -logLevel info
|
||||
@@ -1,206 +0,0 @@
|
||||
name: Database Migration Validation
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'src/backend/base/flow/alembic/**'
|
||||
- 'src/backend/base/flow/services/database/models/**'
|
||||
- 'src/backend/base/flow/services/database/service.py'
|
||||
- 'src/backend/tests/unit/alembic/**'
|
||||
- '.github/workflows/migration-validation.yml'
|
||||
|
||||
jobs:
|
||||
model-migration-consistency:
|
||||
name: Model/Migration Consistency
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16
|
||||
env:
|
||||
POSTGRES_USER: flow
|
||||
POSTGRES_PASSWORD: flow
|
||||
POSTGRES_DB: flow
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd="pg_isready -U flow"
|
||||
--health-interval=10s
|
||||
--health-timeout=5s
|
||||
--health-retries=5
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv sync --extra postgresql
|
||||
|
||||
- name: Check model/migration consistency
|
||||
env:
|
||||
MIGRATION_VALIDATION_CI: "true"
|
||||
FLOW_TEST_DATABASE_URI: "postgresql://flow:flow@localhost:5432/flow"
|
||||
run: |
|
||||
uv run pytest src/backend/tests/unit/alembic/test_migration_execution.py -x -v
|
||||
|
||||
validate-migration:
|
||||
name: Migration Pattern Validation
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Get changed migration files
|
||||
id: changed-files
|
||||
run: |
|
||||
set -euo pipefail
|
||||
CHANGED_FILES=$(git diff --name-only origin/main...HEAD | grep -E 'src/backend/base/flow/alembic/versions/.*\.py$' | grep -v 'test_migrations/' || echo "")
|
||||
|
||||
if [ -z "$CHANGED_FILES" ]; then
|
||||
echo "No migration files changed"
|
||||
echo "files=" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "Changed migration files:"
|
||||
echo "$CHANGED_FILES"
|
||||
echo "files=$(printf '%s' "$CHANGED_FILES" | tr '\n' ' ')" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Validate migration patterns
|
||||
if: steps.changed-files.outputs.files != ''
|
||||
env:
|
||||
MIGRATION_FILES: ${{ steps.changed-files.outputs.files }}
|
||||
run: |
|
||||
python src/backend/base/flow/alembic/migration_validator.py $MIGRATION_FILES
|
||||
|
||||
- name: Generate validation report
|
||||
if: always() && steps.changed-files.outputs.files != ''
|
||||
env:
|
||||
MIGRATION_FILES: ${{ steps.changed-files.outputs.files }}
|
||||
run: |
|
||||
python src/backend/base/flow/alembic/migration_validator.py \
|
||||
--json $MIGRATION_FILES > validation-report.json 2> validation-stderr.txt || true
|
||||
if [ ! -s validation-report.json ]; then
|
||||
echo "::error::Validator produced no output. Stderr:"
|
||||
cat validation-stderr.txt
|
||||
fi
|
||||
|
||||
- name: Post PR comment with results
|
||||
if: always() && steps.changed-files.outputs.files != ''
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
|
||||
let message = '';
|
||||
let validationPassed = true;
|
||||
|
||||
try {
|
||||
const report = JSON.parse(fs.readFileSync('validation-report.json', 'utf8'));
|
||||
|
||||
for (const result of report) {
|
||||
if (!result.valid) {
|
||||
validationPassed = false;
|
||||
}
|
||||
}
|
||||
|
||||
if (validationPassed) {
|
||||
message = `✅ **Migration Validation Passed**\n\n`;
|
||||
message += `All migrations follow the Expand-Contract pattern correctly.\n\n`;
|
||||
} else {
|
||||
message = `❌ **Migration Validation Failed**\n\n`;
|
||||
message += `Your migrations don't follow the Expand-Contract pattern.\n\n`;
|
||||
|
||||
for (const result of report) {
|
||||
if (!result.valid || result.warnings.length > 0) {
|
||||
message += `### File: \`${result.file.split('/').pop()}\`\n`;
|
||||
message += `**Phase:** ${result.phase}\n\n`;
|
||||
|
||||
if (result.violations && result.violations.length > 0) {
|
||||
message += `**Violations:**\n`;
|
||||
for (const v of result.violations) {
|
||||
message += `- Line ${v.line}: ${v.message}\n`;
|
||||
}
|
||||
message += `\n`;
|
||||
}
|
||||
|
||||
if (result.warnings && result.warnings.length > 0) {
|
||||
message += `**Warnings:**\n`;
|
||||
for (const w of result.warnings) {
|
||||
message += `- Line ${w.line}: ${w.message}\n`;
|
||||
}
|
||||
message += `\n`;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
message += `### Resources\n`;
|
||||
message += `- Review the [DB Migration Guide](./src/backend/base/flow/alembic/DB-MIGRATION-GUIDE.MD)\n`;
|
||||
message += `- Use \`python scripts/generate_migration.py --help\` to generate compliant migrations\n\n`;
|
||||
|
||||
message += `### Common Issues & Solutions\n`;
|
||||
message += `- **New columns must be nullable:** Add \`nullable=True\` or \`server_default\`\n`;
|
||||
message += `- **Missing phase marker:** Add \`Phase: EXPAND/MIGRATE/CONTRACT\` to docstring\n`;
|
||||
message += `- **Column drops:** Only allowed in CONTRACT phase\n`;
|
||||
message += `- **Direct renames:** Use expand-contract pattern instead\n`;
|
||||
}
|
||||
} catch (error) {
|
||||
message = `⚠️ **Migration validation check failed to run properly**\n`;
|
||||
message += `Error: ${error.message}\n`;
|
||||
validationPassed = false;
|
||||
}
|
||||
|
||||
// Post or update comment (non-critical — don't let API errors mask validation results)
|
||||
try {
|
||||
const { data: comments } = await github.rest.issues.listComments({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.issue.number,
|
||||
});
|
||||
|
||||
const botComment = comments.find(comment =>
|
||||
comment.user.type === 'Bot' &&
|
||||
comment.body.includes('Migration Validation')
|
||||
);
|
||||
|
||||
if (botComment) {
|
||||
await github.rest.issues.updateComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: botComment.id,
|
||||
body: message
|
||||
});
|
||||
} else {
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.issue.number,
|
||||
body: message
|
||||
});
|
||||
}
|
||||
} catch (apiError) {
|
||||
core.warning(`Failed to post PR comment: ${apiError.message}`);
|
||||
}
|
||||
|
||||
// Fail the workflow if validation didn't pass
|
||||
if (!validationPassed) {
|
||||
core.setFailed('Migration validation failed');
|
||||
}
|
||||
@@ -1,591 +0,0 @@
|
||||
name: Nightly Build
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
runs_on:
|
||||
description: "Runner to use for tests (use self-hosted for safe/release code)"
|
||||
required: false
|
||||
type: choice
|
||||
options:
|
||||
- ubuntu-latest
|
||||
- self-hosted
|
||||
- '["self-hosted", "linux", "ARM64", "flow-ai-arm64-40gb-ephemeral"]'
|
||||
default: ubuntu-latest
|
||||
skip_frontend_tests:
|
||||
description: "Skip frontend tests. Only do this for testing purposes."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
skip_backend_tests:
|
||||
description: "Skip backend tests. Only do this for testing purposes."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
skip_slack:
|
||||
description: "Skip slack message. Only do this for testing purposes."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
push_to_registry:
|
||||
description: "Whether to push images to registries. Set to false for testing builds without publishing."
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
auto_merge_hash_history:
|
||||
description: "Create PR to merge hash history back to main. Set to false when not running from main."
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
schedule:
|
||||
# Run job at 00:00 UTC (4:00 PM PST / 5:00 PM PDT)
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
env:
|
||||
POETRY_VERSION: "1.8.3"
|
||||
PYTHON_VERSION: "3.13"
|
||||
|
||||
jobs:
|
||||
validate-inputs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Validate inputs
|
||||
if: inputs.push_to_registry && (inputs.skip_frontend_tests || inputs.skip_backend_tests)
|
||||
run: |
|
||||
echo "Cannot skip tests while push_to_registry is true."
|
||||
exit 1
|
||||
|
||||
create-nightly-tag:
|
||||
if: github.repository == 'hanzoai/flow'
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
shell: bash -ex -o pipefail {0}
|
||||
permissions:
|
||||
# Required to create tag
|
||||
contents: write
|
||||
outputs:
|
||||
main_tag: ${{ steps.generate_main_tag.outputs.main_tag }}
|
||||
base_tag: ${{ steps.set_base_tag.outputs.base_tag }}
|
||||
lfx_tag: ${{ steps.generate_lfx_tag.outputs.lfx_tag }}
|
||||
sdk_tag: ${{ steps.generate_sdk_tag.outputs.sdk_tag }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: true
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
|
||||
- name: Generate main nightly tag
|
||||
id: generate_main_tag
|
||||
run: |
|
||||
# flow-nightly pins flow-base-nightly[complete]==<exact dev>, so both tags
|
||||
# MUST be identical. Compute them in ONE invocation (single PyPI snapshot) so they
|
||||
# cannot drift across separate calls.
|
||||
# NOTE: This outputs the tag with the `v` prefix.
|
||||
MAIN_TAG="$(uv run ./scripts/ci/pypi_nightly_tag.py main)"
|
||||
echo "main_tag=$MAIN_TAG" >> $GITHUB_OUTPUT
|
||||
echo "main_tag=$MAIN_TAG"
|
||||
|
||||
- name: Delete existing tag if it exists
|
||||
id: check_main_tag
|
||||
run: |
|
||||
git fetch --tags
|
||||
git tag -d ${{ steps.generate_main_tag.outputs.main_tag }} || true
|
||||
git push --delete origin ${{ steps.generate_main_tag.outputs.main_tag }} || true
|
||||
echo "main_tag_exists=false" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Generate LFX nightly tag
|
||||
id: generate_lfx_tag
|
||||
run: |
|
||||
# NOTE: This outputs the tag with the `v` prefix.
|
||||
LFX_TAG="$(uv run ./scripts/ci/lfx_nightly_tag.py)"
|
||||
echo "lfx_tag=$LFX_TAG" >> $GITHUB_OUTPUT
|
||||
echo "lfx_tag=$LFX_TAG"
|
||||
|
||||
- name: Generate SDK nightly tag
|
||||
id: generate_sdk_tag
|
||||
run: |
|
||||
# NOTE: This outputs the tag with the `v` prefix.
|
||||
SDK_TAG="$(uv run ./scripts/ci/sdk_nightly_tag.py)"
|
||||
echo "sdk_tag=$SDK_TAG" >> $GITHUB_OUTPUT
|
||||
echo "sdk_tag=$SDK_TAG"
|
||||
|
||||
- name: Commit tag
|
||||
id: commit_tag
|
||||
run: |
|
||||
# If the main tag does not exist in GH, we create the base tag from the existing codebase.
|
||||
|
||||
git config --global user.email "[email protected]"
|
||||
git config --global user.name "Flow Bot"
|
||||
|
||||
MAIN_TAG="${{ steps.generate_main_tag.outputs.main_tag }}"
|
||||
BASE_TAG="${{ steps.generate_base_tag.outputs.base_tag }}"
|
||||
LFX_TAG="${{ steps.generate_lfx_tag.outputs.lfx_tag }}"
|
||||
SDK_TAG="${{ steps.generate_sdk_tag.outputs.sdk_tag }}"
|
||||
echo "Updating SDK project version to $SDK_TAG"
|
||||
uv run --no-sync ./scripts/ci/update_sdk_version.py $SDK_TAG
|
||||
echo "Updating LFX project version to $LFX_TAG"
|
||||
uv run ./scripts/ci/update_lfx_version.py $LFX_TAG
|
||||
echo "Updating base project version to $BASE_TAG and updating main project version to $MAIN_TAG"
|
||||
uv run --no-sync ./scripts/ci/update_pyproject_combined.py main $MAIN_TAG $BASE_TAG $LFX_TAG
|
||||
|
||||
uv lock
|
||||
cd src/lfx && uv lock && cd ../..
|
||||
|
||||
git add pyproject.toml src/backend/base/pyproject.toml src/lfx/pyproject.toml src/sdk/pyproject.toml uv.lock
|
||||
# update_lfx_version.py re-pins each bundle's `lfx` dep to
|
||||
# lfx-nightly==<dev>, so any modified bundle pyprojects need to
|
||||
# ride the same commit/tag as the rest of the version bumps.
|
||||
if compgen -G "src/bundles/*/pyproject.toml" > /dev/null; then
|
||||
git add src/bundles/*/pyproject.toml
|
||||
fi
|
||||
git commit -m "Update version and project name"
|
||||
|
||||
echo "Tagging main with $MAIN_TAG"
|
||||
if ! git tag -a $MAIN_TAG -m "Flow nightly $MAIN_TAG"; then
|
||||
echo "Tag creation failed. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Pushing main tag $MAIN_TAG"
|
||||
if ! git push origin $MAIN_TAG; then
|
||||
echo "Tag push failed. Check if the tag already exists. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
# TODO: notify on failure
|
||||
|
||||
- name: Checkout main nightly tag
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ steps.generate_main_tag.outputs.main_tag }}
|
||||
|
||||
- name: Retrieve Base Tag
|
||||
id: retrieve_base_tag
|
||||
working-directory: src/backend/base
|
||||
run: |
|
||||
# If the main tag already exists, we need to retrieve the base version from the main tag codebase.
|
||||
version=$(uv tree | grep 'flow-base' | awk '{print $3}' | head -n 1)
|
||||
echo "base_tag=$version" >> $GITHUB_OUTPUT
|
||||
echo "base_tag=$version"
|
||||
|
||||
- name: Set Base Tag
|
||||
id: set_base_tag
|
||||
run: |
|
||||
if [ "${{ steps.retrieve_base_tag.conclusion }}" != "skipped" ] && [ "${{ steps.retrieve_base_tag.outputs.base_tag }}" ]; then
|
||||
BASE_TAG="${{ steps.retrieve_base_tag.outputs.base_tag }}"
|
||||
echo "base_tag=$BASE_TAG" >> $GITHUB_OUTPUT
|
||||
echo "base_tag=$BASE_TAG"
|
||||
elif [ "${{ steps.commit_tag.conclusion }}" != "skipped" ] && [ "${{ steps.generate_shared_tag.outputs.base_tag }}" ]; then
|
||||
BASE_TAG="${{ steps.generate_shared_tag.outputs.base_tag }}"
|
||||
echo "base_tag=$BASE_TAG" >> $GITHUB_OUTPUT
|
||||
echo "base_tag=$BASE_TAG"
|
||||
else
|
||||
echo "No base tag found. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
build-hash-history:
|
||||
if: github.repository == 'hanzoai/flow'
|
||||
name: Build Nightly Hash History
|
||||
needs: create-nightly-tag
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
shell: bash -ex -o pipefail {0}
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- name: Checkout nightly tag
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ needs.create-nightly-tag.outputs.main_tag }}
|
||||
persist-credentials: true
|
||||
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
|
||||
- name: Force reinstall LFX to pick up version change
|
||||
run: |
|
||||
echo "Force reinstalling lfx-nightly to ensure version metadata is updated..."
|
||||
uv pip install --reinstall --no-deps src/lfx
|
||||
|
||||
- name: Verify LFX version
|
||||
run: |
|
||||
echo "Checking installed LFX version..."
|
||||
uv run python -c "from importlib.metadata import version; print(f'lfx-nightly version: {version(\"lfx-nightly\")}')" || \
|
||||
uv run python -c "from importlib.metadata import version; print(f'lfx version: {version(\"lfx\")}')"
|
||||
|
||||
- name: Build and validate nightly hash history
|
||||
run: |
|
||||
# The script includes append-only validation
|
||||
uv run python scripts/build_hash_history.py --nightly
|
||||
|
||||
- name: Commit and push hash history changes
|
||||
run: |
|
||||
echo "=== Configuring git ==="
|
||||
git config --global user.email "[email protected]"
|
||||
git config --global user.name "Flow Bot"
|
||||
|
||||
echo "=== Checking for hash history changes ==="
|
||||
# Check if there are changes to commit (handles if it's a new file for first run)
|
||||
git add src/lfx/src/lfx/_assets/nightly_hash_history.json
|
||||
if git diff --cached --quiet src/lfx/src/lfx/_assets/nightly_hash_history.json; then
|
||||
echo "No changes to nightly hash history"
|
||||
else
|
||||
echo "Hash history changes detected, committing..."
|
||||
# Commit to the nightly tag itself so builds include the hash history
|
||||
git commit -m "Update nightly hash history for ${{ needs.create-nightly-tag.outputs.main_tag }}"
|
||||
|
||||
echo "=== Updating nightly tag ${{ needs.create-nightly-tag.outputs.main_tag }} ==="
|
||||
# Update the tag to include the hash history
|
||||
git tag -f ${{ needs.create-nightly-tag.outputs.main_tag }}
|
||||
git push -f origin ${{ needs.create-nightly-tag.outputs.main_tag }}
|
||||
|
||||
echo "=== Creating clean branch from main with only hash history changes ==="
|
||||
# Fetch latest main
|
||||
git fetch origin main
|
||||
|
||||
# Create a new branch from main
|
||||
git checkout -B nightly-hash-history-updates origin/main
|
||||
|
||||
# Copy ONLY the hash history file from the nightly tag
|
||||
git checkout ${{ needs.create-nightly-tag.outputs.main_tag }} -- src/lfx/src/lfx/_assets/nightly_hash_history.json
|
||||
|
||||
# Commit only this file
|
||||
git add src/lfx/src/lfx/_assets/nightly_hash_history.json
|
||||
git commit -m "Update nightly hash history for ${{ needs.create-nightly-tag.outputs.main_tag }}"
|
||||
|
||||
# Force push the clean branch
|
||||
git push origin nightly-hash-history-updates --force
|
||||
echo "Updated nightly tag and created clean branch with only hash history changes"
|
||||
fi
|
||||
|
||||
frontend-tests:
|
||||
if: github.repository == 'hanzoai/flow' && !inputs.skip_frontend_tests
|
||||
name: Run Frontend Tests
|
||||
needs: create-nightly-tag
|
||||
uses: ./.github/workflows/typescript_test.yml
|
||||
with:
|
||||
tests_folder: "tests"
|
||||
release: true
|
||||
runs-on: ${{ inputs['runs_on'] || github.event.inputs['runs_on'] || 'ubuntu-latest' }}
|
||||
secrets:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
STORE_API_KEY: ${{ secrets.STORE_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }}
|
||||
|
||||
backend-unit-tests:
|
||||
if: github.repository == 'hanzoai/flow' && !inputs.skip_backend_tests
|
||||
name: Run Backend Unit Tests
|
||||
needs: create-nightly-tag
|
||||
uses: ./.github/workflows/python_test.yml
|
||||
with:
|
||||
python-versions: '["3.10", "3.11", "3.12", "3.13", "3.14"]'
|
||||
runs-on: ${{ inputs['runs_on'] || github.event.inputs['runs_on'] || 'ubuntu-latest' }}
|
||||
secrets:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
|
||||
# Not making nightly builds dependent on integration test success
|
||||
# due to inherent flakiness of 3rd party integrations
|
||||
# Revisit when https://github.com/hanzoai/flow/pull/3607 is merged.
|
||||
# backend-integration-tests:
|
||||
# name: Run Backend Integration Tests
|
||||
# needs: create-nightly-tag
|
||||
# uses: ./.github/workflows/integration_tests.yml
|
||||
# with:
|
||||
# python-versions: '["3.10", "3.11", "3.12", "3.13"]'
|
||||
# ref: ${{ needs.create-nightly-tag.outputs.tag }}
|
||||
|
||||
release-nightly-build:
|
||||
if: github.repository == 'hanzoai/flow' && always() && needs.frontend-tests.result != 'failure' && needs.backend-unit-tests.result != 'failure' && needs.build-hash-history.result != 'failure'
|
||||
name: Run Nightly Flow Build
|
||||
needs: [create-nightly-tag, frontend-tests, backend-unit-tests, build-hash-history]
|
||||
uses: ./.github/workflows/release_nightly.yml
|
||||
with:
|
||||
build_docker_base: true
|
||||
build_docker_main: true
|
||||
build_docker_ep: true
|
||||
build_lfx: true
|
||||
nightly_tag_main: ${{ needs.create-nightly-tag.outputs.main_tag }}
|
||||
nightly_tag_base: ${{ needs.create-nightly-tag.outputs.base_tag }}
|
||||
nightly_tag_lfx: ${{ needs.create-nightly-tag.outputs.lfx_tag }}
|
||||
nightly_tag_sdk: ${{ needs.create-nightly-tag.outputs.sdk_tag }}
|
||||
# When triggered by schedule, inputs.push_to_registry is not set, so default to true
|
||||
# When triggered manually, use the provided value (default is also true)
|
||||
push_to_registry: ${{ github.event_name == 'schedule' || inputs.push_to_registry != false }}
|
||||
secrets: inherit
|
||||
db-migration-validation:
|
||||
name: Run DB Migration Validation
|
||||
needs: [create-nightly-tag, release-nightly-build]
|
||||
if: ${{ github.repository == 'hanzoai/flow' && always() && needs.release-nightly-build.result == 'success' }}
|
||||
uses: ./.github/workflows/db-migration-validation.yml
|
||||
with:
|
||||
# Pass Docker image reference for docker-compose and version without 'v' prefix for pip
|
||||
# release_tag format: v1.10.0.dev20260522
|
||||
# We need: flowai/flow-nightly:1.10.0.dev20260522 for Docker
|
||||
nightly_tag: flowai/flow-nightly:${{ needs.create-nightly-tag.outputs.release_tag }}
|
||||
secrets: inherit
|
||||
|
||||
slack-notification:
|
||||
name: Send Slack Notification
|
||||
needs: [frontend-tests, backend-unit-tests, release-nightly-build, merge-hash-history-to-main]
|
||||
if: ${{ github.repository == 'hanzoai/flow' && !inputs.skip_slack && always() && (needs.release-nightly-build.result == 'failure' || needs.frontend-tests.result == 'failure' || needs.backend-unit-tests.result == 'failure' || needs.merge-hash-history-to-main.result == 'failure' || needs.release-nightly-build.result == 'success') }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Send failure notification to Slack
|
||||
if: ${{ needs.release-nightly-build.result == 'failure' || needs.frontend-tests.result == 'failure' || needs.backend-unit-tests.result == 'failure' || needs.merge-hash-history-to-main.result == 'failure' }}
|
||||
run: |
|
||||
# Determine which job failed
|
||||
FAILED_JOB="unknown"
|
||||
if [ "${{ needs.release-nightly-build.result }}" == "failure" ]; then
|
||||
FAILED_JOB="release-nightly-build"
|
||||
elif [ "${{ needs.frontend-tests.result }}" == "failure" ]; then
|
||||
FAILED_JOB="frontend-tests"
|
||||
elif [ "${{ needs.backend-unit-tests.result }}" == "failure" ]; then
|
||||
FAILED_JOB="backend-unit-tests"
|
||||
elif [ "${{ needs.merge-hash-history-to-main.result }}" == "failure" ]; then
|
||||
FAILED_JOB="merge-hash-history-to-main"
|
||||
fi
|
||||
|
||||
curl -X POST -H 'Content-type: application/json' \
|
||||
--data "{
|
||||
\"blocks\": [
|
||||
{
|
||||
\"type\": \"section\",
|
||||
\"text\": {
|
||||
\"type\": \"mrkdwn\",
|
||||
\"text\": \"❌ *Nightly Build Failed*\"
|
||||
}
|
||||
},
|
||||
{
|
||||
\"type\": \"section\",
|
||||
\"fields\": [
|
||||
{
|
||||
\"type\": \"mrkdwn\",
|
||||
\"text\": \"*Failed Job:*\\n$FAILED_JOB\"
|
||||
},
|
||||
{
|
||||
\"type\": \"mrkdwn\",
|
||||
\"text\": \"*Status:*\\nfailure\"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
\"type\": \"context\",
|
||||
\"elements\": [
|
||||
{
|
||||
\"type\": \"mrkdwn\",
|
||||
\"text\": \"<https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View full logs on GitHub>\"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}" ${{ secrets.FLOW_ENG_SLACK_WEBHOOK_URL }}
|
||||
|
||||
merge-hash-history-to-main:
|
||||
name: Create PR to Merge Hash History to Main
|
||||
needs: [create-nightly-tag, build-hash-history, release-nightly-build]
|
||||
# Run if auto_merge_hash_history is true (default) or not set (scheduled runs)
|
||||
# When triggered by schedule, inputs.auto_merge_hash_history is not set, so default to true
|
||||
# Only run when the base branch is 'main' to prevent PRs from feature branches
|
||||
if: github.repository == 'hanzoai/flow' && github.ref == 'refs/heads/main' && always() && needs.build-hash-history.result == 'success' && needs.release-nightly-build.result == 'success' && (github.event_name == 'schedule' || inputs.auto_merge_hash_history != false)
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Checkout main branch
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: main
|
||||
fetch-depth: 0
|
||||
persist-credentials: true
|
||||
|
||||
- name: Configure Git
|
||||
run: |
|
||||
git config --global user.email "[email protected]"
|
||||
git config --global user.name "Flow Bot"
|
||||
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Fetch nightly hash history branch
|
||||
run: |
|
||||
echo "=== Fetching nightly-hash-history-updates branch ==="
|
||||
git fetch origin nightly-hash-history-updates || true
|
||||
echo "Fetch completed"
|
||||
|
||||
- name: Check if branch exists and has changes
|
||||
id: check_branch
|
||||
run: |
|
||||
echo "=== Checking branch status ==="
|
||||
if git rev-parse --verify origin/nightly-hash-history-updates >/dev/null 2>&1; then
|
||||
echo "branch_exists=true" >> $GITHUB_OUTPUT
|
||||
echo "Branch exists"
|
||||
|
||||
# Check if there are differences between main and the nightly branch
|
||||
echo "=== Comparing with main branch ==="
|
||||
if git diff --quiet main origin/nightly-hash-history-updates -- src/lfx/src/lfx/_assets/nightly_hash_history.json; then
|
||||
echo "has_changes=false" >> $GITHUB_OUTPUT
|
||||
echo "No changes to merge"
|
||||
else
|
||||
echo "has_changes=true" >> $GITHUB_OUTPUT
|
||||
echo "Changes detected - PR will be created/updated"
|
||||
fi
|
||||
else
|
||||
echo "branch_exists=false" >> $GITHUB_OUTPUT
|
||||
echo "has_changes=false" >> $GITHUB_OUTPUT
|
||||
echo "ERROR: Branch nightly-hash-history-updates does not exist!"
|
||||
echo "This should have been created by the build-hash-history job."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Validate and prepare PR
|
||||
if: steps.check_branch.outputs.branch_exists == 'true' && steps.check_branch.outputs.has_changes == 'true'
|
||||
run: |
|
||||
echo "=== Checking out hash history file from nightly branch ==="
|
||||
# Checkout the nightly hash history file to verify it
|
||||
git checkout origin/nightly-hash-history-updates -- src/lfx/src/lfx/_assets/nightly_hash_history.json
|
||||
|
||||
echo "=== Verifying only hash history file was modified ==="
|
||||
# Verify that ONLY the nightly_hash_history.json file was modified
|
||||
CHANGED_FILES=$(git diff --name-only main)
|
||||
if [ "$CHANGED_FILES" != "src/lfx/src/lfx/_assets/nightly_hash_history.json" ]; then
|
||||
echo "ERROR: Unexpected files were modified: $CHANGED_FILES"
|
||||
echo "Only nightly_hash_history.json should be changed"
|
||||
exit 1
|
||||
fi
|
||||
echo "Only hash history file modified"
|
||||
|
||||
echo "=== Validating hash history file ==="
|
||||
# Basic validation - ensure file exists and is valid JSON
|
||||
|
||||
if [ ! -f "src/lfx/src/lfx/_assets/nightly_hash_history.json" ]; then
|
||||
echo "ERROR: nightly_hash_history.json file was deleted!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! uv run python -c "import json; json.load(open('src/lfx/src/lfx/_assets/nightly_hash_history.json'))"; then
|
||||
echo "ERROR: nightly_hash_history.json is not valid JSON!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Hash history file validation passed"
|
||||
echo "Note: Append-only validation was performed during the build step"
|
||||
|
||||
- name: Create Pull Request
|
||||
if: steps.check_branch.outputs.branch_exists == 'true' && steps.check_branch.outputs.has_changes == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PR_BODY: >-
|
||||
This PR updates the nightly hash history file with the latest component hashes from the nightly build.
|
||||
|
||||
|
||||
**Automated PR** - Generated by nightly build workflow
|
||||
|
||||
|
||||
- Updates: `src/lfx/src/lfx/_assets/nightly_hash_history.json`
|
||||
|
||||
- Source: Nightly build tag ${{ needs.create-nightly-tag.outputs.main_tag || 'latest nightly' }}
|
||||
|
||||
|
||||
This PR is automatically updated each night with the latest hash history.
|
||||
|
||||
Auto-merge is enabled - PR will merge automatically when checks pass.
|
||||
run: |
|
||||
echo "=== Creating or updating PR ==="
|
||||
|
||||
# Check if PR already exists
|
||||
EXISTING_PR=$(gh pr list --head nightly-hash-history-updates --base main --json number --jq '.[0].number' 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$EXISTING_PR" ]; then
|
||||
echo "PR #$EXISTING_PR already exists, updating..."
|
||||
gh pr edit "$EXISTING_PR" \
|
||||
--title "chore: update nightly hash history" \
|
||||
--body "$PR_BODY"
|
||||
|
||||
echo "PR updated successfully"
|
||||
PR_NUMBER="$EXISTING_PR"
|
||||
else
|
||||
echo "Creating new PR..."
|
||||
PR_URL=$(gh pr create \
|
||||
--title "chore: update nightly hash history" \
|
||||
--body "$PR_BODY" \
|
||||
--base main \
|
||||
--head nightly-hash-history-updates)
|
||||
|
||||
echo "PR created: $PR_URL"
|
||||
# Extract PR number from URL (works with both formats)
|
||||
PR_NUMBER=$(echo "$PR_URL" | sed 's/.*\/pull\///' | sed 's/[^0-9].*//')
|
||||
fi
|
||||
|
||||
# Enable auto-merge on the PR
|
||||
echo "=== Enabling auto-merge ==="
|
||||
if gh pr merge "$PR_NUMBER" --auto --squash; then
|
||||
echo "Auto-merge enabled successfully"
|
||||
else
|
||||
echo "ERROR: Could not enable auto-merge on PR"
|
||||
# TODO: Fail the job here (once you've got everything working on happy path)
|
||||
fi
|
||||
|
||||
- name: Send success notification to Slack
|
||||
if: ${{ !inputs.skip_slack && needs.release-nightly-build.result == 'success' && needs.db-migration-validation.result != 'failure' }}
|
||||
run: |
|
||||
curl -X POST -H 'Content-type: application/json' \
|
||||
--data '{
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": "✅ *Nightly Build Successful*"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "section",
|
||||
"fields": [
|
||||
{
|
||||
"type": "mrkdwn",
|
||||
"text": "*Job:*\nrelease-nightly-build"
|
||||
},
|
||||
{
|
||||
"type": "mrkdwn",
|
||||
"text": "*Status:*\n`${{ needs.release-nightly-build.result }}`"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "context",
|
||||
"elements": [
|
||||
{
|
||||
"type": "mrkdwn",
|
||||
"text": "<https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View full logs on GitHub>"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}' ${{ secrets.FLOW_ENG_SLACK_WEBHOOK_URL }}
|
||||
@@ -1,105 +0,0 @@
|
||||
name: autofix.ci
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "**/*.py"
|
||||
- "src/lfx/src/lfx/components/**"
|
||||
- "scripts/build_component_index.py"
|
||||
types: [opened, synchronize, reopened, labeled]
|
||||
env:
|
||||
PYTHON_VERSION: "3.13"
|
||||
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
name: Run Ruff Check and Format
|
||||
if: github.event.action != 'labeled' && github.actor != 'github-actions[bot]'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
- run: uv run ruff check --fix-only .
|
||||
- run: uv run ruff format . --config pyproject.toml
|
||||
- uses: autofix-ci/action@635ffb0c9798bd160680f18fd73371e355b85f27
|
||||
|
||||
|
||||
update-starter-projects:
|
||||
name: Update Starter Projects
|
||||
if: github.event.action != 'labeled' && github.actor != 'github-actions[bot]'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: "Install dependencies"
|
||||
run: |
|
||||
uv sync
|
||||
uv pip install -e .
|
||||
|
||||
- name: Run starter projects update
|
||||
run: uv run python scripts/ci/update_starter_projects.py
|
||||
|
||||
- uses: autofix-ci/action@635ffb0c9798bd160680f18fd73371e355b85f27
|
||||
|
||||
|
||||
update-component-index:
|
||||
name: Update Component Index
|
||||
if: >-
|
||||
github.actor != 'github-actions[bot]'
|
||||
&& (github.event.action != 'labeled' || github.event.label.name == 'fix-index')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
repository: ${{ github.event.pull_request.head.repo.full_name }}
|
||||
ref: ${{ github.event.pull_request.head.ref }}
|
||||
fetch-depth: 0
|
||||
- name: Merge base branch
|
||||
run: |
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
||||
git merge origin/${{ github.base_ref }} --no-edit || {
|
||||
# Check which files have conflicts
|
||||
CONFLICTED=$(git diff --name-only --diff-filter=U)
|
||||
|
||||
# If any non-generated files conflict, bail out — needs manual resolution
|
||||
if echo "$CONFLICTED" | grep -v component_index.json | grep -q .; then
|
||||
echo "::error::Merge conflicts in non-generated files require manual resolution"
|
||||
git merge --abort
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Only component_index.json conflicts — resolve and finish merge
|
||||
git checkout --theirs src/lfx/src/lfx/_assets/component_index.json
|
||||
git add src/lfx/src/lfx/_assets/component_index.json
|
||||
GIT_EDITOR=true git merge --continue
|
||||
}
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: "Install dependencies"
|
||||
run: uv sync --dev --project .
|
||||
|
||||
- name: Build component index
|
||||
env:
|
||||
LFX_DEV: "1"
|
||||
run: make build_component_index
|
||||
|
||||
- uses: autofix-ci/action@635ffb0c9798bd160680f18fd73371e355b85f27
|
||||
@@ -1,255 +0,0 @@
|
||||
name: Python tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
secrets:
|
||||
OPENAI_API_KEY:
|
||||
required: true
|
||||
ANTHROPIC_API_KEY:
|
||||
required: true
|
||||
CODECOV_TOKEN:
|
||||
required: false
|
||||
inputs:
|
||||
python-versions:
|
||||
description: "(Optional) Python versions to test"
|
||||
required: true
|
||||
type: string
|
||||
default: "['3.10', '3.11', '3.12', '3.13', '3.14']"
|
||||
ref:
|
||||
description: "(Optional) ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
nightly:
|
||||
description: "Whether run is from the nightly build"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
runs-on:
|
||||
description: "Runner to use for the tests"
|
||||
required: false
|
||||
type: string
|
||||
default: "ubuntu-latest"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python-versions:
|
||||
description: "(Optional) Python versions to test"
|
||||
required: true
|
||||
type: string
|
||||
default: "['3.10', '3.11', '3.12', '3.13', '3.14']"
|
||||
runs-on:
|
||||
description: "Runner to use for the tests"
|
||||
required: false
|
||||
type: choice
|
||||
options:
|
||||
- ubuntu-latest
|
||||
- self-hosted
|
||||
- '["self-hosted", "linux", "ARM64", "flow-ai-arm64-40gb-ephemeral"]'
|
||||
default: ubuntu-latest
|
||||
env:
|
||||
POETRY_VERSION: "1.8.2"
|
||||
NODE_VERSION: "22"
|
||||
PYTEST_RUN_PATH: "src/backend/tests"
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
|
||||
jobs:
|
||||
build:
|
||||
name: Unit Tests - Python ${{ matrix.python-version }} - Group ${{ matrix.group }}
|
||||
runs-on: ${{ (inputs['runs-on'] && startsWith(format('{0}', inputs['runs-on']), '[') && fromJSON(inputs['runs-on'])) || inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ${{ fromJson(inputs.python-versions || '["3.10", "3.11", "3.12", "3.13", "3.14"]') }}
|
||||
splitCount: [5]
|
||||
group: [1, 2, 3, 4, 5]
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
id: setup-node
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
|
||||
- name: Generate dynamic coverage configuration
|
||||
run: |
|
||||
echo "Generating dynamic coverage configuration..."
|
||||
python3 scripts/generate_coverage_config.py
|
||||
echo "Generated .coveragerc with the following exclusions:"
|
||||
echo "Bundled components and legacy files excluded from coverage"
|
||||
|
||||
- name: Run unit tests
|
||||
uses: nick-fields/retry@v3
|
||||
with:
|
||||
timeout_minutes: 40
|
||||
max_attempts: 2
|
||||
command: make unit_tests args="-x -vv --splits ${{ matrix.splitCount }} --group ${{ matrix.group }} --reruns 5 --cov --cov-config=src/backend/.coveragerc --cov-report=xml --cov-report=html"
|
||||
|
||||
- name: Upload coverage to Codecov
|
||||
uses: codecov/codecov-action@v5
|
||||
if: matrix.python-version == '3.10'
|
||||
with:
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
files: ./coverage.xml
|
||||
flags: backend
|
||||
name: backend-coverage-group-${{ matrix.group }}
|
||||
fail_ci_if_error: false
|
||||
directory: ./
|
||||
|
||||
- name: Upload coverage artifacts
|
||||
uses: actions/upload-artifact@v6
|
||||
if: matrix.python-version == '3.10'
|
||||
with:
|
||||
name: backend-coverage-report-group-${{ matrix.group }}
|
||||
path: |
|
||||
coverage.xml
|
||||
htmlcov/
|
||||
retention-days: 30
|
||||
|
||||
integration-tests:
|
||||
name: Integration Tests - Python ${{ matrix.python-version }}
|
||||
runs-on: ${{ (inputs['runs-on'] && startsWith(format('{0}', inputs['runs-on']), '[') && fromJSON(inputs['runs-on'])) || inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ${{ fromJson(inputs.python-versions || '["3.10", "3.11", "3.12", "3.13", "3.14"]') }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
- name: Run integration tests
|
||||
run: make integration_tests_no_api_keys
|
||||
env:
|
||||
PYLEAK_LOG_LEVEL: debug # enable pyleak logging
|
||||
DO_NOT_TRACK: true # disable telemetry reporting
|
||||
|
||||
lfx-tests:
|
||||
name: LFX Tests - Python ${{ matrix.python-version }}
|
||||
runs-on: ${{ (inputs['runs-on'] && startsWith(format('{0}', inputs['runs-on']), '[') && fromJSON(inputs['runs-on'])) || inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ${{ fromJson(inputs.python-versions || '["3.10", "3.11", "3.12", "3.13", "3.14"]') }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
- name: Run lfx tests
|
||||
run: make lfx_tests
|
||||
env:
|
||||
DO_NOT_TRACK: true # disable telemetry reporting
|
||||
|
||||
- name: Upload coverage to Codecov
|
||||
uses: codecov/codecov-action@v5
|
||||
if: matrix.python-version == '3.10'
|
||||
with:
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
files: ./src/lfx/coverage.xml
|
||||
flags: lfx
|
||||
name: lfx-coverage
|
||||
fail_ci_if_error: false
|
||||
directory: ./src/lfx/
|
||||
|
||||
- name: Upload coverage artifacts
|
||||
uses: actions/upload-artifact@v6
|
||||
if: matrix.python-version == '3.10'
|
||||
with:
|
||||
name: lfx-coverage-report
|
||||
path: |
|
||||
src/lfx/coverage.xml
|
||||
src/lfx/htmlcov/
|
||||
retention-days: 30
|
||||
|
||||
test-cli:
|
||||
name: Test CLI - Python ${{ matrix.python-version }}
|
||||
runs-on: ${{ (inputs['runs-on'] && startsWith(format('{0}', inputs['runs-on']), '[') && fromJSON(inputs['runs-on'])) || inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ${{ fromJson(inputs.python-versions || '["3.10", "3.11", "3.12", "3.13", "3.14"]') }}
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Check Version
|
||||
id: check-version
|
||||
# We need to print $3 because flow-base is a dependency of flow
|
||||
# For langlow we'd use print $2
|
||||
run: |
|
||||
version=$(uv tree | grep 'flow-base' | awk '{print $3}' | sed 's/^v//' | head -n 1)
|
||||
url="https://pypi.org/pypi/flow-base/json"
|
||||
if [ ${{ inputs.nightly }} == true ]; then
|
||||
url="https://pypi.org/pypi/flow-base-nightly/json"
|
||||
fi
|
||||
|
||||
last_released_version=$(curl -s $url | jq -r '.releases | keys | .[]' | sort -V | tail -n 1)
|
||||
if [ "$version" != "$last_released_version" ]; then
|
||||
echo "Version $version has not been released yet. Skipping the rest of the job."
|
||||
echo skipped=true >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
else
|
||||
echo version=$version >> $GITHUB_OUTPUT
|
||||
echo skipped=false >> $GITHUB_OUTPUT
|
||||
fi
|
||||
- name: Build wheel
|
||||
if: steps.check-version.outputs.skipped == 'false'
|
||||
run: |
|
||||
make build main=true
|
||||
- name: Install wheel and Test CLI
|
||||
if: steps.check-version.outputs.skipped == 'false'
|
||||
run: |
|
||||
uv venv new-venv
|
||||
source new-venv/bin/activate
|
||||
uv pip install dist/*.whl
|
||||
- name: Test CLI
|
||||
if: steps.check-version.outputs.skipped == 'false'
|
||||
run: |
|
||||
source new-venv/bin/activate
|
||||
python -m flow run --host localhost --port 7860 --backend-only &
|
||||
SERVER_PID=$!
|
||||
# Wait for the server to start
|
||||
timeout 120 bash -c 'until curl -f http://localhost:7860/api/v1/auto_login; do sleep 5; done' || (echo "Server did not start in time" && kill $SERVER_PID && exit 1)
|
||||
# Terminate the server
|
||||
kill $SERVER_PID || (echo "Failed to terminate the server" && exit 1)
|
||||
sleep 20 # give the server some time to terminate
|
||||
# Check if the server is still running
|
||||
if kill -0 $SERVER_PID 2>/dev/null; then
|
||||
echo "Failed to terminate the server"
|
||||
exit 0
|
||||
else
|
||||
echo "Server terminated successfully"
|
||||
fi
|
||||
@@ -1,403 +0,0 @@
|
||||
name: LFX Release
|
||||
run-name: LFX Release ${{ github.event.inputs.version || 'dev' }} by @${{ github.actor }}
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version:
|
||||
description: "Version to release (e.g., 0.1.0)"
|
||||
required: true
|
||||
type: string
|
||||
publish_pypi:
|
||||
description: "Publish to PyPI"
|
||||
required: true
|
||||
type: boolean
|
||||
default: true
|
||||
build_docker:
|
||||
description: "Build and publish Docker images"
|
||||
required: true
|
||||
type: boolean
|
||||
default: true
|
||||
pre_release:
|
||||
description: "Mark as pre-release"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
create_github_release:
|
||||
description: "Create GitHub release"
|
||||
required: true
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: "3.13"
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
validate-version:
|
||||
name: Validate Version
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
should_release: ${{ steps.check.outputs.should_release }}
|
||||
current_version: ${{ steps.check.outputs.current_version }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Check version
|
||||
id: check
|
||||
run: |
|
||||
cd src/lfx
|
||||
# Use uv tree to get package info, consistent with nightly workflow
|
||||
name=$(uv tree | grep 'lfx' | head -n 1 | awk '{print $1}')
|
||||
version=$(uv tree | grep 'lfx' | head -n 1 | awk '{print $2}')
|
||||
|
||||
# Strip leading 'v' if present
|
||||
version=$(echo $version | sed 's/^v//')
|
||||
echo "current_version=$version" >> $GITHUB_OUTPUT
|
||||
|
||||
if [ "$version" != "${{ github.event.inputs.version }}" ]; then
|
||||
echo "❌ Version mismatch: package has $version but input is ${{ github.event.inputs.version }}"
|
||||
echo "Please update the version in pyproject.toml first"
|
||||
echo "should_release=false" >> $GITHUB_OUTPUT
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if version already exists on PyPI
|
||||
if curl -s "https://pypi.org/pypi/lfx/json" | jq -r '.releases | keys[]' | grep -q "^${{ github.event.inputs.version }}$"; then
|
||||
echo "❌ Version ${{ github.event.inputs.version }} already exists on PyPI"
|
||||
echo "should_release=false" >> $GITHUB_OUTPUT
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Version ${{ github.event.inputs.version }} is valid and not yet released"
|
||||
echo "should_release=true" >> $GITHUB_OUTPUT
|
||||
|
||||
run-tests:
|
||||
name: Run Tests
|
||||
needs: validate-version
|
||||
if: needs.validate-version.outputs.should_release == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Run LFX tests
|
||||
run: |
|
||||
cd src/lfx
|
||||
make test
|
||||
|
||||
- name: Test CLI installation
|
||||
run: |
|
||||
cd src/lfx
|
||||
uv pip install .
|
||||
uv run lfx --help
|
||||
uv run lfx run --help
|
||||
uv run lfx serve --help
|
||||
|
||||
release-lfx:
|
||||
name: Build and Release LFX
|
||||
needs: [validate-version, run-tests]
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.check-version.outputs.version }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Install LFX dependencies
|
||||
run: uv sync --dev --package lfx
|
||||
|
||||
- name: Verify Version
|
||||
id: check-version
|
||||
run: |
|
||||
cd src/lfx
|
||||
# Use uv tree to get package info, consistent with nightly workflow
|
||||
name=$(uv tree | grep 'lfx' | head -n 1 | awk '{print $1}')
|
||||
version=$(uv tree | grep 'lfx' | head -n 1 | awk '{print $2}')
|
||||
|
||||
# Verify package name
|
||||
if [ "$name" != "lfx" ]; then
|
||||
echo "Package name $name does not match lfx. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Strip leading 'v' if present
|
||||
version=$(echo $version | sed 's/^v//')
|
||||
|
||||
# Verify version matches input
|
||||
if [ "$version" != "${{ github.event.inputs.version }}" ]; then
|
||||
echo "Version $version does not match input ${{ github.event.inputs.version }}. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "version=$version" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Build distribution
|
||||
run: |
|
||||
cd src/lfx
|
||||
rm -rf dist/
|
||||
uv build --wheel --out-dir dist
|
||||
|
||||
- name: Check build artifacts
|
||||
run: |
|
||||
cd src/lfx
|
||||
ls -la dist/
|
||||
# Verify wheel contents
|
||||
unzip -l dist/*.whl | grep -E "(lfx/__main__.py|lfx/cli/run.py|lfx/cli/commands.py)"
|
||||
|
||||
- name: Test installation from wheel
|
||||
run: |
|
||||
cd src/lfx
|
||||
uv pip install dist/*.whl --force-reinstall
|
||||
uv run lfx --help
|
||||
echo "LFX CLI test completed successfully"
|
||||
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: lfx-dist
|
||||
path: src/lfx/dist/
|
||||
retention-days: 5
|
||||
|
||||
- name: Publish to PyPI
|
||||
if: github.event.inputs.publish_pypi == 'true'
|
||||
env:
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
cd src/lfx
|
||||
uv publish dist/*.whl
|
||||
|
||||
build-docker:
|
||||
name: Build Docker Images
|
||||
needs: [validate-version, run-tests]
|
||||
if: github.event.inputs.build_docker == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
variant: [production, alpine]
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@v3
|
||||
|
||||
- name: Docker System Info and Cleanup
|
||||
run: |
|
||||
echo "=== Docker System Usage Before Cleanup ==="
|
||||
docker system df || true
|
||||
docker buildx du || true
|
||||
|
||||
echo "=== Cleaning up Docker System ==="
|
||||
docker system prune -af --volumes || true
|
||||
docker buildx prune -af || true
|
||||
|
||||
echo "=== Docker System Usage After Cleanup ==="
|
||||
docker system df || true
|
||||
docker buildx du || true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Prepare Docker metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: |
|
||||
flowai/lfx
|
||||
ghcr.io/flow-ai/lfx
|
||||
tags: |
|
||||
type=raw,value=${{ github.event.inputs.version }}${{ matrix.variant == 'alpine' && '-alpine' || '' }}
|
||||
type=raw,value=latest${{ matrix.variant == 'alpine' && '-alpine' || '' }},enable=${{ github.event.inputs.pre_release == 'false' }}
|
||||
labels: |
|
||||
org.opencontainers.image.title=LFX
|
||||
org.opencontainers.image.description=Flow Executor - CLI tool for running Flow AI workflows
|
||||
org.opencontainers.image.vendor=Flow
|
||||
org.opencontainers.image.version=${{ github.event.inputs.version }}
|
||||
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: src/lfx/docker/Dockerfile${{ matrix.variant == 'alpine' && '.alpine' || '' }}
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
build-args: |
|
||||
LFX_VERSION=${{ github.event.inputs.version }}
|
||||
|
||||
create-release:
|
||||
name: Create GitHub Release
|
||||
needs: [release-lfx, build-docker]
|
||||
if: always() && github.event.inputs.create_github_release == 'true' && needs.release-lfx.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Download artifacts
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: lfx-dist
|
||||
path: dist/
|
||||
|
||||
- name: Generate release notes
|
||||
id: notes
|
||||
run: |
|
||||
cat > release_notes.md << EOF
|
||||
# LFX ${{ github.event.inputs.version }}
|
||||
|
||||
## 🚀 Installation
|
||||
|
||||
### PyPI
|
||||
\`\`\`bash
|
||||
pip install lfx==${{ github.event.inputs.version }}
|
||||
# or
|
||||
uv pip install lfx==${{ github.event.inputs.version }}
|
||||
# or run without installing
|
||||
uvx lfx@${{ github.event.inputs.version }} --help
|
||||
\`\`\`
|
||||
|
||||
### Docker
|
||||
\`\`\`bash
|
||||
# Standard image
|
||||
docker pull flowai/lfx:${{ github.event.inputs.version }}
|
||||
|
||||
# Alpine image (smaller)
|
||||
docker pull flowai/lfx:${{ github.event.inputs.version }}-alpine
|
||||
|
||||
# Run a flow
|
||||
docker run --rm -v \$(pwd):/app/data flowai/lfx:${{ github.event.inputs.version }} lfx run flow.json --input-value "Hello"
|
||||
\`\`\`
|
||||
|
||||
## 📦 What's New
|
||||
|
||||
<!-- Add release notes here -->
|
||||
|
||||
## 📋 Checksums
|
||||
|
||||
\`\`\`
|
||||
$(cd dist && sha256sum *)
|
||||
\`\`\`
|
||||
|
||||
---
|
||||
|
||||
**Full Changelog**: https://github.com/${{ github.repository }}/compare/v${{ needs.validate-version.outputs.current_version }}...lfx-v${{ github.event.inputs.version }}
|
||||
EOF
|
||||
|
||||
- name: Create Release
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
tag_name: lfx-v${{ github.event.inputs.version }}
|
||||
name: LFX ${{ github.event.inputs.version }}
|
||||
body_path: release_notes.md
|
||||
draft: false
|
||||
prerelease: ${{ github.event.inputs.pre_release }}
|
||||
files: |
|
||||
dist/*
|
||||
generate_release_notes: true
|
||||
|
||||
test-release:
|
||||
name: Test Release
|
||||
needs: [release-lfx, build-docker]
|
||||
if: always() && (needs.release-lfx.result == 'success' || needs.build-docker.result == 'success')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Wait for PyPI propagation
|
||||
if: needs.release-lfx.result == 'success'
|
||||
run: sleep 60
|
||||
|
||||
- name: Test PyPI installation
|
||||
if: needs.release-lfx.result == 'success'
|
||||
run: |
|
||||
# Test installation using uv
|
||||
uv pip install lfx==${{ github.event.inputs.version }}
|
||||
uv run lfx --help
|
||||
|
||||
- name: Test Docker image
|
||||
if: needs.build-docker.result == 'success'
|
||||
run: |
|
||||
# Test standard image
|
||||
docker run --rm flowai/lfx:${{ github.event.inputs.version }} lfx --help
|
||||
|
||||
# Test alpine image
|
||||
docker run --rm flowai/lfx:${{ github.event.inputs.version }}-alpine lfx --help
|
||||
|
||||
# Test with a simple flow
|
||||
cat > test_flow.json << 'EOF'
|
||||
{
|
||||
"nodes": [],
|
||||
"edges": []
|
||||
}
|
||||
EOF
|
||||
|
||||
docker run --rm -v $(pwd):/app/data flowai/lfx:${{ github.event.inputs.version }} \
|
||||
lfx run /app/data/test_flow.json --input-value "test" || true
|
||||
|
||||
notify:
|
||||
name: Notify Release Status
|
||||
needs: [create-release, test-release]
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Notify success
|
||||
if: needs.create-release.result == 'success'
|
||||
run: |
|
||||
echo "✅ LFX ${{ github.event.inputs.version }} released successfully!"
|
||||
echo "PyPI: https://pypi.org/project/lfx/${{ github.event.inputs.version }}/"
|
||||
echo "Docker Hub: https://hub.docker.com/r/flowai/lfx/tags"
|
||||
echo "GitHub Release: https://github.com/${{ github.repository }}/releases/tag/lfx-v${{ github.event.inputs.version }}"
|
||||
|
||||
- name: Notify failure
|
||||
if: needs.create-release.result != 'success'
|
||||
run: |
|
||||
echo "❌ LFX ${{ github.event.inputs.version }} release failed!"
|
||||
exit 1
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,223 +0,0 @@
|
||||
name: Release Bundles
|
||||
|
||||
# Manually-triggered workflow for publishing stable `lfx-<name>` bundle wheels
|
||||
# from src/bundles/* to PyPI. Bundles change infrequently, so we release them
|
||||
# on demand rather than on a schedule. Re-running the workflow without a
|
||||
# version bump is a no-op (PyPI rejects the duplicate upload and we tolerate
|
||||
# the conflict).
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_tag:
|
||||
description: "Git ref to build from (branch, tag, or SHA). Defaults to the workflow's checked-out ref."
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
dry_run:
|
||||
description: "Build and test only; skip the PyPI publish step."
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: "3.13"
|
||||
|
||||
jobs:
|
||||
build-bundles:
|
||||
name: Build Flow Bundles
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
outputs:
|
||||
bundles-found: ${{ steps.build.outputs.bundles-found }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.release_tag || github.ref }}
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
- name: Build bundle wheels
|
||||
id: build
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
bundles=(src/bundles/*/pyproject.toml)
|
||||
if [ ${#bundles[@]} -eq 0 ]; then
|
||||
echo "No bundles found under src/bundles/*/"
|
||||
echo "bundles-found=0" >> $GITHUB_OUTPUT
|
||||
exit 0
|
||||
fi
|
||||
mkdir -p bundles-dist
|
||||
BUNDLES_DIST="$PWD/bundles-dist"
|
||||
for bundle_pyproject in "${bundles[@]}"; do
|
||||
bundle_dir=$(dirname "$bundle_pyproject")
|
||||
echo "Building wheel for $bundle_dir"
|
||||
(cd "$bundle_dir" && uv build --wheel --out-dir "$BUNDLES_DIST")
|
||||
done
|
||||
echo "bundles-found=1" >> $GITHUB_OUTPUT
|
||||
ls -la bundles-dist/
|
||||
# Bundles pin lfx (e.g. lfx>=0.5.0,<0.6.0). The smoke test below
|
||||
# installs the bundles in a fresh venv, so it needs an lfx wheel that
|
||||
# satisfies the pin available locally — PyPI may not yet have a
|
||||
# matching lfx published when bundles are released ahead of a main
|
||||
# release. We build the lfx wheel from the current ref so the smoke
|
||||
# test resolves against it; this artifact is not published.
|
||||
- name: Build lfx wheel for smoke test
|
||||
if: steps.build.outputs.bundles-found == '1'
|
||||
run: |
|
||||
mkdir -p lfx-dist
|
||||
LFX_DIST="$PWD/lfx-dist"
|
||||
(cd src/lfx && uv build --wheel --out-dir "$LFX_DIST")
|
||||
ls -la lfx-dist/
|
||||
- name: Upload bundles artifact
|
||||
if: steps.build.outputs.bundles-found == '1'
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: dist-bundles
|
||||
path: bundles-dist
|
||||
- name: Upload lfx artifact (smoke-test only; not published)
|
||||
if: steps.build.outputs.bundles-found == '1'
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: dist-lfx-smoketest
|
||||
path: lfx-dist
|
||||
|
||||
test-install:
|
||||
name: Install Smoke Test - ${{ matrix.os }} ${{ matrix.python-version }}
|
||||
needs: [build-bundles]
|
||||
if: needs.build-bundles.outputs.bundles-found == '1'
|
||||
runs-on: ${{ matrix.runner }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- os: linux
|
||||
runner: ubuntu-latest
|
||||
python-version: "3.10"
|
||||
- os: linux
|
||||
runner: ubuntu-latest
|
||||
python-version: "3.12"
|
||||
- os: macos
|
||||
runner: macos-latest
|
||||
python-version: "3.12"
|
||||
- os: windows
|
||||
runner: windows-latest
|
||||
python-version: "3.12"
|
||||
steps:
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
- name: Setup UV
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
ignore-empty-workdir: true
|
||||
- name: Download bundles artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-bundles
|
||||
path: ./bundles-dist
|
||||
- name: Download lfx artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-lfx-smoketest
|
||||
path: ./lfx-dist
|
||||
- name: Create fresh virtual environment
|
||||
run: uv venv test-env --seed
|
||||
shell: bash
|
||||
- name: Install lfx + bundle wheels (Unix)
|
||||
if: matrix.os != 'windows'
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
BUNDLE_WHEELS=(./bundles-dist/*.whl)
|
||||
LFX_WHEELS=(./lfx-dist/*.whl)
|
||||
if [ ${#BUNDLE_WHEELS[@]} -eq 0 ]; then
|
||||
echo "No bundle wheels found in ./bundles-dist/"
|
||||
exit 1
|
||||
fi
|
||||
if [ ${#LFX_WHEELS[@]} -eq 0 ]; then
|
||||
echo "No lfx wheel found in ./lfx-dist/"
|
||||
exit 1
|
||||
fi
|
||||
printf 'Installing: %s\n' "${LFX_WHEELS[@]}" "${BUNDLE_WHEELS[@]}"
|
||||
uv pip install --prerelease=allow --python ./test-env/bin/python "${LFX_WHEELS[@]}" "${BUNDLE_WHEELS[@]}"
|
||||
shell: bash
|
||||
- name: Install lfx + bundle wheels (Windows)
|
||||
if: matrix.os == 'windows'
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
BUNDLE_WHEELS=(./bundles-dist/*.whl)
|
||||
LFX_WHEELS=(./lfx-dist/*.whl)
|
||||
if [ ${#BUNDLE_WHEELS[@]} -eq 0 ]; then
|
||||
echo "No bundle wheels found in ./bundles-dist/"
|
||||
exit 1
|
||||
fi
|
||||
if [ ${#LFX_WHEELS[@]} -eq 0 ]; then
|
||||
echo "No lfx wheel found in ./lfx-dist/"
|
||||
exit 1
|
||||
fi
|
||||
printf 'Installing: %s\n' "${LFX_WHEELS[@]}" "${BUNDLE_WHEELS[@]}"
|
||||
uv pip install --prerelease=allow --python ./test-env/Scripts/python.exe "${LFX_WHEELS[@]}" "${BUNDLE_WHEELS[@]}"
|
||||
shell: bash
|
||||
|
||||
publish-bundles:
|
||||
name: Publish Bundles to PyPI
|
||||
needs: [build-bundles, test-install]
|
||||
if: |
|
||||
always() &&
|
||||
!cancelled() &&
|
||||
!inputs.dry_run &&
|
||||
needs.build-bundles.result == 'success' &&
|
||||
needs.build-bundles.outputs.bundles-found == '1' &&
|
||||
needs.test-install.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Download bundles artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-bundles
|
||||
path: bundles-dist
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
- name: Publish bundles to PyPI
|
||||
env:
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
wheels=(bundles-dist/*.whl)
|
||||
if [ ${#wheels[@]} -eq 0 ]; then
|
||||
echo "No bundle wheels to publish."
|
||||
exit 0
|
||||
fi
|
||||
failed=0
|
||||
for wheel in "${wheels[@]}"; do
|
||||
echo "Publishing $wheel"
|
||||
# Capture stderr so we can tolerate "already exists" without failing
|
||||
# the whole run — re-running this workflow without bumping a bundle
|
||||
# version should be a no-op for that bundle.
|
||||
if ! err=$(uv publish "$wheel" 2>&1); then
|
||||
echo "$err"
|
||||
if echo "$err" | grep -qiE 'already exists|file already exists|HTTP 400|duplicate'; then
|
||||
echo "Skipping $wheel: already published."
|
||||
else
|
||||
echo "Publish failed for $wheel"
|
||||
failed=1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
if [ "$failed" = "1" ]; then
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,599 +0,0 @@
|
||||
name: Flow Nightly Build
|
||||
run-name: Flow Nightly Release by @${{ github.actor }}
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
build_docker_base:
|
||||
description: "Build Docker Image for Flow Nightly Base"
|
||||
required: true
|
||||
type: boolean
|
||||
default: false
|
||||
build_docker_main:
|
||||
description: "Build Docker Image for Flow Nightly"
|
||||
required: true
|
||||
type: boolean
|
||||
default: false
|
||||
build_docker_ep:
|
||||
description: "Build Docker Image for Flow Nightly with Entrypoint"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
build_lfx:
|
||||
description: "Build and release LFX package"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
nightly_tag_main:
|
||||
description: "Tag for the nightly main build"
|
||||
required: true
|
||||
type: string
|
||||
nightly_tag_base:
|
||||
description: "Tag for the nightly base build"
|
||||
required: true
|
||||
type: string
|
||||
nightly_tag_lfx:
|
||||
description: "Tag for the nightly LFX build"
|
||||
required: false
|
||||
type: string
|
||||
nightly_tag_sdk:
|
||||
description: "Tag for the nightly SDK build"
|
||||
required: false
|
||||
type: string
|
||||
push_to_registry:
|
||||
description: "Whether to push images to registries. Set to false for testing builds without publishing."
|
||||
required: true
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
env:
|
||||
POETRY_VERSION: "1.8.3"
|
||||
PYTHON_VERSION: "3.13"
|
||||
|
||||
jobs:
|
||||
build-nightly-lfx:
|
||||
name: Build LFX Nightly
|
||||
if: ${{ inputs.build_lfx }}
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.verify.outputs.version }}
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
persist-credentials: true
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
- name: Install LFX dependencies
|
||||
run: cd src/lfx && uv sync
|
||||
|
||||
- name: Verify SDK Nightly Name and Version
|
||||
if: ${{ inputs.build_lfx }}
|
||||
run: |
|
||||
name=$(grep '^name = "' src/sdk/pyproject.toml | head -n 1 | sed 's/.*"\(.*\)".*/\1/')
|
||||
version=$(grep '^version = "' src/sdk/pyproject.toml | head -n 1 | sed 's/.*"\(.*\)".*/\1/')
|
||||
if [ "$name" != "flow-sdk-nightly" ]; then
|
||||
echo "Name $name does not match flow-sdk-nightly. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
if [ "v$version" != "${{ inputs.nightly_tag_sdk }}" ]; then
|
||||
echo "Version v$version does not match nightly tag ${{ inputs.nightly_tag_sdk }}. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Verify Nightly Name and Version
|
||||
id: verify
|
||||
run: |
|
||||
cd src/lfx
|
||||
name=$(uv tree | grep 'lfx' | head -n 1 | awk '{print $1}')
|
||||
version=$(uv tree | grep 'lfx' | head -n 1 | awk '{print $2}')
|
||||
if [ "$name" != "lfx-nightly" ]; then
|
||||
echo "Name $name does not match lfx-nightly. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
if [ "$version" != "${{ inputs.nightly_tag_lfx }}" ]; then
|
||||
echo "Version $version does not match nightly tag ${{ inputs.nightly_tag_lfx }}. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
# Strip the leading `v` from the version
|
||||
version=$(echo $version | sed 's/^v//')
|
||||
echo "version=$version" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Build flow-sdk for distribution
|
||||
run: |
|
||||
make sdk_build args="--wheel"
|
||||
|
||||
- name: Build LFX for distribution
|
||||
run: |
|
||||
cd src/lfx
|
||||
rm -rf dist/
|
||||
uv build --wheel --out-dir dist
|
||||
|
||||
- name: Test LFX CLI
|
||||
run: |
|
||||
cd src/lfx
|
||||
# Use a clean venv and install both wheels together so uv can resolve
|
||||
# the local SDK dependency without falling back to PyPI.
|
||||
uv venv test-env --seed
|
||||
uv pip install --python ./test-env/bin/python --force-reinstall ../sdk/dist/*.whl dist/*.whl
|
||||
./test-env/bin/lfx --help
|
||||
echo "LFX CLI test completed successfully"
|
||||
|
||||
# PyPI publishing moved to after cross-platform testing
|
||||
|
||||
- name: Upload flow-sdk Artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: dist-nightly-sdk
|
||||
path: src/sdk/dist
|
||||
|
||||
- name: Upload LFX Artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: dist-nightly-lfx
|
||||
path: src/lfx/dist
|
||||
|
||||
build-nightly-base:
|
||||
name: Build Flow Nightly Base
|
||||
needs: [build-nightly-lfx]
|
||||
if: ${{ always() && (needs.build-nightly-lfx.result == 'success' || inputs.build_lfx == false) }}
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
outputs:
|
||||
version: ${{ steps.verify.outputs.version }}
|
||||
skipped: ${{ steps.verify.outputs.skipped }}
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
persist-credentials: true
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Download LFX artifact
|
||||
if: inputs.build_lfx
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-lfx
|
||||
path: lfx-dist
|
||||
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
|
||||
- name: Force reinstall LFX from built wheel
|
||||
if: inputs.build_lfx
|
||||
run: |
|
||||
echo "Installing LFX from built wheel to test the actual distribution package..."
|
||||
# While workspace resolution installs from source, we want to test the exact
|
||||
# wheel that will be published to PyPI to catch any packaging issues
|
||||
uv pip install --force-reinstall --no-deps lfx-dist/*.whl
|
||||
|
||||
- name: Verify Nightly Name and Version
|
||||
id: verify
|
||||
run: |
|
||||
name=$(uv tree | grep 'flow-base' | awk '{print $2}' | head -n 1)
|
||||
version=$(uv tree | grep 'flow-base' | awk '{print $3}' | head -n 1)
|
||||
# Strip extras from package name (e.g., "flow-base-nightly[complete]" -> "flow-base-nightly")
|
||||
name_without_extras=$(echo $name | sed 's/\[.*\]//')
|
||||
if [ "$name_without_extras" != "flow-base-nightly" ]; then
|
||||
echo "Name $name_without_extras does not match flow-base-nightly. Exiting the workflow."
|
||||
echo "skipped=true" >> $GITHUB_OUTPUT
|
||||
exit 1
|
||||
fi
|
||||
if [ "$version" != "${{ inputs.nightly_tag_base }}" ]; then
|
||||
echo "Version $version does not match nightly tag ${{ inputs.nightly_tag_base }}. Exiting the workflow."
|
||||
echo "skipped=true" >> $GITHUB_OUTPUT
|
||||
exit 1
|
||||
fi
|
||||
# Strip the leading `v` from the version
|
||||
version=$(echo $version | sed 's/^v//')
|
||||
echo "version=$version" >> $GITHUB_OUTPUT
|
||||
echo "skipped=false" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Build Flow Base for distribution
|
||||
run: |
|
||||
rm -rf src/backend/base/dist
|
||||
rm -rf dist
|
||||
make build base=true args="--no-sources --wheel"
|
||||
|
||||
- name: Test Flow Base CLI
|
||||
run: |
|
||||
# TODO: Unsure why the whl is not built in src/backend/base/dist
|
||||
mkdir src/backend/base/dist
|
||||
mv dist/*.whl src/backend/base/dist/
|
||||
# Install with [complete] extra to test all optional dependencies
|
||||
WHEEL_FILE=$(ls src/backend/base/dist/*.whl)
|
||||
uv pip install "${WHEEL_FILE}[complete]"
|
||||
uv run python -m flow run --host localhost --port 7860 --backend-only &
|
||||
SERVER_PID=$!
|
||||
# Wait for the server to start
|
||||
timeout 120 bash -c 'until curl -f http://localhost:7860/api/v1/auto_login; do sleep 2; done' || (echo "Server did not start in time" && kill $SERVER_PID && exit 1)
|
||||
# Terminate the server
|
||||
kill $SERVER_PID || (echo "Failed to terminate the server" && exit 1)
|
||||
sleep 20 # give the server some time to terminate
|
||||
# Check if the server is still running
|
||||
if kill -0 $SERVER_PID 2>/dev/null; then
|
||||
echo "Failed to terminate the server"
|
||||
exit 0
|
||||
else
|
||||
echo "Server terminated successfully"
|
||||
fi
|
||||
|
||||
# PyPI publishing moved to after cross-platform testing
|
||||
|
||||
- name: Upload Artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: dist-nightly-base
|
||||
path: src/backend/base/dist
|
||||
|
||||
build-nightly-main:
|
||||
name: Build Flow Nightly Main
|
||||
needs: [build-nightly-base]
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.verify.outputs.version }}
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main}}
|
||||
persist-credentials: true
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Download LFX artifact
|
||||
if: inputs.build_lfx
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-lfx
|
||||
path: lfx-dist
|
||||
|
||||
- name: Download base artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-base
|
||||
path: base-dist
|
||||
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
|
||||
- name: Force reinstall packages from built wheels
|
||||
run: |
|
||||
# While workspace resolution installs from source, we want to test the exact
|
||||
# wheels that will be published to PyPI to catch any packaging issues
|
||||
if [ "${{ inputs.build_lfx }}" == "true" ]; then
|
||||
echo "Installing LFX from built wheel..."
|
||||
uv pip install --force-reinstall --no-deps lfx-dist/*.whl
|
||||
fi
|
||||
echo "Installing flow-base from built wheel..."
|
||||
uv pip install --force-reinstall --no-deps base-dist/*.whl
|
||||
|
||||
- name: Verify Nightly Name and Version
|
||||
id: verify
|
||||
run: |
|
||||
name=$(uv tree | grep -E '^flow(-nightly)?[[:space:]]' | awk '{print $1}')
|
||||
version=$(uv tree | grep -E '^flow(-nightly)?[[:space:]]' | awk '{print $2}')
|
||||
if [ "$name" != "flow-nightly" ]; then
|
||||
echo "Name $name does not match flow-nightly. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
if [ "$version" != "${{ inputs.nightly_tag_main }}" ]; then
|
||||
echo "Version $version does not match nightly tag ${{ inputs.nightly_tag_main }}. Exiting the workflow."
|
||||
exit 1
|
||||
fi
|
||||
# Strip the leading `v` from the version
|
||||
version=$(echo $version | sed 's/^v//')
|
||||
echo "version=$version" >> $GITHUB_OUTPUT
|
||||
- name: Wait for PyPI Propagation
|
||||
if: needs.build-nightly-base.outputs.skipped == 'false'
|
||||
run: sleep 300 # wait for 5 minutes to ensure PyPI propagation of base
|
||||
|
||||
- name: Build Flow Main for distribution
|
||||
run: make build main=true args="--no-sources --wheel"
|
||||
- name: Test Flow Main CLI
|
||||
run: |
|
||||
uv pip install dist/*.whl
|
||||
uv run python -m flow run --host localhost --port 7860 --backend-only &
|
||||
SERVER_PID=$!
|
||||
# Wait for the server to start
|
||||
timeout 120 bash -c 'until curl -f http://localhost:7860/health_check; do sleep 2; done' || (echo "Server did not start in time" && kill $SERVER_PID && exit 1)
|
||||
# Terminate the server
|
||||
kill $SERVER_PID || (echo "Failed to terminate the server" && exit 1)
|
||||
sleep 20 # give the server some time to terminate
|
||||
# Check if the server is still running
|
||||
if kill -0 $SERVER_PID 2>/dev/null; then
|
||||
echo "Failed to terminate the server"
|
||||
exit 0
|
||||
else
|
||||
echo "Server terminated successfully"
|
||||
fi
|
||||
|
||||
- name: Prepare Flow Main artifact
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
MAIN_WHEELS=(dist/flow_nightly-*.whl)
|
||||
if [ ${#MAIN_WHEELS[@]} -ne 1 ]; then
|
||||
echo "Expected exactly one flow-nightly wheel in dist/, found ${#MAIN_WHEELS[@]}"
|
||||
ls -la dist/
|
||||
exit 1
|
||||
fi
|
||||
rm -rf main-dist
|
||||
mkdir -p main-dist
|
||||
cp "${MAIN_WHEELS[0]}" main-dist/
|
||||
|
||||
- name: Build bundle wheels for cross-platform test
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
bundles=(src/bundles/*/pyproject.toml)
|
||||
if [ ${#bundles[@]} -eq 0 ]; then
|
||||
echo "No bundles found under src/bundles/*/"
|
||||
exit 1
|
||||
fi
|
||||
rm -rf bundles-dist
|
||||
mkdir -p bundles-dist
|
||||
BUNDLES_DIST="$PWD/bundles-dist"
|
||||
for bundle_pyproject in "${bundles[@]}"; do
|
||||
bundle_dir=$(dirname "$bundle_pyproject")
|
||||
echo "Building wheel for $bundle_dir"
|
||||
(cd "$bundle_dir" && uv build --wheel --out-dir "$BUNDLES_DIST")
|
||||
done
|
||||
ls -la bundles-dist/
|
||||
|
||||
# PyPI publishing moved to after cross-platform testing
|
||||
|
||||
- name: Upload Artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: dist-nightly-main
|
||||
path: main-dist
|
||||
|
||||
- name: Upload bundles artifact
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: dist-nightly-bundles
|
||||
path: bundles-dist
|
||||
|
||||
test-cross-platform:
|
||||
name: Test Cross-Platform Installation
|
||||
needs: [build-nightly-lfx, build-nightly-base, build-nightly-main]
|
||||
uses: ./.github/workflows/cross-platform-test.yml
|
||||
with:
|
||||
base-artifact-name: "dist-nightly-base"
|
||||
main-artifact-name: "dist-nightly-main"
|
||||
lfx-artifact-name: "dist-nightly-lfx"
|
||||
sdk-artifact-name: "dist-nightly-sdk"
|
||||
bundles-artifact-name: "dist-nightly-bundles"
|
||||
|
||||
publish-nightly-lfx:
|
||||
name: Publish LFX Nightly to PyPI
|
||||
needs: [build-nightly-lfx, test-cross-platform, publish-nightly-sdk]
|
||||
if: ${{ inputs.build_lfx }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
persist-credentials: true
|
||||
- name: Download LFX artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-lfx
|
||||
path: src/lfx/dist
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
python-version: "3.13"
|
||||
- name: Publish LFX to PyPI
|
||||
env:
|
||||
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_API_TOKEN }}
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
make lfx_publish
|
||||
|
||||
publish-nightly-sdk:
|
||||
name: Publish SDK Nightly to PyPI
|
||||
needs: [build-nightly-lfx, test-cross-platform]
|
||||
if: ${{ inputs.build_lfx }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check out the code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_release }}
|
||||
persist-credentials: true
|
||||
- name: Download flow-sdk artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-sdk
|
||||
path: src/sdk/dist
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
python-version: "3.13"
|
||||
- name: Publish SDK to PyPI
|
||||
env:
|
||||
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_API_TOKEN }}
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
make sdk_publish
|
||||
|
||||
publish-nightly-base:
|
||||
name: Publish Flow Base Nightly to PyPI
|
||||
needs: [build-nightly-base, test-cross-platform, publish-nightly-lfx]
|
||||
if: ${{ always() && needs.build-nightly-base.result == 'success' && needs.test-cross-platform.result == 'success' && (needs.publish-nightly-lfx.result == 'success' || inputs.build_lfx == false) }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
persist-credentials: true
|
||||
- name: Download base artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-base
|
||||
path: src/backend/base/dist
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
python-version: "3.13"
|
||||
- name: Publish base to PyPI
|
||||
if: needs.build-nightly-base.outputs.skipped == 'false'
|
||||
env:
|
||||
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_API_TOKEN }}
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
make publish base=true
|
||||
|
||||
publish-nightly-bundles:
|
||||
name: Publish Flow Bundles Nightly to PyPI
|
||||
# flow-nightly pins each `lfx-<name>-nightly==<dev>` exactly, so the
|
||||
# bundles must be on PyPI before main is published (publish-nightly-main
|
||||
# gates on this job). They depend on `lfx-nightly`, so gate on the lfx
|
||||
# publish too (or skip it when lfx isn't being rebuilt).
|
||||
needs: [build-nightly-main, test-cross-platform, publish-nightly-lfx]
|
||||
if: ${{ always() && needs.build-nightly-main.result == 'success' && needs.test-cross-platform.result == 'success' && (needs.publish-nightly-lfx.result == 'success' || inputs.build_lfx == false) }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Download bundles artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-bundles
|
||||
path: bundles-dist
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
python-version: "3.13"
|
||||
- name: Publish bundles to PyPI
|
||||
env:
|
||||
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_API_TOKEN }}
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
shopt -s nullglob
|
||||
wheels=(bundles-dist/*.whl)
|
||||
if [ ${#wheels[@]} -eq 0 ]; then
|
||||
echo "No bundle wheels to publish."
|
||||
exit 1
|
||||
fi
|
||||
failed=0
|
||||
for wheel in "${wheels[@]}"; do
|
||||
echo "Publishing $wheel"
|
||||
# Capture stderr so a re-run without a version bump is a no-op for
|
||||
# an already-published bundle instead of failing the whole job.
|
||||
if ! err=$(uv publish "$wheel" 2>&1); then
|
||||
echo "$err"
|
||||
if echo "$err" | grep -qiE 'already exists|file already exists|HTTP 400|duplicate'; then
|
||||
echo "Skipping $wheel: already published."
|
||||
else
|
||||
echo "Publish failed for $wheel"
|
||||
failed=1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
if [ "$failed" = "1" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
publish-nightly-main:
|
||||
name: Publish Flow Main Nightly to PyPI
|
||||
needs: [build-nightly-main, test-cross-platform, publish-nightly-base, publish-nightly-bundles]
|
||||
if: ${{ always() && needs.build-nightly-main.result == 'success' && needs.test-cross-platform.result == 'success' && needs.publish-nightly-base.result == 'success' && needs.publish-nightly-bundles.result == 'success' }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
persist-credentials: true
|
||||
- name: Download main artifact
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: dist-nightly-main
|
||||
path: dist
|
||||
- name: Setup Environment
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: false
|
||||
python-version: "3.13"
|
||||
- name: Publish to PyPI
|
||||
env:
|
||||
POETRY_PYPI_TOKEN_PYPI: ${{ secrets.PYPI_API_TOKEN }}
|
||||
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
|
||||
run: |
|
||||
make publish main=true
|
||||
|
||||
call_docker_build_base:
|
||||
name: Call Docker Build Workflow for Flow Base
|
||||
if: ${{ always() && inputs.build_docker_base }}
|
||||
needs: [build-nightly-base, build-nightly-main]
|
||||
uses: ./.github/workflows/docker-nightly-build.yml
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
release_type: nightly-base
|
||||
push_to_registry: ${{ inputs.push_to_registry }}
|
||||
secrets: inherit
|
||||
|
||||
call_docker_build_main:
|
||||
name: Call Docker Build Workflow for Flow
|
||||
if: ${{ always() && inputs.build_docker_main }}
|
||||
needs: [build-nightly-main, call_docker_build_base]
|
||||
uses: ./.github/workflows/docker-nightly-build.yml
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
release_type: nightly-main
|
||||
push_to_registry: ${{ inputs.push_to_registry }}
|
||||
secrets: inherit
|
||||
|
||||
# TODO: Uncomment this when our runner can fit the builds that contain pytorch (and other large dependencies)
|
||||
# call_docker_build_main_all:
|
||||
# name: Call Docker Build Workflow for flow-all
|
||||
# if: always() && ${{ inputs.build_docker_main == 'true' }}
|
||||
# needs: [build-nightly-main]
|
||||
# uses: ./.github/workflows/docker-nightly-build.yml
|
||||
# with:
|
||||
# ref: ${{ inputs.nightly_tag_main }}
|
||||
# release_type: nightly-main-all
|
||||
# main_version: ${{ inputs.nightly_tag_main }}
|
||||
# secrets: inherit
|
||||
|
||||
call_docker_build_main_ep:
|
||||
name: Call Docker Build Workflow for Flow with Entrypoint
|
||||
if: ${{ always() && inputs.build_docker_ep }}
|
||||
needs: [build-nightly-main, call_docker_build_main]
|
||||
uses: ./.github/workflows/docker-build-v2.yml
|
||||
with:
|
||||
ref: ${{ inputs.nightly_tag_main }}
|
||||
release_type: main-ep
|
||||
push_to_registry: ${{ inputs.push_to_registry }}
|
||||
secrets: inherit
|
||||
@@ -1,37 +0,0 @@
|
||||
name: Request Docs Review
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
|
||||
jobs:
|
||||
request-docs-review:
|
||||
if: github.event.label.name == 'needs-docs'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const prAuthor = context.payload.pull_request.user.login;
|
||||
console.log(`PR author: ${prAuthor}`);
|
||||
|
||||
const reviewers = '${{ vars.DOCS_TEAM_MEMBERS }}'
|
||||
.split(',')
|
||||
.map(u => u.trim())
|
||||
.filter(u => u.length > 0 && u !== prAuthor);
|
||||
|
||||
if (reviewers.length > 0) {
|
||||
console.log(`Requesting review from: ${reviewers.join(', ')}`);
|
||||
|
||||
await github.rest.pulls.requestReviewers({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: context.issue.number,
|
||||
reviewers: reviewers
|
||||
});
|
||||
|
||||
console.log('Review request sent successfully');
|
||||
} else {
|
||||
console.log('No eligible reviewers found (all team members may be the PR author)');
|
||||
}
|
||||
@@ -1,186 +0,0 @@
|
||||
name: Smoke Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, labeled, synchronize, reopened]
|
||||
release:
|
||||
types: [published]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
ref:
|
||||
description: 'Git ref to checkout (branch, tag, or commit SHA)'
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
backend-smoke-tests:
|
||||
if: contains(github.event.pull_request.labels.*.name, 'smoke-test') || github.event_name == 'release' || github.event_name == 'workflow_dispatch'
|
||||
name: "Backend Smoke Tests"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout PR
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
ref: ${{ github.event.inputs.ref || github.ref }}
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install backend dependencies
|
||||
run: |
|
||||
uv sync --dev
|
||||
|
||||
- name: Run backend smoke tests (critical tests only)
|
||||
run: |
|
||||
uv run pytest \
|
||||
src/backend/tests/unit/test_database.py \
|
||||
src/backend/tests/unit/test_login.py \
|
||||
src/backend/tests/unit/api/v1/test_validate.py \
|
||||
src/backend/tests/unit/test_endpoints.py \
|
||||
src/backend/tests/unit/api/v1/test_flows.py \
|
||||
src/backend/tests/unit/test_chat_endpoint.py \
|
||||
src/backend/tests/unit/api/v1/test_api_key.py \
|
||||
src/backend/tests/unit/api/v1/test_endpoints.py \
|
||||
src/backend/tests/unit/components/languagemodels/test_openai_model.py \
|
||||
src/backend/tests/unit/components/agents/test_agent_component.py \
|
||||
src/backend/tests/unit/services/tracing/test_tracing_service.py \
|
||||
-m 'not api_key_required' \
|
||||
--tb=short \
|
||||
-v
|
||||
env:
|
||||
FLOW_SUPERUSER: admin
|
||||
FLOW_SUPERUSER_PASSWORD: 123456
|
||||
|
||||
lfx-smoke-tests:
|
||||
# created a separate label for this since it's a different test suite for LFX components
|
||||
if: contains(github.event.pull_request.labels.*.name, 'smoke-test') || github.event_name == 'release' || github.event_name == 'workflow_dispatch'
|
||||
name: "LFX Smoke Tests"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
steps:
|
||||
- name: Checkout PR
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
ref: ${{ github.event.inputs.ref || github.ref }}
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install lfx dependencies (isolated)
|
||||
run: |
|
||||
cd src/lfx
|
||||
uv sync --dev
|
||||
|
||||
- name: Run lfx smoke tests (critical tests only)
|
||||
run: |
|
||||
cd src/lfx
|
||||
uv run pytest \
|
||||
tests/unit/graph/test_graph.py \
|
||||
tests/unit/custom/component/test_component_instance_attributes.py \
|
||||
tests/unit/schema/test_schema_message.py \
|
||||
-m 'not api_key_required' \
|
||||
--tb=short \
|
||||
--maxfail=5 \
|
||||
-v
|
||||
env:
|
||||
FLOW_SUPERUSER: admin
|
||||
FLOW_SUPERUSER_PASSWORD: 123456
|
||||
|
||||
frontend-smoke-tests:
|
||||
if: contains(github.event.pull_request.labels.*.name, 'smoke-test') || github.event_name == 'release' || github.event_name == 'workflow_dispatch'
|
||||
name: "Frontend Smoke Tests"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
steps:
|
||||
- name: Checkout PR
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
ref: ${{ github.event.inputs.ref || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: "22"
|
||||
cache: "npm"
|
||||
cache-dependency-path: src/frontend/package-lock.json
|
||||
|
||||
- name: Install frontend dependencies
|
||||
run: |
|
||||
cd src/frontend
|
||||
npm ci
|
||||
|
||||
- name: Run frontend smoke tests (unit tests only)
|
||||
run: |
|
||||
cd src/frontend
|
||||
CI=true npx jest --ci --watchAll=false --passWithNoTests
|
||||
env:
|
||||
NODE_ENV: test
|
||||
|
||||
comment-results:
|
||||
if: always() && (contains(github.event.pull_request.labels.*.name, 'smoke-test') || github.event_name == 'release' || github.event_name == 'workflow_dispatch')
|
||||
name: "Comment Results"
|
||||
needs: [backend-smoke-tests, lfx-smoke-tests, frontend-smoke-tests]
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Comment on PR with results
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const backendStatus = '${{ needs.backend-smoke-tests.result }}';
|
||||
const lfxStatus = '${{ needs.lfx-smoke-tests.result }}';
|
||||
const frontendStatus = '${{ needs.frontend-smoke-tests.result }}';
|
||||
|
||||
const overallSuccess = backendStatus === 'success' && lfxStatus === 'success' && frontendStatus === 'success';
|
||||
const emoji = overallSuccess ? '✅' : '❌';
|
||||
const status = overallSuccess ? 'passed' : 'failed';
|
||||
|
||||
const backendEmoji = backendStatus === 'success' ? '✅' : '❌';
|
||||
const lfxEmoji = lfxStatus === 'success' ? '✅' : '❌';
|
||||
const frontendEmoji = frontendStatus === 'success' ? '✅' : '❌';
|
||||
|
||||
const comment = `${emoji} **Smoke tests ${status}**
|
||||
|
||||
Critical functionality validated:
|
||||
|
||||
- ${backendEmoji} **Backend**: 11 essential test files (database, auth, API endpoints, flows, components)
|
||||
- ${lfxEmoji} **LFX**: Graph tests (flow execution) - runs in isolation
|
||||
- ${frontendEmoji} **Frontend**: Unit tests only (components, utilities)
|
||||
|
||||
**Coverage**: Core functionality without external dependencies
|
||||
|
||||
View details in the [Actions tab](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}).`;
|
||||
|
||||
// Only comment on PRs, not on releases or manual runs
|
||||
if (context.payload.pull_request) {
|
||||
await github.rest.issues.createComment({
|
||||
issue_number: context.payload.pull_request.number,
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
body: comment
|
||||
});
|
||||
} else {
|
||||
console.log('Smoke test results:', comment);
|
||||
}
|
||||
@@ -1,76 +0,0 @@
|
||||
name: Store pytest durations
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
# Run job at 6:30 UTC every Monday (10.30pm PST/11.30pm PDT Sunday night)
|
||||
- cron: "30 6 * * 1"
|
||||
|
||||
env:
|
||||
PYTEST_RUN_PATH: "src/backend/tests"
|
||||
|
||||
jobs:
|
||||
build:
|
||||
name: Run pytest and store durations
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
|
||||
ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: "3.13"
|
||||
prune-cache: false
|
||||
- name: Install the project
|
||||
run: uv sync
|
||||
- name: Run unit tests
|
||||
id: run_tests
|
||||
continue-on-error: true
|
||||
run: uv run pytest src/backend/tests/unit --timeout=150 --durations-path src/backend/tests/.test_durations --splitting-algorithm least_duration --store-durations
|
||||
|
||||
|
||||
- name: Close existing PRs
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
script: |
|
||||
const { data: pulls } = await github.rest.pulls.list({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
state: 'open'
|
||||
});
|
||||
|
||||
for (const pull of pulls) {
|
||||
if (pull.title === "chore: update test durations") {
|
||||
await github.rest.pulls.update({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: pull.number,
|
||||
state: 'closed'
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
branch-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
commit-message: "chore: update test durations"
|
||||
title: "chore: update test durations"
|
||||
body: |
|
||||
Automated PR to update test durations file.
|
||||
|
||||
This PR was automatically created by the store_pytest_durations workflow.
|
||||
branch: update-test-durations
|
||||
branch-suffix: timestamp
|
||||
delete-branch: true
|
||||
maintainer-can-modify: true
|
||||
@@ -1,31 +0,0 @@
|
||||
name: Ruff Style Check
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, auto_merge_enabled]
|
||||
paths:
|
||||
- "**/*.py"
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
name: Ruff Style Check
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- "3.13"
|
||||
steps:
|
||||
- name: Check out the code at a specific ref
|
||||
uses: actions/checkout@v6
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ matrix.python-version }}
|
||||
prune-cache: false
|
||||
- name: Register problem matcher
|
||||
run: echo "::add-matcher::.github/workflows/matchers/ruff.json"
|
||||
- name: Run Ruff Check
|
||||
run: uv run --only-dev ruff check --output-format=github .
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
name: Template Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'src/backend/base/flow/initial_setup/starter_projects/**'
|
||||
- 'src/backend/tests/unit/template/test_starter_projects.py'
|
||||
- 'src/backend/base/flow/utils/template_validation.py'
|
||||
- '.github/workflows/template-tests.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
|
||||
jobs:
|
||||
test-starter-projects:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: 3.12
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv sync --dev
|
||||
|
||||
- name: Test all starter project templates
|
||||
run: |
|
||||
uv run pytest src/backend/tests/unit/template/test_starter_projects.py -v -n auto
|
||||
@@ -1,175 +0,0 @@
|
||||
# Advisory-only check: flags PRs that change source code (.py / .ts / .tsx)
|
||||
# without touching any test file. It NEVER fails and is NOT a required check,
|
||||
# so it cannot block other jobs or the merge — it only posts a sticky PR
|
||||
# comment + job summary so the author knows tests appear to be missing.
|
||||
name: Test Coverage Advisor
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
|
||||
# Dedicated concurrency group so this never cancels or queues behind CI.
|
||||
concurrency:
|
||||
group: test-coverage-advisor-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
advise:
|
||||
name: Test Coverage Advisor
|
||||
# Skip drafts to avoid noise; they re-run on "ready_for_review".
|
||||
if: github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
# Full history so merge-base against the PR target works.
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Detect missing tests
|
||||
id: detect
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
BASE_REF="${{ github.event.pull_request.base.ref }}"
|
||||
git fetch --no-tags --depth=50 origin "$BASE_REF" || true
|
||||
MERGE_BASE=$(git merge-base HEAD "origin/$BASE_REF" || git rev-parse HEAD~1)
|
||||
|
||||
echo "BASE_REF=$BASE_REF"
|
||||
echo "MERGE_BASE=$MERGE_BASE"
|
||||
|
||||
CHANGED=$(git diff --name-only --diff-filter=ACMR "$MERGE_BASE"...HEAD || true)
|
||||
echo "=== Changed files ==="
|
||||
echo "$CHANGED"
|
||||
|
||||
# --- Backend Python ---------------------------------------------
|
||||
# Source: .py under src/backend or src/lfx, excluding tests,
|
||||
# migrations and dunder/init plumbing.
|
||||
PY_SRC=$(echo "$CHANGED" \
|
||||
| grep -E '^(src/backend|src/lfx)/.*\.py$' \
|
||||
| grep -Ev '(^|/)tests?/' \
|
||||
| grep -Ev '(^|/)(test_[^/]+|conftest)\.py$' \
|
||||
| grep -Ev '/alembic/versions/' \
|
||||
| grep -Ev '/__init__\.py$' \
|
||||
|| true)
|
||||
PY_TEST=$(echo "$CHANGED" \
|
||||
| grep -E '\.py$' \
|
||||
| grep -E '((^|/)tests?/|(^|/)test_[^/]+\.py$|(^|/)conftest\.py$)' \
|
||||
|| true)
|
||||
|
||||
# --- Frontend TS/TSX --------------------------------------------
|
||||
# Source: .ts/.tsx under src/frontend/src, excluding *.test.* /
|
||||
# *.spec.* / __tests__ and type declaration files.
|
||||
FE_SRC=$(echo "$CHANGED" \
|
||||
| grep -E '^src/frontend/src/.*\.(ts|tsx)$' \
|
||||
| grep -Ev '\.(test|spec)\.(ts|tsx)$' \
|
||||
| grep -Ev '(^|/)__tests__/' \
|
||||
| grep -Ev '\.d\.ts$' \
|
||||
|| true)
|
||||
FE_TEST=$(echo "$CHANGED" \
|
||||
| grep -E '\.(ts|tsx)$' \
|
||||
| grep -E '(^src/frontend/tests/|\.(test|spec)\.(ts|tsx)$|(^|/)__tests__/)' \
|
||||
|| true)
|
||||
|
||||
need_be=false
|
||||
need_fe=false
|
||||
[ -n "$PY_SRC" ] && [ -z "$PY_TEST" ] && need_be=true
|
||||
[ -n "$FE_SRC" ] && [ -z "$FE_TEST" ] && need_fe=true
|
||||
|
||||
{
|
||||
echo "need_be=$need_be"
|
||||
echo "need_fe=$need_fe"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Multiline outputs for the comment body.
|
||||
{
|
||||
echo "py_src<<EOF"
|
||||
echo "$PY_SRC"
|
||||
echo "EOF"
|
||||
echo "fe_src<<EOF"
|
||||
echo "$FE_SRC"
|
||||
echo "EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Build advisory message
|
||||
id: msg
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
NEED_BE="${{ steps.detect.outputs.need_be }}"
|
||||
NEED_FE="${{ steps.detect.outputs.need_fe }}"
|
||||
|
||||
BODY_FILE="$(mktemp)"
|
||||
echo '<!-- test-coverage-advisor -->' > "$BODY_FILE"
|
||||
|
||||
if [ "$NEED_BE" != "true" ] && [ "$NEED_FE" != "true" ]; then
|
||||
{
|
||||
echo '### ✅ Test Coverage Advisor'
|
||||
echo
|
||||
echo 'No source changes detected without accompanying tests. Thanks for keeping coverage up! 🎉'
|
||||
echo
|
||||
echo '> _Advisory check only — never blocks merge._'
|
||||
} >> "$BODY_FILE"
|
||||
else
|
||||
{
|
||||
echo '### ⚠️ Test Coverage Advisor — tests appear to be missing'
|
||||
echo
|
||||
echo 'This PR changes source code but does **not** add or modify any test files.'
|
||||
echo 'This is an **advisory** check: it will **not** block this PR or any other job —'
|
||||
echo 'please add tests if the change is testable.'
|
||||
echo
|
||||
} >> "$BODY_FILE"
|
||||
if [ "$NEED_BE" = "true" ]; then
|
||||
{
|
||||
echo '#### Backend (Python) — no `test_*.py` / `tests/` change found'
|
||||
echo '```'
|
||||
echo '${{ steps.detect.outputs.py_src }}'
|
||||
echo '```'
|
||||
} >> "$BODY_FILE"
|
||||
fi
|
||||
if [ "$NEED_FE" = "true" ]; then
|
||||
{
|
||||
echo '#### Frontend (TS/TSX) — no `*.test.*` / `*.spec.*` / `tests/` change found'
|
||||
echo '```'
|
||||
echo '${{ steps.detect.outputs.fe_src }}'
|
||||
echo '```'
|
||||
} >> "$BODY_FILE"
|
||||
fi
|
||||
{
|
||||
echo
|
||||
echo '> _Advisory check only — it always passes and is never a required status._'
|
||||
} >> "$BODY_FILE"
|
||||
fi
|
||||
|
||||
# Mirror into the run's Job Summary (works even on fork PRs).
|
||||
cat "$BODY_FILE" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
echo "body_file=$BODY_FILE" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Find existing advisory comment
|
||||
id: find
|
||||
uses: peter-evans/find-comment@v3
|
||||
continue-on-error: true
|
||||
with:
|
||||
issue-number: ${{ github.event.pull_request.number }}
|
||||
comment-author: github-actions[bot]
|
||||
body-includes: "<!-- test-coverage-advisor -->"
|
||||
|
||||
- name: Upsert advisory comment
|
||||
uses: peter-evans/create-or-update-comment@v4
|
||||
# Fork PRs get a read-only token — don't fail the run if so.
|
||||
continue-on-error: true
|
||||
with:
|
||||
comment-id: ${{ steps.find.outputs.comment-id }}
|
||||
issue-number: ${{ github.event.pull_request.number }}
|
||||
body-path: ${{ steps.msg.outputs.body_file }}
|
||||
edit-mode: replace
|
||||
|
||||
- name: Always succeed
|
||||
# Explicit: this job never blocks anything.
|
||||
run: 'echo "Advisory check complete — this status is informational only."'
|
||||
@@ -1,158 +0,0 @@
|
||||
name: Flow Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main, develop ]
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
env:
|
||||
POSTGRES_USER: flow
|
||||
POSTGRES_PASSWORD: flow123
|
||||
POSTGRES_DB: flow_test
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
ports:
|
||||
- 5432:5432
|
||||
|
||||
mongodb:
|
||||
image: mongo:7
|
||||
ports:
|
||||
- 27017:27017
|
||||
options: >-
|
||||
--health-cmd "mongosh --eval 'db.adminCommand(\"ping\")'"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install uv
|
||||
run: |
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
echo "$HOME/.local/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Cache dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
~/.cache/uv
|
||||
.venv
|
||||
key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock', 'pyproject.toml') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-uv-
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv sync --frozen
|
||||
|
||||
- name: Create test environment
|
||||
run: |
|
||||
cat > .env.test << EOF
|
||||
DATABASE_URL=postgresql://flow:flow123@localhost:5432/flow_test
|
||||
MONGODB_URL=mongodb://localhost:27017/flow_test
|
||||
JWT_SECRET=test-secret
|
||||
ENABLE_WORKFLOW_EXECUTION=true
|
||||
MAX_CONCURRENT_WORKFLOWS=5
|
||||
EOF
|
||||
|
||||
- name: Run database migrations
|
||||
run: |
|
||||
uv run alembic upgrade head || echo "No migrations to run"
|
||||
|
||||
- name: Lint
|
||||
run: |
|
||||
uv run ruff check . || true
|
||||
|
||||
- name: Type check
|
||||
run: |
|
||||
uv run mypy . || true
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
uv run pytest -v
|
||||
env:
|
||||
PYTHONPATH: .
|
||||
|
||||
- name: Test workflow execution
|
||||
run: |
|
||||
# Create a simple test workflow
|
||||
cat > test-workflow.json << 'EOF'
|
||||
{
|
||||
"name": "Test Workflow",
|
||||
"nodes": [
|
||||
{
|
||||
"id": "start",
|
||||
"type": "trigger",
|
||||
"data": { "event": "manual" }
|
||||
},
|
||||
{
|
||||
"id": "transform",
|
||||
"type": "transform",
|
||||
"data": { "expression": "input.value * 2" }
|
||||
}
|
||||
],
|
||||
"edges": [
|
||||
{ "source": "start", "target": "transform" }
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Test workflow validation
|
||||
uv run python -m flow.validate test-workflow.json || echo "No validation script"
|
||||
|
||||
docker-test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Build Docker image
|
||||
run: |
|
||||
docker build -t flow-test .
|
||||
|
||||
- name: Test container startup
|
||||
run: |
|
||||
docker run -d \
|
||||
--name flow-test \
|
||||
-p 3020:3020 \
|
||||
-e DATABASE_URL="postgresql://test:test@localhost:5432/flow" \
|
||||
-e MONGODB_URL="mongodb://localhost:27017/flow" \
|
||||
flow-test
|
||||
|
||||
sleep 5
|
||||
docker logs flow-test
|
||||
docker stop flow-test
|
||||
docker rm flow-test
|
||||
|
||||
security-scan:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Run Semgrep
|
||||
uses: returntocorp/semgrep-action@v1
|
||||
with:
|
||||
config: >-
|
||||
p/python
|
||||
p/fastapi
|
||||
p/security-audit
|
||||
p/secrets
|
||||
@@ -1,468 +0,0 @@
|
||||
name: Run Frontend Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
secrets:
|
||||
OPENAI_API_KEY:
|
||||
required: true
|
||||
STORE_API_KEY:
|
||||
required: true
|
||||
ANTHROPIC_API_KEY:
|
||||
required: true
|
||||
TAVILY_API_KEY:
|
||||
required: true
|
||||
inputs:
|
||||
suites:
|
||||
description: "Test suites to run (JSON array)"
|
||||
required: false
|
||||
type: string
|
||||
default: "[]"
|
||||
release:
|
||||
description: "Whether this is a release build"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
tests_folder:
|
||||
description: "(Optional) Tests to run"
|
||||
required: false
|
||||
type: string
|
||||
default: "tests"
|
||||
ref:
|
||||
description: "(Optional) ref to checkout"
|
||||
required: false
|
||||
type: string
|
||||
runs-on:
|
||||
description: "Runner to use for the tests"
|
||||
required: false
|
||||
type: string
|
||||
default: "ubuntu-latest"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
suites:
|
||||
description: "Test suites to run (JSON array)"
|
||||
required: false
|
||||
type: string
|
||||
default: "[]"
|
||||
release:
|
||||
description: "Whether this is a release build"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
tests_folder:
|
||||
description: "(Optional) Tests to run"
|
||||
required: false
|
||||
type: string
|
||||
default: "tests"
|
||||
runs-on:
|
||||
description: "Runner to use for the tests"
|
||||
required: false
|
||||
type: choice
|
||||
options:
|
||||
- ubuntu-latest
|
||||
- self-hosted
|
||||
- '["self-hosted", "linux", "ARM64", "flow-ai-arm64-40gb-ephemeral"]'
|
||||
default: ubuntu-latest
|
||||
|
||||
env:
|
||||
NODE_VERSION: "22"
|
||||
PYTHON_VERSION: "3.13"
|
||||
# Define the directory where Playwright browsers will be installed.
|
||||
# This path is used for caching across workflows
|
||||
PLAYWRIGHT_BROWSERS_PATH: "ms-playwright"
|
||||
PLAYWRIGHT_VERSION: "1.59.1"
|
||||
FLOW_FEATURE_WXO_DEPLOYMENTS: "true"
|
||||
|
||||
jobs:
|
||||
determine-test-suite:
|
||||
name: Determine Test Suites and Shard Distribution
|
||||
runs-on: ${{ (inputs['runs-on'] && startsWith(format('{0}', inputs['runs-on']), '[') && fromJSON(inputs['runs-on'])) || inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
outputs:
|
||||
matrix: ${{ steps.setup-matrix.outputs.matrix }}
|
||||
test_grep: ${{ steps.set-matrix.outputs.test_grep }}
|
||||
suites: ${{ steps.set-matrix.outputs.suites }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Paths Filter
|
||||
id: filter
|
||||
uses: dorny/paths-filter@v3
|
||||
with:
|
||||
filters: .github/changes-filter.yaml
|
||||
|
||||
- name: Determine Test Suites from Changes
|
||||
id: set-matrix
|
||||
run: |
|
||||
# Start with input suites if provided, otherwise empty array
|
||||
echo "Changes filter output: $(echo '${{ toJSON(steps.filter.outputs) }}')"
|
||||
SUITES='${{ inputs.suites }}'
|
||||
echo "Initial suites: $SUITES"
|
||||
TEST_GREP=""
|
||||
echo "Inputs Release: ${{ inputs.release }}"
|
||||
RELEASE="${{ inputs.release || 'false' }}"
|
||||
echo "Release build: $RELEASE"
|
||||
|
||||
# Only set to release if it's explicitly a release build
|
||||
if [[ "$RELEASE" == "true" ]]; then
|
||||
SUITES='["release"]'
|
||||
echo "Release build detected - setting suites to: $SUITES"
|
||||
# grep pattern for release is the @release tag - run all tests
|
||||
TEST_GREP='--grep="@release"'
|
||||
else
|
||||
# If input suites were not provided, determine based on changes
|
||||
if [[ "$SUITES" == "[]" ]]; then
|
||||
echo "No input suites provided - determining from changes"
|
||||
SUITES='[]' # Ensure we start with a valid JSON array
|
||||
TAGS=()
|
||||
# Add suites and tags based on changed files
|
||||
if [[ "${{ steps.filter.outputs.components }}" == "true" ]]; then
|
||||
SUITES=$(echo $SUITES | jq -c '. += ["components"]')
|
||||
TAGS+=("@components")
|
||||
echo "Added components suite"
|
||||
fi
|
||||
if [[ "${{ steps.filter.outputs.starter-projects }}" == "true" ]]; then
|
||||
SUITES=$(echo $SUITES | jq -c '. += ["starter-projects"]')
|
||||
TAGS+=("@starter-projects")
|
||||
echo "Added starter-projects suite"
|
||||
fi
|
||||
if [[ "${{ steps.filter.outputs.workspace }}" == "true" ]]; then
|
||||
SUITES=$(echo $SUITES | jq -c '. += ["workspace"]')
|
||||
TAGS+=("@workspace")
|
||||
echo "Added workspace suite"
|
||||
fi
|
||||
if [[ "${{ steps.filter.outputs.api }}" == "true" ]]; then
|
||||
SUITES=$(echo $SUITES | jq -c '. += ["api"]')
|
||||
TAGS+=("@api")
|
||||
echo "Added api suite"
|
||||
fi
|
||||
if [[ "${{ steps.filter.outputs.database }}" == "true" ]]; then
|
||||
SUITES=$(echo $SUITES | jq -c '. += ["database"]')
|
||||
TAGS+=("@database")
|
||||
echo "Added database suite"
|
||||
fi
|
||||
if [[ "${{ steps.filter.outputs.mainpage }}" == "true" ]]; then
|
||||
SUITES=$(echo $SUITES | jq -c '. += ["mainpage"]')
|
||||
TAGS+=("@mainpage")
|
||||
echo "Added mainpage suite"
|
||||
fi
|
||||
if [[ "${{ steps.filter.outputs.development }}" == "true" ]]; then
|
||||
SUITES=$(echo $SUITES | jq -c '. += ["development"]')
|
||||
TAGS+=("@development")
|
||||
echo "Added development suite"
|
||||
fi
|
||||
|
||||
# Create grep pattern if we have tags
|
||||
if [ ${#TAGS[@]} -gt 0 ]; then
|
||||
# Join tags with | for OR logic
|
||||
REGEX_PATTERN=$(IFS='|'; echo "${TAGS[*]}")
|
||||
TEST_GREP='--grep="${REGEX_PATTERN}"'
|
||||
else
|
||||
# No path-specific changes detected - default to running release tests
|
||||
# so that FE tests always run even for backend-only changes
|
||||
SUITES='["release"]'
|
||||
TEST_GREP='--grep="@release"'
|
||||
echo "No path-specific changes detected - defaulting to release suite"
|
||||
fi
|
||||
else
|
||||
# Process input suites to tags
|
||||
# First ensure SUITES is valid JSON
|
||||
if ! echo "$SUITES" | jq -e . > /dev/null 2>&1; then
|
||||
echo "Warning: Input suites is not valid JSON, attempting to fix"
|
||||
# Try to fix common issues like missing quotes
|
||||
if [[ "$SUITES" == "[development]" ]]; then
|
||||
SUITES='["development"]'
|
||||
elif [[ "$SUITES" =~ ^\[(.*)\]$ ]]; then
|
||||
# Extract items and add quotes
|
||||
ITEMS="${BASH_REMATCH[1]}"
|
||||
QUOTED_ITEMS=$(echo "$ITEMS" | sed 's/\([^,]*\)/"\1"/g')
|
||||
SUITES="[$QUOTED_ITEMS]"
|
||||
fi
|
||||
echo "Fixed suites: $SUITES"
|
||||
fi
|
||||
|
||||
TAGS=()
|
||||
if echo "$SUITES" | jq -e 'contains(["components"])' > /dev/null; then
|
||||
TAGS+=("@components")
|
||||
fi
|
||||
if echo "$SUITES" | jq -e 'contains(["starter-projects"])' > /dev/null; then
|
||||
TAGS+=("@starter-projects")
|
||||
fi
|
||||
if echo "$SUITES" | jq -e 'contains(["workspace"])' > /dev/null; then
|
||||
TAGS+=("@workspace")
|
||||
fi
|
||||
if echo "$SUITES" | jq -e 'contains(["api"])' > /dev/null; then
|
||||
TAGS+=("@api")
|
||||
fi
|
||||
if echo "$SUITES" | jq -e 'contains(["database"])' > /dev/null; then
|
||||
TAGS+=("@database")
|
||||
fi
|
||||
if echo "$SUITES" | jq -e 'contains(["development"])' > /dev/null; then
|
||||
TAGS+=("@development")
|
||||
fi
|
||||
|
||||
if [ ${#TAGS[@]} -gt 0 ]; then
|
||||
# Join tags with | for OR logic
|
||||
REGEX_PATTERN=$(IFS='|'; echo "${TAGS[*]}")
|
||||
TEST_GREP='--grep "${REGEX_PATTERN}"'
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Ensure compact JSON output
|
||||
SUITES=$(echo "$SUITES" | jq -c '.')
|
||||
|
||||
echo "Final test suites to run: $SUITES"
|
||||
echo "Test grep pattern: $TEST_GREP"
|
||||
# Ensure proper JSON formatting for matrix output
|
||||
echo "matrix=$(echo $SUITES | jq -c .)" >> $GITHUB_OUTPUT
|
||||
echo "test_grep=$TEST_GREP" >> $GITHUB_OUTPUT
|
||||
echo "suites=$SUITES" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Setup Node ${{ env.NODE_VERSION }}
|
||||
uses: actions/setup-node@v6
|
||||
id: setup-node
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
# npm caching is decoupled from setup-node so a transient cache-service
|
||||
# hiccup (frequent on Windows runners) never fails the job. See #25 / #1.
|
||||
- name: Get npm cache directory
|
||||
id: npm-cache-dir
|
||||
shell: bash
|
||||
run: echo "dir=$(npm config get cache)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Cache npm dependencies
|
||||
uses: actions/cache@v5
|
||||
continue-on-error: true
|
||||
with:
|
||||
path: ${{ steps.npm-cache-dir.outputs.dir }}
|
||||
key: ${{ runner.os }}-npm-${{ env.NODE_VERSION }}-${{ hashFiles('src/frontend/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-npm-${{ env.NODE_VERSION }}-
|
||||
|
||||
- name: Install Frontend Dependencies
|
||||
run: npm ci
|
||||
working-directory: ./src/frontend
|
||||
|
||||
- name: Calculate Test Shards Distribution
|
||||
id: setup-matrix
|
||||
run: |
|
||||
cd src/frontend
|
||||
|
||||
# Get the test count using playwright's built-in grep
|
||||
if [ -n "${{ steps.set-matrix.outputs.test_grep }}" ]; then
|
||||
TEST_COUNT=$(npx playwright test ${{ inputs.tests_folder }} ${{ steps.set-matrix.outputs.test_grep }} --list | wc -l)
|
||||
else
|
||||
TEST_COUNT=$(npx playwright test ${{ inputs.tests_folder }} --list | wc -l)
|
||||
fi
|
||||
|
||||
echo "Total tests to run: $TEST_COUNT"
|
||||
|
||||
# Calculate optimal shard count - 1 shard per 5 tests, min 1, max 70
|
||||
SHARD_COUNT=$(( (TEST_COUNT + 4) / 5 ))
|
||||
if [ $SHARD_COUNT -lt 1 ]; then
|
||||
SHARD_COUNT=1
|
||||
elif [ $SHARD_COUNT -gt 70 ]; then
|
||||
SHARD_COUNT=70
|
||||
fi
|
||||
|
||||
# Create the matrix combinations string
|
||||
MATRIX_COMBINATIONS=""
|
||||
for i in $(seq 1 $SHARD_COUNT); do
|
||||
if [ $i -gt 1 ]; then
|
||||
MATRIX_COMBINATIONS="$MATRIX_COMBINATIONS,"
|
||||
fi
|
||||
MATRIX_COMBINATIONS="$MATRIX_COMBINATIONS{\"shardIndex\": $i, \"shardTotal\": $SHARD_COUNT}"
|
||||
done
|
||||
|
||||
echo "matrix={\"include\":[$MATRIX_COMBINATIONS]}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
setup-and-test:
|
||||
name: Playwright Tests - Shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
|
||||
runs-on: ${{ (inputs['runs-on'] && startsWith(format('{0}', inputs['runs-on']), '[') && fromJSON(inputs['runs-on'])) || inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
if: ${{ needs.determine-test-suite.outputs.test_grep != '' }}
|
||||
needs: determine-test-suite
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.determine-test-suite.outputs.matrix) }}
|
||||
env:
|
||||
OPENAI_API_KEY: ${{ inputs.openai_api_key || secrets.OPENAI_API_KEY }}
|
||||
STORE_API_KEY: ${{ inputs.store_api_key || secrets.STORE_API_KEY }}
|
||||
SEARCH_API_KEY: "${{ secrets.SEARCH_API_KEY }}"
|
||||
ASTRA_DB_APPLICATION_TOKEN: "${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}"
|
||||
ASTRA_DB_API_ENDPOINT: "${{ secrets.ASTRA_DB_API_ENDPOINT }}"
|
||||
ANTHROPIC_API_KEY: "${{ secrets.ANTHROPIC_API_KEY }}"
|
||||
TAVILY_API_KEY: "${{ secrets.TAVILY_API_KEY }}"
|
||||
FLOW_DEACTIVE_TRACING: "true"
|
||||
outputs:
|
||||
failed: ${{ steps.check-failure.outputs.failed }}
|
||||
steps:
|
||||
- name: Checkout Repository
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Setup Node.js Environment
|
||||
uses: actions/setup-node@v6
|
||||
id: setup-node
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
# npm caching is decoupled from setup-node so a transient cache-service
|
||||
# hiccup (frequent on Windows runners) never fails the job. See #25 / #1.
|
||||
- name: Get npm cache directory
|
||||
id: npm-cache-dir
|
||||
shell: bash
|
||||
run: echo "dir=$(npm config get cache)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Cache npm dependencies
|
||||
uses: actions/cache@v5
|
||||
continue-on-error: true
|
||||
with:
|
||||
path: ${{ steps.npm-cache-dir.outputs.dir }}
|
||||
key: ${{ runner.os }}-npm-${{ env.NODE_VERSION }}-${{ hashFiles('src/frontend/package-lock.json') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-npm-${{ env.NODE_VERSION }}-
|
||||
|
||||
- name: Install Frontend Dependencies
|
||||
run: npm ci
|
||||
working-directory: ./src/frontend
|
||||
|
||||
# Cache Playwright browsers using a composite key
|
||||
- name: Cache Playwright Browsers
|
||||
id: cache-playwright
|
||||
uses: actions/cache@v5
|
||||
continue-on-error: true
|
||||
with:
|
||||
path: ${{ env.PLAYWRIGHT_BROWSERS_PATH }}
|
||||
key: playwright-${{ env.PLAYWRIGHT_VERSION }}-chromium-${{ runner.os }}
|
||||
restore-keys: |
|
||||
playwright-${{ env.PLAYWRIGHT_VERSION }}-chromium-${{ runner.os }}
|
||||
|
||||
- name: Install Playwright Browser Dependencies
|
||||
if: steps.cache-playwright.outputs.cache-hit != 'true' && inputs.runs-on != 'self-hosted'
|
||||
run: |
|
||||
cd ./src/frontend
|
||||
npx playwright install --with-deps chromium
|
||||
|
||||
- name: Install Playwright Browsers (Self-Hosted)
|
||||
if: steps.cache-playwright.outputs.cache-hit != 'true' && inputs.runs-on == 'self-hosted'
|
||||
run: |
|
||||
cd ./src/frontend
|
||||
npx playwright install chromium
|
||||
|
||||
- name: "Setup Environment"
|
||||
uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
prune-cache: false
|
||||
|
||||
- name: Install Python Dependencies
|
||||
run: uv sync --extra audio
|
||||
|
||||
- name: Configure Environment Variables
|
||||
run: |
|
||||
touch .env
|
||||
echo "${{ secrets.ENV_VARS }}" > .env
|
||||
|
||||
- name: Execute Playwright Tests
|
||||
uses: nick-fields/retry@v3
|
||||
with:
|
||||
# Bumped from 12 → 18 minutes. Shard 38 carries all four
|
||||
# starter-projects-shardN tests + the sentiment streaming test;
|
||||
# iterating 8 templates per test under Linux nightly load
|
||||
# consistently runs past 12 minutes once a single iteration
|
||||
# triggers a per-test Playwright retry.
|
||||
timeout_minutes: 18
|
||||
max_attempts: 2
|
||||
command: |
|
||||
cd src/frontend
|
||||
echo 'Running tests with pattern: ${{ needs.determine-test-suite.outputs.test_grep }}'
|
||||
npx playwright test ${{ inputs.tests_folder }} ${{ needs.determine-test-suite.outputs.test_grep }} --shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --list --retries=3
|
||||
# echo command before running
|
||||
echo "npx playwright test ${{ inputs.tests_folder }} ${{ needs.determine-test-suite.outputs.test_grep }} --trace on --shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --workers 2 --retries=3"
|
||||
|
||||
npx playwright test ${{ inputs.tests_folder }} ${{ needs.determine-test-suite.outputs.test_grep }} --trace on --shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --workers 2 --retries=3
|
||||
|
||||
- name: Upload Test Results
|
||||
if: always()
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: blob-report-${{ matrix.shardIndex }}
|
||||
path: src/frontend/blob-report
|
||||
retention-days: 1
|
||||
overwrite: true
|
||||
|
||||
- name: Upload Playwright Coverage Artifact
|
||||
if: always() && hashFiles('src/frontend/coverage/playwright/**') != ''
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: playwright-coverage-${{ matrix.shardIndex }}
|
||||
path: src/frontend/coverage/playwright
|
||||
retention-days: 7
|
||||
overwrite: true
|
||||
|
||||
- name: Cleanup UV Cache
|
||||
run: uv cache prune --ci
|
||||
|
||||
merge-reports:
|
||||
# We need to repeat the condition at every step
|
||||
# https://github.com/actions/runner/issues/662
|
||||
needs: setup-and-test
|
||||
runs-on: ${{ (inputs['runs-on'] && startsWith(format('{0}', inputs['runs-on']), '[') && fromJSON(inputs['runs-on'])) || inputs['runs-on'] || github.event.inputs['runs-on'] || 'ubuntu-latest' }}
|
||||
if: always()
|
||||
env:
|
||||
EXIT_CODE: ${{!contains(needs.setup-and-test.result, 'failure') && !contains(needs.setup-and-test.result, 'cancelled') && '0' || '1'}}
|
||||
steps:
|
||||
- name: "Should Merge Reports"
|
||||
# If the CI was successful, we don't need to merge the reports
|
||||
# so we can skip all the steps below
|
||||
id: should_merge_reports
|
||||
run: |
|
||||
if [ "$EXIT_CODE" == "0" ]; then
|
||||
echo "should_merge_reports=false" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "should_merge_reports=true" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
- name: Checkout code
|
||||
if: ${{ steps.should_merge_reports.outputs.should_merge_reports == 'true' }}
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
|
||||
if: ${{ steps.should_merge_reports.outputs.should_merge_reports == 'true' }}
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
|
||||
- name: Download blob reports from GitHub Actions Artifacts
|
||||
|
||||
if: ${{ steps.should_merge_reports.outputs.should_merge_reports == 'true' }}
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
path: all-blob-reports
|
||||
pattern: blob-report-*
|
||||
merge-multiple: true
|
||||
|
||||
- name: Merge into HTML Report
|
||||
|
||||
if: ${{ steps.should_merge_reports.outputs.should_merge_reports == 'true' }}
|
||||
run: |
|
||||
npx playwright merge-reports --reporter html ./all-blob-reports
|
||||
|
||||
- name: Upload HTML report
|
||||
|
||||
if: ${{ steps.should_merge_reports.outputs.should_merge_reports == 'true' }}
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: html-report--attempt-${{ github.run_attempt }}
|
||||
path: playwright-report
|
||||
retention-days: 14
|
||||
@@ -1,7 +0,0 @@
|
||||
name: Workflow Sanity
|
||||
on:
|
||||
pull_request:
|
||||
paths: ['.github/workflows/**']
|
||||
jobs:
|
||||
sanity:
|
||||
uses: hanzoai/.github/.github/workflows/workflow-sanity.yml@main
|
||||
@@ -0,0 +1,59 @@
|
||||
name: security
|
||||
|
||||
# Static analysis for the Python and TypeScript trees. This replaces two dead
|
||||
# scanners rather than restoring them.
|
||||
#
|
||||
# Both had stopped running for the same structural reason: the forge collects
|
||||
# workflows from the FIRST entry of WORKFLOW_DIRS that exists, so the day
|
||||
# .hanzo/workflows/ appeared, every file still under .github/workflows/ went
|
||||
# dark. 43 of them, including codeql.yml and mend.yml. Nothing went red, because
|
||||
# the checks that would go red were the ones no longer running.
|
||||
#
|
||||
# Neither could come back as written:
|
||||
#
|
||||
# - codeql.yml ended in github/codeql-action/analyze under `security-events:
|
||||
# write`. That permission names an upload to GitHub's code-scanning store,
|
||||
# which is where CodeQL findings live — there is no such store here, and a
|
||||
# scanner whose results have nowhere to go is not a scanner. Semgrep covers
|
||||
# the same two languages CodeQL was configured for (python, javascript) and
|
||||
# reports through the exit code, which this forge does have.
|
||||
# - mend.yml fetched wss-unified-agent.jar and ran it against WS_APIKEY /
|
||||
# WS_USERKEY / WS_WSS_URL. Those are a commercial Mend subscription; this
|
||||
# org has no such account and no such secrets. Dependency/licence scanning
|
||||
# is therefore GONE, not replaced — semgrep reads code, not a dependency
|
||||
# graph, and pretending otherwise would be the more expensive mistake.
|
||||
#
|
||||
# Measured before it was written, on the tree at the time: 98 rules over 3558
|
||||
# git-tracked files in 20s, 4 ERROR findings. Fast enough to run on every push.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: security-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
semgrep:
|
||||
# Must name a label a runner actually advertises. This fleet serves
|
||||
# hanzo-build-linux-amd64 and deliberately offers no ubuntu-* label, so the
|
||||
# `runs-on: ubuntu-latest` these files carried would not have failed — it
|
||||
# would have queued until the timeout. Silence, not an error.
|
||||
runs-on: hanzo-build-linux-amd64
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# Pinned. A scanner that changes rules underneath the build turns an
|
||||
# unrelated commit red and teaches people to ignore it.
|
||||
- name: install semgrep
|
||||
run: python3 -m pip install --break-system-packages --quiet semgrep==1.167.0
|
||||
|
||||
# The whole tree, not a named list — an allowlist stops covering whatever
|
||||
# is added after it is written. semgrep walks only git-tracked files, so
|
||||
# node_modules and build output are already out of scope.
|
||||
- name: enforce (ERROR — fails the build)
|
||||
run: semgrep --config p/security-audit --config p/python --config p/javascript --error --severity=ERROR --metrics=off --timeout 300 --jobs 4 .
|
||||
Reference in New Issue
Block a user