mirror of
https://github.com/hanzoai/insights.git
synced 2026-10-30 16:15:31 +00:00
zeekay
10e6fec4c0
The rule table mapped HOG_ -> FN_ ahead of any POSTHOG rule, so POSTHOG_FOO split into POST + HOG_ and came out POSTFN_FOO. Nothing sets that. The live pods set INSIGHTS_SKIP_MIGRATION_CHECKS, which bin/migrate-check reads on the container entrypoint, so the misspelling does not degrade a feature -- it runs migrate --check against a database whose history this tree does not share and the container never starts. Both replacements are measured against the shipped tree rather than chosen: every HOG_ name with a counterpart there is spelled INSIGHTS_ (HOG_QL -> INSIGHTS_QL, HOG_FUNCTION_TEMPLATES -> INSIGHTS_FUNCTION_TEMPLATES, 12 of 12 determinate cases), and FN_ appears in it nowhere. Upstream carries 95 HOG_ tokens and zero FN_ ones, so repairing FN_ back out is exactly the inverse of the bad rule and cannot collide with an upstream name. The rule is fixed for the next convergence, and the tree it already produced is repaired in the same commit -- the codemod is idempotent against its own output, so it cannot undo this itself.
2.3 KiB
2.3 KiB
LLMA Acceptance Test Implementation Plan
Overview
Python-based acceptance tests for the AI observability capture pipeline, testing against an existing Insights instance.
Test Execution
Local Development
- User starts Insights stack manually (using docker-compose or other method)
- Set INSIGHTS_TEST_BASE_URL environment variable to Insights instance URL
- Run acceptance tests:
python run_tests.py
GitHub Actions
- Triggered on PRs affecting rust/capture/ or nodejs/
- Sets up Insights stack in CI
- Runs acceptance tests against the stack
- Collects logs on failure
Implementation Steps
Commit 1: Basic acceptance test infrastructure ✓
- Create common/ingestion/acceptance_tests/ directory structure
- Add utils.py for service URL management
- Add requirements.txt with test dependencies
- Add run_tests.py test runner script
Commit 2: Insights API client for test setup
- Add api_client.py with InsightsTestClient class
- Implement organization creation via API
- Implement project creation with API key retrieval
- Add project deletion for cleanup
- Add event querying through Insights Query API
- Implement polling mechanism for event arrival
Commit 3: Basic event capture test
- Add test_basic_capture.py
- Test sending a regular Insights event to /capture
- Poll Query API until event appears
- Verify event properties match what was sent
- Confirm end-to-end pipeline works
Commit 4: Test orchestration
- Add conftest.py with pytest fixtures
- Add per-test project isolation
- Add environment variable handling
- Handle cleanup of test data
Commit 5: GitHub Actions workflow
- Add .github/workflows/llma-acceptance-tests.yml
- Configure triggers for relevant paths
- Add Insights stack startup steps
- Add test execution with pytest
- Implement log collection on failure
- Add cleanup steps
Commit 6: Documentation
- Add README.md with setup instructions
- Document local running procedures
- Document environment variables
- Add troubleshooting guide
- Create .env.example file
- Document how to extend the test suite
Usage
# Start Insights manually (e.g., with docker-compose)
docker-compose -f docker-compose.dev-full.yml up -d
# Set environment variable
export INSIGHTS_TEST_BASE_URL=http://localhost:8010
# Run tests
cd common/ingestion/acceptance_tests
python run_tests.py