kserve/test/e2e/llmisvc/README.md

LLM Inference Service E2E Tests

Configuration Composition Pattern

Tests combine config fragments from different categories to create complete scenarios:

pytest.param(
    TestCase(["router-managed", "workload-single-cpu", "model-fb-opt-125m"]),
    marks=pytest.mark.cluster_cpu,
)

The llm_config_factory fixture automatically creates/cleans up LLMInferenceServiceConfig objects.

Test Filtering

Tests are marked with both general and cluster-specific capability markers:

• @pytest.mark.llminferenceservice - All LLM inference service tests
• @pytest.mark.cluster_cpu - CPU-only tests
• @pytest.mark.cluster_amd - AMD GPU tests
• @pytest.mark.cluster_nvidia - NVIDIA GPU tests
• @pytest.mark.cluster_nvidia_roce - NVIDIA ROCe tests
• @pytest.mark.cluster_intel - Intel GPU tests

Examples:

# Run all LLM inference service tests
pytest -m "llminferenceservice" test/e2e/llmisvc/

# Run only CPU tests
pytest -m "llminferenceservice and cluster_cpu" test/e2e/llmisvc/

# Run only NVIDIA GPU tests
pytest -m "llminferenceservice and cluster_nvidia" test/e2e/llmisvc/

# Run all GPU tests (any vendor)
pytest -m "llminferenceservice and (cluster_amd or cluster_nvidia or cluster_intel)" test/e2e/llmisvc/

# Run CPU and AMD GPU tests only
pytest -m "llminferenceservice and (cluster_cpu or cluster_amd)" test/e2e/llmisvc/

Adding New Configs

1. Add to LLMINFERENCESERVICE_CONFIGS in fixtures.py

2. Follow category-descriptor naming (described in the subsequent section)

3. Add new cluster capability test cases using pytest.param with appropriate marks:

   pytest.param(
       TestCase(["router-managed", "workload-nvidia-a100-gpu", "model-llama-70b"]),
       marks=pytest.mark.cluster_nvidia,
   ),
   

   You can also customize test behavior with additional LlmDTestCase parameters:

   pytest.param(
       TestCase(
           base_refs=["router-managed", "workload-single-cpu", "model-fb-opt-125m"],
           prompt="What is the capital of France?",
           max_tokens=50,
           response_assertion=lambda response: (
               response.status_code == 200
               and response.json().get("choices") is not None
               and len(response.json().get("choices", [])) > 0
           ),
       ),
       marks=pytest.mark.cluster_cpu,
   ),
   

Config Naming Convention

Use prefixed categories that get composed together:

• workload-*: workload topology, container specs and resource specs (e.g., workload-single-cpu, workload-multi-node-gpu)
• model-*: Model sources (e.g., model-fb-opt-125m, model-gpt2)
• router-*: Routing configs (e.g., router-managed, router-with-scheduler)

Test IDs are generated by combining the cluster capability from pytest marks with all config names:

• Test ID format: {cluster_capability}-{config1}-{config2}-{config3}
• Example: cluster_cpu-router-managed-workload-single-cpu-model-fb-opt-125m