insights/tools/insightscli/README.md

insightscli

A developer CLI framework for defining commands in YAML. Think of it as "GitHub Actions for your local dev environment" - declarative, composable, and easy to maintain.

Why insightscli?

• Declarative: Define commands in YAML instead of scattered shell scripts
• Composable: Chain commands together with steps
• Discoverable: Auto-generated help with categories, --help on every command
• Extensible: Add complex commands in Python when YAML isn't enough

Installation

pip install insightscli

Quick Start

Create insightscli.yaml in your repo root:

config:
  scripts_dir: bin # Where bin_script looks for scripts (default: bin/)

metadata:
  categories:
    - key: dev
      title: Development
    - key: test
      title: Testing

dev:
  dev:start:
    cmd: docker compose up -d && npm run dev
    description: Start development environment

  dev:reset:
    steps:
      - dev:stop
      - dev:clean
      - dev:start
    description: Full environment reset

test:
  test:unit:
    cmd: pytest tests/unit
    description: Run unit tests

Run commands:

insightscli dev:start
insightscli --help        # Shows all commands grouped by category
insightscli dev:start -h  # Help for specific command

Command Types

Shell commands (cmd)

Run shell commands directly. Supports shell operators (&&, ||, |):

build:
  cmd: npm run build && npm run test
  description: Build and test

Script delegation (bin_script)

Delegate to scripts in your scripts_dir:

deploy:
  bin_script: deploy.sh
  description: Deploy to production

Composite commands (steps)

Chain multiple insightscli commands:

release:
  steps:
    - test:all
    - build
    - deploy
  description: Full release pipeline

Steps can also include inline commands:

setup:
  steps:
    - name: Install deps
      cmd: npm install
    - name: Build
      cmd: npm run build
    - test:unit

Command Options

my:command:
  cmd: echo "hello"
  description: Short description for --help
  destructive: true # Prompts for confirmation before running
  hidden: true # Hides from --help (still runnable)
  needs_secrets: true # Triggers the configured `env.secrets.wrap` (see below)
  untracked: true # Skips command_started/command_completed telemetry (use for exec-style commands that replace the process, so a completed event could never fire)

Python Commands

For complex logic, define plain Click commands and reference them from insightscli.yaml with click: module.path:attribute.

Click command modules are lazy-loaded. Top-level insightscli --help uses the manifest description without importing Python command modules; insightscli <command> --help and command execution import the target on demand.

The Click command name must match the manifest key — drift surfaces as a ClickException on resolution. The framework follows Click's recommendation that lazy loading be paired with a test that runs --help on each subcommand; Insights's test suite parametrizes over every click: entry in insightscli.yaml to do exactly that.

Mark commands hidden via hidden: true in insightscli.yaml. Don't use @click.command(hidden=True) — the manifest is the single source of truth.

Minimal: one importable package

your-repo/
├── insightscli.yaml
└── tools/
    └── insightscli_commands/
        ├── __init__.py
        └── db.py

# insightscli.yaml
config:
  commands_dir: tools/insightscli_commands

db:
  db:migrate:
    click: insightscli_commands.db:db_migrate
    description: Run database migrations

# tools/insightscli_commands/db.py
import click

@click.command(name="db:migrate")
@click.option("--dry-run", is_flag=True, help="Show SQL without executing")
def db_migrate(dry_run: bool) -> None:
    """Run database migrations."""
    if dry_run:
        click.echo("Would run migrations...")
    else:
        # Your migration logic here
        pass

commands_dir is optional and explicit: insightscli only uses it when configured. It must be a relative path to an existing directory. insightscli puts that directory's parent on sys.path, so the directory should be an importable package or module tree. It must not be named insightscli, since that shadows the installed framework.

Full: project package with submodules

For a larger command surface, keep the distribution/project directory separate from the import package:

your-repo/
├── insightscli.yaml
└── tools/
    └── insightscli-commands/
        ├── pyproject.toml
        └── insightscli_commands/    # underscored: this is the import name
            ├── __init__.py
            ├── build.py
            ├── db.py
            └── deploy.py

# insightscli.yaml
config:
  commands_dir: tools/insightscli-commands/insightscli_commands

build:
  build:
    click: insightscli_commands.build:build
    description: Run build pipelines

The dashed outer dir + underscored inner package follows PEP 8 (dashed project name, underscored import name). This is the layout Insights itself uses.

Extension Hooks

Extensions can inject behavior at three framework call sites without forking. Register hooks from modules listed in config.boot_modules; those modules are imported once at startup, before command dispatch. Keep boot modules cheap to import and move heavy work inside hook functions. Exceptions raised by hooks are swallowed, so one extension can't break another.

config:
  commands_dir: tools/insightscli_commands
  boot_modules:
    - insightscli_commands.boot

Prechecks

Run validation before a command executes, keyed by type: in a prechecks: entry in insightscli.yaml. Return False to abort, True/None to continue.

from insightscli.hooks import register_precheck

def check_migrations(check: dict, yes: bool) -> bool | None:
    # inspect check config, prompt user, decide
    return None

register_precheck("migrations", check_migrations)

dev:start:
  cmd: docker compose up -d
  prechecks:
    - type: migrations

Telemetry properties

Inject extra key/value pairs into the command_started / command_completed telemetry events. Receives the invoked command name.

from insightscli.hooks import register_telemetry_properties

def env_props(command: str | None) -> dict[str, object]:
    return {"in_my_env": True}

register_telemetry_properties(env_props)

Post-command hooks

Run after every command completes, regardless of success. Good for contextual hints, cleanup, or notifications.

from insightscli.hooks import register_post_command_hook

def maybe_show_hint(command: str | None, exit_code: int) -> None:
    if exit_code == 0:
        ...

register_post_command_hook(maybe_show_hint)

Environment files

insightscli can load dotenv files into every command's environment, the same way just, mise, and task do. Declare them in insightscli.yaml:

config:
  env:
    files:
      - .env.development # loaded in order; earlier wins for duplicate keys
      - .env.services # shell env always wins (only_if_unset)

Missing files are silently skipped — same convention as the other tools — so .env.local is fine to leave optional.

Secret references (config.env.secrets)

Loading raw secrets from disk is a separate problem from loading dotenv files, and tools like 1Password CLI, Doppler, Infisical, Vault, and dotenvx all expose the same pattern: wrap the command they should run, fetch the secrets at runtime, inject them into the subprocess env. insightscli supports this generically — no provider-specific knowledge in core:

config:
  env:
    secrets:
      file: .env.local # the file that contains secret references
      marker: 'op://' # substring that triggers the wrap; required, non-empty
      wrap: [op, run, --env-file, '{file}', --]

Opt commands into the wrap with needs_secrets: true on the manifest entry:

start:
  bin_script: start
  description: Launch the dev stack
  needs_secrets: true # this command needs runtime secrets resolved

Commands without the flag never trigger the wrap — important because some wrap binaries (1Password's op on macOS, for example) prompt for biometric auth on every invocation, and you don't want lint / format / typecheck / pre-commit hooks pulling on that. The built-in insightscli run <cmd> always opts in, since its whole job is to forward the resolved env.

What happens at startup, for a command that opted in:

1. If the secrets file exists AND contains marker AND wrap[0] is on PATH: insightscli re-execs itself under wrap (with {file} substituted to the absolute path of the secrets file). The wrap binary resolves secrets and re-runs insightscli with them in the env. A INSIGHTSCLI_SECRETS_WRAPPED=1 sentinel is set in the wrap-child's env and is inherited by any subprocesses it spawns (so composite commands like dev:reset only prompt for auth once, not once per step).
2. If the wrap binary is missing or the marker isn't present, insightscli loads the file directly with marker-matching lines skipped (so unresolved op://... strings don't leak as garbage env values that produce confusing 401s downstream — only literal values get loaded).

For commands without needs_secrets: true, insightscli skips the wrap entirely and only loads literal values from the secrets file (marker-matching lines are skipped). So .env.local stays useful for non-secret overrides (DEBUG=1 etc.) even when the wrap doesn't fire.

Precedence in either path (highest wins): shell env > secrets file > env files. So a literal override in .env.local always beats .env.development, matching what op run does when it's available.

marker is required (not optional) — an "always wrap" mode would force a process exec on every insightscli invocation, which we never want.

The same shape works for Doppler (wrap: [doppler, run, --]), Vault (wrap: [vault, exec, --]), Infisical (wrap: [infisical, run, --]), dotenvx (wrap: [dotenvx, run, --]), or anything else following the wrap-and-exec convention. insightscli itself stays ignorant of which tool you use.

Why this split (and not built-in 1Password/Doppler/Vault knowledge)? Because mise's maintainer explicitly excluded secret resolution from mise core for good reasons: secret APIs are slow, caching them is a security risk, and CLI frameworks reload env too often for that to play nicely. Better to delegate to a purpose-built tool.

Configuration Reference

config:
  commands_dir: path/to/commands # Optional local Python command package
  boot_modules:
    - package.boot # Optional eager hook registration modules
  scripts_dir: scripts # For bin_script resolution (default: bin/)
  env:
    files: [.env.development, .env.services] # Optional dotenv files (in order)
    secrets: # Optional secret-wrapper hook
      file: .env.local
      marker: 'op://'
      wrap: [op, run, --env-file, '{file}', --]

metadata:
  categories:
    - key: dev
      title: Development Commands
    - key: test
      title: Test Commands

Built-in Commands

• insightscli quickstart - Getting started guide
• insightscli meta:check - Validate manifest against bin scripts (for CI)
• insightscli meta:concepts - Show infrastructure concepts (if defined)

Requirements

• Python 3.10+
• click
• pyyaml

Releasing

Releases are published to PyPI via .github/workflows/publish-insightscli.yml, triggered by pushing a insightscli-v* tag from master.

1. Bump version in tools/insightscli/pyproject.toml and merge to master.

2. From master, tag and push:

   git tag insightscli-v0.1.1
   git push origin insightscli-v0.1.1
   

The workflow verifies the tag matches the pyproject.toml version, builds the sdist and wheel with uv build, smoke-tests the wheel in a fresh venv, publishes via PyPI trusted publishing (OIDC) — no API tokens — and creates a GitHub Release with auto-generated notes from the commits since the previous tag.

To re-trigger after a failed publish, dispatch the workflow against the existing tag — no need to retag:

gh workflow run publish-insightscli.yml --ref insightscli-v0.1.1

The publish job is guarded by if: startsWith(github.ref, 'refs/tags/insightscli-v'), so dispatches from a branch are no-ops.

License

MIT