Improve logging

[rmarescu]

What

Log levels

type LogLevel = "debug" | "info" | "warn" | "error"

• error - test failures and critical issues
• warn - above + non-critical issues, deprecations
• info - above + test progress (default)
• debug - above + detailed info including AI conversations, browser actions, etc

Example outputs per level:

# error
✗ Login Flow
  Error: Element not found

# warn
⚠ Deprecated: --debug-ai flag, use --log-level=debug instead
✗ Login Flow
  Error: Element not found

# info (default)
File: login.test.ts
  ✓ Login Flow (2.1s)
  ✗ Logout Flow
    Error: Element not found

# debug
File: login.test.ts
  Test: Login Flow
    AI: Navigating to login page
      Prompt: "Click the login button"
      Response: "Looking for login button..."
    Browser: Clicking login button at (120, 50)

CLI usage

pnpm shortest --log-level=error
pnpm shortest --log-level=debug

Make --debug-ai an alias for --log-level=debug with a warn message for future deprecation

Test result tracking

Logger will track:

• Test status (pending/running/passed/failed)
• Errors with file/line info
• Token usage metrics
• Summary stats for final report

Log format

Normal output (info/warn/error):

File: login.test.ts
  ✓ Login Flow
  ✗ Logout Flow
    Error: Element not found

Debug output:

File: login.test.ts
  Test: Login Flow
    AI: Navigating to login page
      Prompt: "Click the login button"
      Response: "Looking for login button..."
    Browser: Clicking login button at (120, 50)

CI output:

::error file=login.test.ts,line=12::Element not found
::group::Test: Login Flow
AI: Navigating to login page
Browser: Clicking login button
::endgroup::

Log Grouping

• Group related logs (test execution, AI conversation, browser actions)
• Collapsible sections in both local (if possible?) and CI output
• Clear visual hierarchy

Example hierarchy:

File: login.test.ts
  ├─ Test: Login Flow
  │  ├─ Browser Setup
  │  ├─ AI Conversation
  │  │  ├─ Prompt
  │  │  └─ Response
  │  └─ Browser Actions
  │
  └─ Test: Logout Flow
     ├─ Browser Setup
     ├─ AI Conversation
     └─ Browser Actions

File: dashboard.test.ts
  └─ Test: View Invoices
     ├─ Browser Setup
     └─ AI Conversation

GitHub Actions integration

• Use proper annotations for errors/warnings

  ::debug::Debug message
  ::warning file={name},line={line}::{message}
  ::error file={name},line={line}::{message}
  ::notice::General information
  

• Grouping

  ::group::Group title
  log content...
  ::endgroup::
  

Summary output

Normal output:

Tests: 3 passed, 1 failed, 4 total
Time:  2.45s
Token usage: ≈ $0.15 (2,345 tokens)

CI output:

::group::Test Summary
::set-output name=total_tests::4
::set-output name=failed_tests::1
::set-output name=total_time::2.45
::set-output name=total_cost::0.15
::set-output name=total_tokens::2345
::endgroup::

Implementation

• Keep existing TestResult tracking
• Add log level filtering
• Detect CI environment automatically
• Format output based on environment
• Timestamps only in debug mode
• Deprecate --debug-ai gracefully

Atomic PRs

1. Basic log levels:

• Add LogLevel type and logger options
• Implement level filtering
• Add --log-level CLI flag
• Make --debug-ai an alias

2. Output formatting

• Normal output formatting
• Debug output formatting
• Keep existing test result tracking

3. CI integration

• Add CI detection
• GitHub Actions annotations
• Group formatting
• Test summary output

4. Clean-up

• Remove --debug-ai flag
• Update documentation
• Add examples

Resources

• https://github.blog/news-insights/product-news/a-better-logs-experience-with-github-actions/
• https://octocat.dev/posts/Groups-and-formatting-in-GitHub-Actions

Why

Currently debugging relies on a simple —debug-ai flag. We need a more robust logging system that follows standard practices and integrates well with CI.

[gladyshcodes]

This is awesome! We should perhaps also add an ESLint plugin to make sure everyone uses our Logger instead of console.log. That'll keep our logging clean and tidy. Plus, if we log an error that's our fault, we can add a friendly message asking the user to report the bug on GitHub. This'll make it super easy for them to help us fix it