LLM Schema Reference

# Revyl YAML Test Schema - LLM Reference

## Purpose

This document provides a structured, machine-readable reference for generating Revyl YAML tests programmatically.

## Critical Behavior for Test Generation

**IMPORTANT**: When generating tests, DO NOT include manual navigation at the start:
- **Web tests**: Automatically navigate to build URL at test start
- **Mobile tests**: Automatically open the app at test start

**Manual navigation steps are ONLY needed for**:
- `navigate`: Navigating to a DIFFERENT URL mid-test (web only)
- `open_app`: Reopening app after `close_app` (mobile only)

**Correct pattern**: Tests should start with `instructions`, `validation`, or `extraction` blocks.

## Complete Schema Structure

    test:
      metadata:
        name: string                    # Required. Test name
        platform: enum                  # Optional. Values: "web" | "ios" | "android"

      build:
        name: string                    # Required. Build variable name
        pinned_version: string          # Optional. Specific version

      blocks:
        - # Block objects (see Block Type Definitions below)

## Block Type Definitions

### instructions
    type: "instructions"
    step_description: string          # Required. Natural language instruction

### validation
    type: "validation"
    step_description: string          # Required. Condition to validate

### extraction
    type: "extraction"
    step_description: string          # Required. What to extract
    variable_name: string             # Required. Use kebab-case

### manual
    type: "manual"
    step_type: enum                   # Required. See values below
    step_description: string          # Required for some step_types

**step_type values (Web)**: `wait`, `navigate`, `back`, `forward`, `refresh`, `browser_code_execution`
**step_type values (Mobile)**: `wait`, `open_app`, `close_app`

**step_description requirements by step_type**:
- `wait`: Must be a NUMBER representing seconds (e.g., "2", "5", "10")
- `navigate`: Must be a URL (e.g., "https://example.com")
- `browser_code_execution`: Must be valid JavaScript code
- `back`, `forward`, `refresh`, `open_app`, `close_app`: step_description is optional

**Examples**:

    # Wait - step_description must be a number
    - type: manual
      step_type: wait
      step_description: "3"           # Wait 3 seconds

    # Navigate - step_description must be a URL
    - type: manual
      step_type: navigate
      step_description: "https://checkout.example.com"

**IMPORTANT**: Do NOT use `navigate` or `open_app` at test start.

### if
    type: "if"
    condition: string                 # Required. Natural language condition
    then: array                       # Required. Array of blocks
    else: array                       # Optional. Array of blocks

### while
    type: "while"
    condition: string                 # Required. Loop condition
    body: array                       # Required. Array of blocks

## Variable System

**Definition** (in extraction block):

    - type: extraction
      step_description: "The product price"
      variable_name: product-price

**Usage** (in any step_description):

    - type: validation
      step_description: "Total is {{product-price}}"

## High-Level Instructions (Recommended for Complex Flows)

When there is indeterminism about what actions need to be taken, **use high-level instructions** instead of specifying every step. The AI will intelligently handle all the intermediate steps.

**Example - Instead of guessing every step**:

    # DON'T do this - too specific and may not match actual flow
    - type: instructions
      step_description: "Click sign up button"
    - type: instructions
      step_description: "Enter email in email field"
    - type: instructions
      step_description: "Enter password in password field"
    # etc... you may not know all the steps

**DO this - Use high-level instruction**:

    - type: instructions
      step_description: "Complete the sign up flow"

The system will automatically navigate through all required screens and states without needing to know the exact flow in advance.

**When to use high-level instructions**:
- Multi-step flows where exact screens are unknown (sign up, checkout, onboarding)
- Flows that may vary based on user state
- Complex processes where you want the AI to handle the details

## Validation Best Practices

When you don't know exactly what to verify, **use broad validations** instead of guessing specific components.

**Broad validation examples**:

    # General state validations - GOOD
    - type: validation
      step_description: "The checkout page is displayed"

    - type: validation
      step_description: "Login was successful"

    - type: validation
      step_description: "The user is on the dashboard"

**Negative validations (what shouldn't be visible)**:

    # Verify errors are NOT shown - GOOD
    - type: validation
      step_description: "No error message is displayed"

    - type: validation
      step_description: "The login screen is no longer visible"

**Avoid overly specific validations when uncertain**:

    # DON'T guess specific components you're not sure about
    - type: validation
      step_description: "The blue confirmation button in the top right is visible"

    # DO use general validations
    - type: validation
      step_description: "The confirmation screen is displayed"

## Test Generation Rules

1. **NEVER start with manual navigation** - auto-navigate to build URL/app
2. **First block should be**: instructions, validation, or extraction
3. **Use high-level instructions for complex flows** - "Complete the checkout process" instead of specifying every step
4. **Use broad validations when uncertain** - "Login was successful" instead of guessing specific UI elements
5. **Negative validations are powerful** - "No error message is displayed" confirms success
6. **Wait step_description must be a NUMBER** - Use "3" not "Wait for page to load" (represents seconds)
7. **Use navigate/open_app ONLY when**: Changing URL/reopening closed app mid-test
8. **Always include**: test object, metadata.name, build.name, blocks array
9. **Variables**: Define with extraction, reference with `{{variable}}`
10. **Platform awareness**: Only use platform-appropriate step_types

## Complete Example

    test:
      metadata:
        name: "E-commerce Purchase"
        platform: web

      build:
        name: "Shopping Site"

      blocks:
        # Test auto-starts at build URL - NO navigate needed

        - type: instructions
          step_description: "Search for 'laptop'"

        - type: instructions
          step_description: "Click first result"

        - type: extraction
          step_description: "The product price"
          variable_name: price

        - type: instructions
          step_description: "Click 'Add to Cart'"

        - type: validation
          step_description: "Cart shows price {{price}}"

        - type: if
          condition: "Discount field is visible"
          then:
            - type: instructions
              step_description: "Enter 'SAVE10'"

        - type: instructions
          step_description: "Click 'Checkout'"

        - type: validation
          step_description: "Checkout page displayed"

## Common Validation Errors to Avoid

1. Starting test with unnecessary manual navigation (navigate/open_app)
2. Using descriptive text for wait instead of a number - "3" not "Wait briefly"
3. Missing required fields (type, step_description, etc.)
4. Invalid block type values
5. Invalid step_type for platform
6. Missing variable_name in extraction blocks
7. Empty then/body arrays in if/while blocks