Workflow bulk testing guide – Assembled

Overview

Bulk testing allows you to evaluate your workflow's accuracy at scale by testing it against multiple cases simultaneously. This feature is essential for validating workflow performance before moving from "Agent review" to fully automated operation, ensuring your workflows trigger correctly and handle cases as expected.

Why Use Bulk Testing?

Build Confidence in Automation

• Validate workflow accuracy before going fully automated
• Test against dozens of cases in a single run
• Identify potential issues before they impact customers

Measure Performance

• Get clear metrics on true positives, false positives, and false negatives
• Understand how accurately your workflow triggers
• Track improvements after making changes

Regression Testing

• Ensure changes don't break existing functionality
• Validate workflow updates before publishing

Getting Started

Step 1: Build Your Test Case Library

Before running bulk tests, you need to collect test cases. There are two ways to do this:

Method 1: Add Cases During QA Review

1. Navigate to Quality > QA Review or the Monitor page
2. When reviewing tickets, look for cases that should or shouldn't have triggered your workflow
3. Click "Add test case" next to relevant tickets
4. Specify which workflow the case should run on (or indicate it shouldn't run on any workflow)
5. The system saves these ticket IDs to your workflow's test case library

Method 2: Add Cases from Monitor Page

1. Go to the Monitor page
2. Review recent workflow executions
3. Identify cases that were correctly or incorrectly handled
4. Add them as test cases using the same process above

Step 2: Access Bulk Testing

1. Navigate to Workflows in your main menu
2. Select the workflow you want to test
3. Click on the "Test" tab
4. You'll see the bulk evaluation interface

Improving Your Workflow with AI Suggestions

Before diving into bulk testing, consider using the "Get AI suggestions" feature to improve your workflow's trigger conditions:

How AI Suggestions Work

1. Access the Feature: In your workflow's trigger prompt section, click "Get AI suggestions"
2. Provide Context: You can optionally provide details about cases where your workflow is over-triggering or under-triggering
3. Get Recommendations: The system analyzes your current trigger prompt and provides specific improvements
4. Review Changes: You'll see a detailed analysis with recommended additions and deletions to your trigger conditions

When to Use AI Suggestions

• Over-triggering: Your workflow runs on cases it shouldn't handle
• Under-triggering: Your workflow misses cases it should handle
• New workflows: Getting help with initial trigger prompt creation
• After bulk test results: Using failed cases as examples to improve triggers

Example Usage

If your bulk test shows false positives, you can:

1. Click "Get AI suggestions"
2. In the context box, write: "This workflow is incorrectly triggering on cases about shipping delays, but it should only handle order cancellations"
3. Apply the recommended improvements
4. Run another bulk test to validate the changes

Configuring Your Bulk Test

True Positives (Cases Your Workflow Should Handle)

These are cases where your workflow should trigger and handle the customer inquiry.

Random Sample Option:

• Select "Random sample from assigned test cases"
• Set your sample size (recommended: 10-30 cases for initial testing)
• The system will randomly select from cases you've marked as appropriate for this workflow

Specific Cases Option:

• Choose "Specific case IDs"
• Enter the exact ticket IDs you want to test
• Useful for testing specific edge cases or scenarios

True Negatives (Cases Your Workflow Should NOT Handle)

These are cases where your workflow should not trigger.

Random Sample Options:

• "Random sample from test cases not assigned to this workflow" - Tests against a broad set of cases
• "Random sample of test cases assigned to other workflows" - Tests against cases meant for other workflows
• Set your sample size (recommended: 10-30 cases initially)

Specific Cases Option:

• Enter specific ticket IDs that should not trigger your workflow
• Helpful for testing known edge cases that should be excluded

Test Parameters

Other Workflows to Test Against:

• Select which other workflows should be included in the test
• This helps ensure your workflow doesn't conflict with others

Interaction Type:

• Choose whether to test from the initial customer interaction
• Most tests should use the initial interaction

Draft Mode:

• Enable this to test draft versions before publishing
• Useful for validating changes before they go live

Running the Test

1. Configure Your Test Settings as described above
2. Start Small: Begin with 10-20 test cases per category for your first tests
3. Click "Test" to start the bulk evaluation
4. Wait for Results - The process typically takes about 10 minutes to complete
5. Review the Results once the test finishes

Understanding Your Results

Key Metrics

True Positives (TP):

• Cases where your workflow correctly triggered when it should have
• Higher numbers indicate better performance

True Negatives (TN):

• Cases where your workflow correctly did NOT trigger when it shouldn't have
• Higher numbers indicate good selectivity

False Positives (FP):

• Cases where your workflow incorrectly triggered when it shouldn't have
• Lower numbers are better - these represent over-triggering

False Negatives (FN):

• Cases where your workflow failed to trigger when it should have
• Lower numbers are better - these represent missed opportunities

Calculating Accuracy

While the system shows raw counts, you can calculate key performance metrics:

Overall Accuracy = (TP + TN) / (TP + TN + FP + FN)

• Target accuracy varies by workflow type and business requirements
• Consult with your Customer Success Manager to determine appropriate thresholds

Precision = TP / (TP + FP)

• Measures how often your workflow is correct when it triggers

Recall = TP / (TP + FN)

• Measures how well your workflow catches relevant cases

Best Practices

Building Effective Test Cases

Cover Multiple Scenarios:

• Include normal cases, edge cases, and error scenarios
• Add regional or country-specific variations if applicable
• Test different customer language styles and complexity levels

Start Small, Build Gradually:

• Begin with 5-10 test cases per category
• Add more cases as you encounter them during regular QA review
• Build your test library organically over time

Document Expected Outcomes:

• Keep notes on why certain cases should or shouldn't trigger your workflow
• This helps when reviewing results and making improvements

When to Run Bulk Tests

Before Publishing Changes:

• Always test workflow modifications before they go live
• Target accuracy varies by use case - discuss appropriate thresholds with your team

Periodic Performance Checks:

• Consider monthly tests for critical workflows
• Check after significant changes to your knowledge base or business processes

Before Going Fully Automated:

• More comprehensive testing is crucial when moving from agent review to automation
• Test with larger sample sizes (30-50 cases) for high-confidence validation

Interpreting and Acting on Results

High False Positive Rate:

• Your workflow may be triggering too broadly
• Use the "Get AI suggestions" feature with examples of incorrectly triggered cases
• Consider adding more specific exclusion criteria

High False Negative Rate:

• Your workflow may be too restrictive
• Use "Get AI suggestions" with examples of missed cases
• Review and potentially broaden trigger conditions

Low Overall Accuracy:

• May indicate fundamental issues with workflow design
• Consider redesigning trigger logic or splitting into multiple workflows
• Review your training data and knowledge sources

Troubleshooting Common Issues

"No Test Cases Available"

• You need to add test cases first through QA Review or Monitor pages
• Start by identifying 5-10 cases that should trigger your workflow

"Test Taking Too Long"

• Large sample sizes can take 15-20 minutes
• Consider testing with smaller samples initially (20-30 cases)

"Unexpected Results"

• Review individual cases that performed poorly
• Use the "Get AI suggestions" feature to improve trigger conditions
• Verify test cases are correctly categorized

Advanced Testing Strategies

Progressive Testing

1. Start with a small, high-confidence test set (10-15 cases)
2. Achieve good accuracy on this set (consult your team for targets)
3. Gradually expand to larger, more diverse test sets
4. Maintain consistent performance as you scale

Using AI Suggestions Iteratively

1. Run initial bulk test
2. Identify patterns in false positives/negatives
3. Use "Get AI suggestions" with specific examples
4. Apply recommended changes
5. Re-test with the same cases to validate improvements

A/B Testing Workflow Changes

• Create two versions of your workflow
• Test both against the same set of cases
• Compare results to determine which performs better

Getting Help

If you encounter issues with bulk testing or need assistance interpreting results:

• Contact your Customer Success Manager
• Review individual failed cases to understand patterns
• Use the "Get AI suggestions" feature with specific problem examples

Remember: Bulk testing is your safety net for automation. Start small, build your test library gradually, and use AI suggestions to continuously improve your workflow's accuracy.