What's Changing
Microsoft has announced the public preview of the Microsoft 365 Copilot Agent Evaluations tool—a command-line interface (CLI) that enables developers to measure and improve the quality of custom agents built for Microsoft 365 Copilot. Available starting today, the free preview tool automates agent testing through Node.js-based workflows and integrates directly with CI/CD pipelines, allowing teams to establish quality gates before agent deployment.
Who's Affected & When
Affected Tenants: All Microsoft 365 organizations with Microsoft 365 Copilot licenses. The tool is available immediately to any tenant where a Microsoft 365 administrator explicitly enables it.
License Requirements: Users must hold an active Microsoft 365 Copilot license. The Evaluations tool itself is free during the public preview period.
Rollout Timeline: No mandatory rollout date has been announced. Organizations can opt in to the public preview now via the Microsoft 365 Developers hub. General availability (GA) date is not yet specified.
Regional Availability: The tool operates within a single Microsoft 365 tenant. Multi-tenant evaluation is not currently supported. Azure OpenAI endpoint accessibility depends on your network and regional deployment.
Architecture & How It Works
What This Means for Your Environment
For Development Teams: You now have an automated way to continuously test agent quality without manual testing. Integration into pull request workflows enables quality gates—preventing agents with low coherence or groundedness scores from deploying to production.
For Platform Admins: You must explicitly enable the tool at the tenant level before any developer can use it. This requires:
- Admin consent for the Agent Evaluations tool in your tenant
- Configuration of an Azure OpenAI endpoint (if using LLM-based evaluators)
- Monitoring of Azure OpenAI token consumption to manage costs
For Compliance & Security: The tool runs locally via CLI and communicates with your Azure OpenAI endpoint. No evaluation data is stored externally. However, test prompts and agent responses traverse Azure OpenAI during LLM-based scoring.
For Infrastructure Teams: Node.js 24.12.0 or higher is required to run the CLI. The tool integrates with CI/CD pipelines (GitHub Actions, Azure Pipelines, etc.) and outputs HTML artifacts. Plan for automated archival of scorecard reports for audit and version-control purposes.
Prerequisites & Setup
| Requirement | Details | Mandatory |
|---|---|---|
| Microsoft 365 Copilot License | Per-user subscription required for all evaluators | ✓ |
| Node.js Version | 24.12.0 or higher; older versions have async/await compatibility issues | ✓ |
| Deployed Agent | Active agent in Microsoft 365 Agents Toolkit running on the tenant | ✓ |
| Admin Consent | Tenant administrator must explicitly enable Agent Evaluations tool | ✓ |
| Azure OpenAI Endpoint | Required ONLY for LLM-based evaluators (Coherence, Groundedness) | If using LLM evaluators |
| Network Access | CLI must reach Azure OpenAI endpoint; verify firewall rules | If using LLM evaluators |
| GitHub Repo | Optional: samples at microsoft/m365-copilot-eval | ✗ |
Action Items
For Immediate Implementation (This Week):
-
Request Admin Enablement
Contact your tenant administrator or M365 service owner. Ask them to enable the Agent Evaluations tool at the organizational level. Provide this context: the tool is free during public preview and required for agent quality testing. -
Audit Node.js Versions
Check your development and CI/CD environments. Runnode --version. If you have Node.js below 24.12.0, schedule an upgrade. Confirm compatibility with your existing agent projects before updating. -
Identify Your Azure OpenAI Endpoint
If you plan to use LLM-based evaluators, locate your Azure OpenAI resource in the Azure Portal. Obtain the endpoint URL and verify it is accessible from your development network. Test connectivity from at least one machine. -
Clone the Sample Repository
Visitgithub.com/microsoft/m365-copilot-evaland review sample evaluation configurations. This will help you understand expected CLI syntax and evaluator setup before running against your production agents.
For Planning & Documentation (Next 2–3 Weeks):
-
Design Evaluation Baselines
Define what "quality" means for your agents. Establish baseline evaluation runs with representative test prompts. Create a reference scorecard to compare future agent versions against. -
Set Up CI/CD Integration
Add evaluation steps to your pull request validation workflows. Example: require Coherence score ≥ 0.8 and Groundedness ≥ 0.75 before merge. Configure artifact storage for HTML scorecards. -
Monitor Azure OpenAI Costs
Enable cost tracking on your Azure OpenAI endpoint. LLM-based evaluators consume tokens. Establish usage alerts to prevent unexpected cost overages during large-scale evaluation runs. -
Document Evaluator Selection
Create a team standard: which evaluators are required for each agent type? Code-based only for simple agents, LLM-based for complex conversational agents? Document this in your development guidelines.
Known Limitations & Caveats
Quick Reference: Key Evaluator Types
| Evaluator Type | Use Case | Speed | Accuracy | External Dependency |
|---|---|---|---|---|
| ExactMatch (Code-Based) | Deterministic outputs; structured data; JSON responses | Instant | Binary (pass/fail) | None |
| PartialMatch (Code-Based) | Fuzzy validation; keyword presence; semantic matching | Instant | Fuzzy (0–100%) | None |
| Coherence (LLM-Based) | Response clarity; logical consistency; readability | Seconds per prompt | Semantic (LLM-judged) | ✓ Azure OpenAI |
| Groundedness (LLM-Based) | Context alignment; factual accuracy; citation validation | Seconds per prompt | Semantic (LLM-judged) | ✓ Azure OpenAI |
Integration with CI/CD & DevOps
The Agent Evaluations CLI is designed for automation. Its outputs—HTML scorecards—are suitable for artifact storage in Azure Pipelines, GitHub Actions, or Jenkins. You can implement quality gates by:
-
Add Evaluation Step to Pipeline
In your PR validation workflow, invoke the CLI with your test-prompt configuration file after agent code is committed. -
Define Score Thresholds
Set minimum acceptable scores for each evaluator. Example: Coherence ≥ 0.8, Groundedness ≥ 0.75. Fail the build if scores drop below thresholds. -
Archive Scorecards
Store HTML reports as pipeline artifacts. Create a versioned folder structure to compare agent quality across iterations and releases. -
Monitor Trends
Track average scores per agent version over time. Identify regressions before they reach production.
Feedback & Support
This is a public preview feature actively seeking feedback. Report issues, suggest features, and share success stories via the microsoft/m365-copilot-eval GitHub repository. Your input will directly influence the general availability roadmap and final feature set.
For technical questions, consult the Microsoft 365 Developer Blog and official documentation at /microsoft365/copilot/agent-evaluations/overview.
