Review and improve agent effectiveness in Copilot Studio

The Analytics page in Copilot Studio provides an aggregated insight into the overall effectiveness of your agent across analytics sessions.

There are three core areas to focus on when reviewing and improving agent effectiveness:

• Outcomes and engagement: Knowing the end result of a conversation helps you begin to identify where your agent is succeeding and where it needs improvement.
• Knowledge source use: Seeing how often your knowledge sources are used helps you understand how well your agent is able to provide answers to user questions.
• Action use: Learning how often actions are used and how often they succeed can help you understand if those actions are useful and successful for users.
• User feedback: Reviewing user feedback helps you identify new user scenarios and issues and make improvements based directly on what your users are asking for.

You can view analytics for events that occurred in the last 360 days.

Outcomes and engagement

The Outcomes and engagement section shows a chart that tracks the type of outcome for each session between your agent and users.

To open a side panel with a pie chart breakdown of session outcomes, along with the top topics that led to each outcome, select See details on the chart.

A session falls into one of the following two states:

• Unengaged: A session starts when a user interacts with your agent or the agent sends a proactive message to the user. The session begins in an unengaged state.
• Engaged: A session becomes engaged when one of the following occurs:
  • a non-system topic is triggered
  • the session is escalated
  • the fallback topic is triggered
  • the conversational boosting topic is triggered

Once the session becomes engaged, it remains engaged. An engaged session has one of the following outcomes:

• Escalated: A session ends and is considered escalated when the Escalate topic is triggered or a Transfer to agent node is run (the current analytics session ends, whether the conversation transfers to a live agent or not).
• Resolved: A session ends and is considered resolved when the End of Conversation topic is triggered and the user confirms that the interaction was a success or lets the session time out.
• Abandoned: A session ends and is considered abandoned when an engaged session times out after 36 minutes and didn't reach a resolved or escalated state.

You can also set the outcome for actions with the conversationOutcome parameter using the action code editor. For example, conversationOutcome: ResolvedConfirmed for confirmed success or conversationOutcome: ResolvedImplied for implied success.

See the guidance documentation Measuring copilot engagement for suggestions and best practices on how to measure and improve engagement.

Knowledge source use

The Knowledge source use section shows a chart and metrics that track how often your knowledge sources are used in conversations between your agent and users.

It also shows trend indicators for how often your sources are being used, how many errors are being generated, and how many times the source is being used to provide answers to user questions.

The chart displays the top five knowledge sources used over the date range defined at the top of the Analytics page.

Select See details to open a side panel with knowledge source usage and error rates over your selected time period. You can use these charts to identify which knowledge sources work well to help users, and which to target for improvements.

• Knowledge source use shows the percentage of sessions that used each knowledge source the agent has access to.
• Errors shows the percentage of sessions that used each knowledge source type (for example, SharePoint) that resulted in an error.

Action usage

The Action use section shows a chart and metrics that track how often your actions are started over time, and how often those actions complete successfully.

It also shows trend indicators for how often your actions are being used and the percentage of called actions that completed successfully.

The chart displays the top five actions used over the date period defined at the top of the Analytics page.

To open a side panel with a list of all actions used in the specified time period, along with trend indicators, select See details on the chart.

User feedback

The User feedback section shows a chart and metrics that track the average customer satisfaction (CSAT) scores for sessions in which customers respond to an end-of-session request to take the survey.