KQL query gives different output in Dev and prod environment

Question

KQL query gives different output in Dev and prod environment

Administrator Deepak Shaw 0

Hi,

I'm having identical test for logic apps and connected app insights in two sperate subscription (DEV and PROD)

I'm running below KQL in both envirnaments but getting no response from PROD Azure subscription. In short : KQL query gives different output in Dev and prod environments.

My query looks like:

let Failed = view () { 
    traces
    | where operation_Name contains "wf-sl-hm-" or operation_Name contains "wf-sf-hm-"
    | where message contains "Workflow run ends" 
    | extend d = parse_json(customDimensions)
    | where d.status contains "Failed"
    | summarize Count=sum(itemCount), impactedUsers=dcount(user_Id) by operation_Name
};
let Succeeded = view () {
    traces
    | where operation_Name contains "wf-sl-hm-" or operation_Name contains "wf-sf-hm-"
    | where message contains "Workflow run ends" 
    | extend d = parse_json(customDimensions)
    | where d.status contains "Succeeded"
    | summarize Count=sum(itemCount), impactedUsers=dcount(user_Id) by operation_Name
}; 
union withsource="HM-Snapshot" Failed, Succeeded

Can someone please advice what is the issue and fix.

Thanks

Deepak Shaw

Sai Prabhu Naveen Parimi 485 Reputation points Microsoft External Staff

2025-03-13T03:36:18.1533333+00:00
Administrator Deepak Shaw

Noticed the discrepancy with the KQL query results between Dev and Prod. Could you please perform the following checks in Prod to help identify the issue?

-->Check if data exists Run this to see if any data is coming through: traces

| where timestamp > ago(1d)

| take 10

If this returns nothing, there might be an issue with logs not reaching Application Insights. -->Verify operation_Name values Check if the expected workflows exist:

| summarize count() by operation_Name If wf-sl-hm- or wf-sf-hm- are missing, the workflow names might be different in Prod.

-->Confirm customDimensions.status is available Make sure the status field exists and is correctly structured: traces

| extend d = parse_json(customDimensions)

| summarize count() by tostring(d.status)

If status is missing, there might be a difference in logging configuration. -->Check App Insights and Logging Setup

Ensure logs are correctly sent to Application Insights in Prod.

Compare Diagnostic Settings with Dev.

Check if the Instrumentation Key is correct.

Additional Checks to Further Isolate the Issue

Since the query uses union, try running both Failed and Succeeded queries separately in both Dev and Prod. Compare the results to see if they match.

If there are any missing or unexpected results, check the Logic App run history in the specific environment and verify if the customDimensions property is populated correctly.

If custom dimension values are missing or different, troubleshoot by validating each action output in the Logic App run history.

Try these and let me know what you find. If the issue persists, we can dig deeper.

Your answer

Sai Prabhu Naveen Parimi 485 Reputation points Microsoft External Staff

2025-03-13T03:36:18.1533333+00:00

Administrator Deepak Shaw

Noticed the discrepancy with the KQL query results between Dev and Prod. Could you please perform the following checks in Prod to help identify the issue?

-->Check if data exists Run this to see if any data is coming through: traces

| where timestamp > ago(1d)

| take 10

If this returns nothing, there might be an issue with logs not reaching Application Insights. -->Verify operation_Name values Check if the expected workflows exist:

| summarize count() by operation_Name If wf-sl-hm- or wf-sf-hm- are missing, the workflow names might be different in Prod.

-->Confirm customDimensions.status is available Make sure the status field exists and is correctly structured: traces

| extend d = parse_json(customDimensions)

| summarize count() by tostring(d.status)

If status is missing, there might be a difference in logging configuration. -->Check App Insights and Logging Setup

Ensure logs are correctly sent to Application Insights in Prod.

Compare Diagnostic Settings with Dev.

Check if the Instrumentation Key is correct.

Additional Checks to Further Isolate the Issue

Since the query uses union, try running both Failed and Succeeded queries separately in both Dev and Prod. Compare the results to see if they match.

If there are any missing or unexpected results, check the Logic App run history in the specific environment and verify if the customDimensions property is populated correctly.

If custom dimension values are missing or different, troubleshoot by validating each action output in the Logic App run history.

Try these and let me know what you find. If the issue persists, we can dig deeper.

Share via

KQL query gives different output in Dev and prod environment

Your answer