ADF Dataflow issue with source API query

Thanks for your response.

Below is the full JSON from ADF dataflow. The JSON is automatically created and structured by ADF.

{
    "name": "dataflow1",
    "properties": {
        "type": "MappingDataFlow",
        "typeProperties": {
            "sources": [
                {
                    "dataset": {
                        "referenceName": "monday_api",
                        "type": "DatasetReference"
                    },
                    "name": "source1"
                }
            ],
            "sinks": [],
            "transformations": [],
            "scriptLines": [
                "source(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     httpMethod: 'POST',",
                "     body: '{",
                "\"query\": \"query { boards(ids: someid) { items_page(limit: 1) { cursor items { id name column_values { column { title } text } } } } }\"",
                "}',",
                "     timeout: 30,",
                "     headers: ['Content-Type' -> 'application/json', 'API-Version' -> '2025-01', 'Authorization' -> 'Some API Token'],",
                "     paginationRules: ['supportRFC5988' -> 'true'],",
                "     responseFormat: ['type' -> 'json', 'documentForm' -> 'documentPerLine']) ~> source1"
            ]
        }
    }
}

@Daniel Baird

ADF’s low-code environment generates code based on UI settings, which may not meet strict API requirements like GraphQL’s JSON format. It misinterprets GraphQL queries as raw REST payloads due to the lack of built-in support.
JSON bodies are treated as string literals instead of proper JSON objects, causing extra escaping issues.
Pagination rules are applied based on REST standards, which do not align with GraphQL’s cursor-based pagination.

ADF automatically applies pagination rules, such as paginationRules: ['supportRFC5988' -> 'true'], even when they are unnecessary. However, GraphQL manages pagination within the query using cursors, making ADF's pagination settings incompatible. This mismatch can confuse the API, leading to errors like 400 Bad Request or 415 Unsupported Media Type.

Alternative solutions:

Manually override headers and body by using a parameterized dataset instead of relying on ADF’s auto-generated JSON to ensure correct formatting.
Use a Web Activity instead of a Mapping Data Flow to maintain full control over the request format and avoid ADF’s auto-formatting issues when sending raw JSON.

I hope the above information helps, If the issue continues, let me know, and we can look into other possible causes!

@Daniel Baird
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

@J N S S Kasyap Thanks for your response.

Just to give some additional background on this.

I built the Graph query in Monday.com API playground. The query works in the body of both web and copy data activities. Unfortunately, the schema column values are repeated from the source, and I need to map the column value {text} to their respective columns in Azure SQL table.

I have the process working in a pipeline fine by using an initial web activity to then store the cursor into a variable which then an Until activity runs a copy activity with another web activity/set variable to continue to update the cursor until the if condition is met which is cursor results to null.

The problem I have with this method is it takes approximately 15 seconds to run each process and to write the data using the limit to 1 result per cursor (this enables the rows to be wrote separately for the repeated column values). I have a large amount of data to query and write and would take days.

My idea was to use the dataflow so I could call the source REST API and then flatten the JSON to be able to then re-map the schema before its wrote to the Azure SQL table.

I have tried manually overriding the headers and body by using a parameterized dataset but it is failing with parse issues.

Can you provide an example of how to override the headers and body in the dataset?

Thanks

@Daniel Baird
Thanks for the providing additional information

Parameterize the body (GraphQL query) and headers so that they can be passed dynamically into the dataset when invoked in the data flow.

{
  "name": "monday_api",
  "properties": {
    "type": "RestDataset",
    "typeProperties": {
      "linkedServiceName": {
        "referenceName": "monday_com_api_linked_service",
        "type": "LinkedServiceReference"
      },
      "relativeUrl": "/v2/",
      "requestMethod": "POST",
      "requestBody": {
        "value": "@{activity('Get GraphQL Query').output.queryBody}"
      },
      "headers": {
        "Content-Type": "application/json",
        "API-Version": "2025-01",
        "Authorization": "@{linkedService().apiToken}",
        "Accept": "application/json"
      },
      "requestTimeout": "00:01:00",
      "paginationRules": {}
    }
  }
}

Since GraphQL handles pagination via cursors within the query, you might not need the standard REST pagination. You can leave it empty.

In the pipeline, first use a Set Variable activity to store the GraphQL query, then use a Web Activity to trigger the API call and store the cursor for pagination. Finally, leverage a Mapping Data Flow to flatten the API response and transform it into the desired format before loading it into SQL.

By using a Web Activity with a Set Variable for the query and using a parameterized REST Dataset, you'll have more control over how headers and bodies are formatted and passed. This method should allow you to get rid of the parsing issues and improve the speed of your data load, as you're directly interacting with the API

I hope the above information helps, If the issue continues, let me know, and we can look into other possible causes!