Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,369 questions
0 answers
Data Truncation Issue When Using COPY INTO in Databricks for Binary Column
Issue: When loading a long string value containing trailing null bytes (e.g., \x12\x34\x56\x00\x00\x00...) into a binary column using the COPY INTO statement, the data gets truncated, storing only the first few characters (\x12\x34\x56 → EjRW in…
asked 2025-03-14T18:51:59.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Suwetha Sivakumar
0
Reputation points
asked 2025-03-14T18:51:59.73+00:00
Suwetha Sivakumar
0
Reputation points
1 answer
Azure Databricks Workspace: Users are being added to the admins group on their own
We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…
asked 2024-11-01T10:04:34.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 72%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECO%3C/text%3E%3C/svg%3E)
Chris O'Brien
0
Reputation points
commented 2025-03-14T16:47:56.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
10,080
Reputation points Microsoft External Staff
5 answers
Allow only a specific Azure Databricks instance to connect to keyvault
I want to connect to Azure Keyvault using "Azure Key Vault-backed scopes" in Databricks. Now on the Keyvault side, the way we allow an application to connect is via: RBAC - We are using the RBAC permission model. Add an identity in RBAC and…
asked 2022-01-11T01:21:28.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 87%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Nidhi Singh
6
Reputation points
answered 2025-03-14T11:21:01.1533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 78%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Martyna Magdalena Sikorska
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Serverless not showing for my notebooks in Databricks
Hi. Serverless option is not showing for my notebooks and I cannot even see it in the feature enablement in accounts administration. In the attached screenshots, you can see that I have unity catalog enable and that serverless does not appear in feature…
asked 2025-01-26T12:10:14.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 39%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Mohammed Amine Mostefai
20
Reputation points
commented 2025-03-14T09:23:37.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 43%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Simon D'Morias
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to create a Materialized View in Azure Databricks stored in a Managed Catalog, based on data loaded from a foreign catalog
I am currently facing a problem with creating a materialized view in Catalog_2.Schema_B (Managed_Catalog) based on data I have connected from postgre SQL external source loaded into Catalog1_Schema_A (Foreign_Catalog). The error I get only says…
asked 2025-03-07T05:03:46.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 49%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
Kohdai Kurihara
40
Reputation points
accepted 2025-03-14T08:28:35.7166667+00:00
Kohdai Kurihara
40
Reputation points
0 answers
Connecting Spark to Azure SQL
Hi, I'm following this article to connect Spark to Azure SQL https://learn.microsoft.com/en-us/azure/azure-sql/database/spark-connector?view=azuresql&source=docs It says that I need to get the Spark connector and it has a link to a maven repo that…
asked 2025-03-12T10:06:58.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 36%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYB%3C/text%3E%3C/svg%3E)
Yaish, Batel
0
Reputation points
commented 2025-03-13T21:07:02.1666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
860
Reputation points Microsoft External Staff
1 answer
Azure Databricks
We have our production Azure Databricks resources configured with two subnets: one for the host and one for the container, both using a /27 CIDR block. As we plan to execute multiple jobs and only have 16 IPs remaining, we have decided to expand the…
asked 2025-03-11T13:19:59.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 34%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHK%3C/text%3E%3C/svg%3E)
Hemanth Kumar
0
Reputation points
commented 2025-03-13T20:28:53.1833333+00:00
Chandra Boorla
10,080
Reputation points Microsoft External Staff
1 answer
Help needed to sign onto Azure databricks
I recently created account to access Azure Databricks service. I created the service and the resource group and tried to access the databricks workspace using the private link (https://adb-4253786588898730.10.azuredatabricks.net). But I enter my…
asked 2025-03-13T19:24:48.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 8%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
SM
0
Reputation points
edited the question 2025-03-13T20:12:38.3433333+00:00
Chandra Boorla
10,080
Reputation points Microsoft External Staff
1 answer
Run query against delta table using ADF managed identity, but Insufficient privileges error
In ADF, I created a web activity to run query using rest api against deltalake table. Managed Identity is Authentication type table path is like this: catalogA.schemaB.tableC. catalogA permission is controlled by AD Group groupD. ADF managed…
asked 2025-03-13T19:32:40.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 66%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Ni, Shaojun SITI-PTIY/BAC
0
Reputation points
answered 2025-03-13T19:48:15.5766667+00:00
hossein jalilian
10,175
Reputation points
1 answer
Databrick: pros and cons
Hi, Is it possible to highlight five pros and five cons of using Databricks for data engineering? This would help me make a more informed decision about the tool
asked 2025-03-13T10:55:40.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 59%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Renan
60
Reputation points
answered 2025-03-13T18:56:05.8433333+00:00
Chandra Boorla
10,080
Reputation points Microsoft External Staff
0 answers
Databricks AI_QUERY fails with certain OpenAI models
We have been using successfully using the AI_QUERY function in Azure Databricks to interact with o1-mini and text-embedding-ada-002 models. However in the past few days, we have started to encounter errors. Calling our o1-mini model as usual…
asked 2025-03-12T17:41:44.71+00:00
James Bentley
10
Reputation points
commented 2025-03-13T11:35:37.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
5,045
Reputation points Microsoft External Staff
1 answer
How can I make sure that a table and csv file can merge in an azure databricks query?
I am currently trying to copy data from a csv file into my table in azure databricks and am getting the following error. [DELTA FAILED TO MERGE FIELDS] Failed to merge fields ‘transfer_month’ and ‘transfer_month’ I am unable to solve the error. I can…
asked 2025-03-09T03:42:05.1566667+00:00
Kohdai Kurihara
40
Reputation points
commented 2025-03-12T08:05:53.2966667+00:00
Ganesh Gurram
5,045
Reputation points Microsoft External Staff
1 answer
How do I use the azure databricks dlt pipeline to consume azure Event Hub data
Hi there, How do I use the azure databricks dlt pipeline to consume azure Event Hub data? EH_NAME = "myeventhub" TOPIC = "myeventhub" KAFKA_BROKER = "{EH_NAMESPACE}.servicebus.windows.net:9093" GROUP_ID =…
asked 2025-03-10T03:52:50.48+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 53%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZ%3C/text%3E%3C/svg%3E)
zmsoft
365
Reputation points
edited a comment 2025-03-12T04:30:28.01+00:00
Ganesh Gurram
5,045
Reputation points Microsoft External Staff
2 answers
One of the answers was accepted by the question author.
Need to connect to IMB DB2 from Azure databricks workspace
Need to connect to IMB DB2 (DB version IBM DB2 for i 7.3) form azure databricks workspace, the DB2 is hosted is AS400 server and DB port 8471. Please suggest on the step by step procedure to connect to DB and query a table, like installing required…
asked 2023-08-22T04:58:34.3733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 96%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
veerabhadra reddy kovvuri
181
Reputation points
commented 2025-03-11T21:34:52.1633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 33%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGR%3C/text%3E%3C/svg%3E)
Gummadavelli, Raghu
0
Reputation points
1 answer
manage account in azure databricks
Hi Guys hope you are fine. I am trying to login to https://accounts.azuredatabricks.net/login with my intra id and giving it same password of my azure portal acount but it is giving me the error that password is not correct. although it is working fine…
asked 2025-03-06T09:57:37.7066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 86%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Muhammad Awais Javed
20
Reputation points
commented 2025-03-11T11:06:49.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
21,410
Reputation points Microsoft External Staff
2 answers
One of the answers was accepted by the question author.
How do I add users from different domains to azure databricks
Hi there, How do I add users from different domains to azure databricks? Thanks&Regards, zmsoft
2 answers
How to become azure databricks expert?
How to become azure databricks expert? Any Microsoft offered courses apart from microsoft learn
asked 2025-03-08T22:15:26.7966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 9%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGD%3C/text%3E%3C/svg%3E)
Goddeti Dhanasekhara Reddy
20
Reputation points
edited the question 2025-03-11T07:50:44.5466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 24%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAA%3C/text%3E%3C/svg%3E)
Abiola Akinbade
24,175
Reputation points
2 answers
Users in a specific region are experiencing connection issues to Databricks workspaces
We have users that are experiencing issues with connecting to both our production and development environments of databricks through our Azure subscription. Everything else in Azure is working fine for them, though. We have other users in different…
asked 2025-03-03T18:21:41.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 47%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKA%3C/text%3E%3C/svg%3E)
Kiel Auer
0
Reputation points
commented 2025-03-10T15:15:55.0433333+00:00
Chandra Boorla
10,080
Reputation points Microsoft External Staff
3 answers
No Support for reading zip files store in Azure Datalake storage using abfss protocol url via python notebook.
Hi, This is so strange that microsoft have created the abfss protocol but have zero support to read zip files using abfss protocol via python notebook. For example I have this sample code that gives me hard time identifying a zip file as a zip…
asked 2025-03-03T11:08:21.7866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 56.00000000000001%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jeevesh
0
Reputation points
commented 2025-03-10T08:41:17.19+00:00
J N S S Kasyap
860
Reputation points Microsoft External Staff
1 answer
What are the differences between notebooks (written in SQL) and queries in Azure Databricks?
What are the differences between notebooks (written in SQL) and queries in Azure Databricks? I am particularly interested in what the differences are in terms of performance, billing and maintenance. If there are no significant functional differences, is…
asked 2025-03-06T01:01:02.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 50%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPB%3C/text%3E%3C/svg%3E)
Power BI Test User 2
0
Reputation points
commented 2025-03-10T05:31:02.3866667+00:00
Ganesh Gurram
5,045
Reputation points Microsoft External Staff