Request DefaultAzureCredential Support on Fabric Notebook

[tinadu0806]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Currently, the DefaultAzureCredential class does not function within the Fabric notebook environment. This limitation particularly affects the Fabric PySCOPE notebooks (currently in private preview), where users cannot leverage DefaultAzureCredential to create authenticated sessions for PySCOPE operations. For example, the common pattern of initializing a PySCOPE authenticated session with ps.AdlScopeJobManagementClient.create_session(DefaultAzureCredential()) is widely used when running PySCOPE locally but is currently not supported in Fabric notebooks.

For the majority of PySCOPE users, who are familiar with this authentication mechanism on local machines, it is important that we aim for seamless code portability. Our goal is to enable the same code to run in Fabric PySCOPE notebooks without requiring any modifications compared to local execution. This consistency is key to improving user experience and minimizing the friction associated with migrating workloads to the Fabric environment.

In addition to user-facing APIs such as create_session, our internal PySCOPE telemetry collection also relies on DefaultAzureCredential to authenticate with Azure Monitor. Due to the current lack of support, we have to disable telemetry collection when running in Fabric notebooks, which impacts our ability to gather critical usage and performance data on Fabric notebooks.

Beyond PySCOPE, DefaultAzureCredential is widely adopted across numerous libraries, including popular ones like pyarrow and fsspec, which are commonly used for data processing, storage, and machine learning tasks. In the Fabric notebook environment, which is designed to support a broad range of Python use cases, the lack of DefaultAzureCredential support could negatively impact users who rely on these libraries for accessing Azure resources. The absence of seamless authentication may introduce friction in setting up credentials manually, creating an additional barrier to adoption for those using Fabric notebooks for large-scale data workflows or machine learning projects.

By addressing this limitation and enabling support for DefaultAzureCredential, Fabric notebooks can offer a more consistent and integrated experience for users, allowing them to easily transition their existing workflows and take full advantage of the Azure ecosystem without modification to their code.

Describe the solution you'd like
Develop an AzureFabricCredential tailored for the Fabric notebook environment, leveraging the Fabric token library for authentication. Integrate this credential into the DefaultAzureCredential chain, allowing DefaultAzureCredential to seamlessly retrieve and utilize AzureFabricCredential when running in Fabric notebooks. This approach ensures a consistent and unified authentication experience across both Fabric and other Azure environments, enabling users to authenticate without modifying their existing code.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.