This is the third post in my series on Fabric Spark Notebooks and CU consumption. In Part 1, we covered the fundamentals of CU consumption in Fabric Spark notebooks. In Part 2, we explored running multiple notebooks on the same capacity and how Fabric handles concurrency, bursting, and job queueing.
In this post, we’ll focus on High Concurrency Mode in Fabric Spark, a feature that allows multiple notebooks to reuse the same Spark session, helping to improve performance and optimize CU usage.
Why High Concurrency Mode Matters
By default, each Fabric Spark notebook runs in its own isolated compute session. This means:
- Every notebook execution creates a new Spark session, consuming additional capacity.
- Running multiple notebooks simultaneously allocates separate compute clusters, even if they are doing similar work.
- This results in higher CU consumption and increased resource overhead.
High Concurrency Mode solves this by allowing multiple notebooks to share the same Spark session, reducing redundant resource usage and improving efficiency.
How to Enable High Concurrency Mode
Fabric defaults to Standard Sessions, but you can switch to High Concurrency Mode directly from the notebook toolbar.
Steps to enable High Concurrency Mode
- Open a Fabric Spark Notebook.
- Click “Connect” in the notebook toolbar.
- Select “New high concurrency session” instead of “New standard session.”
- If an active high concurrency session exists, you can connect to it instead of creating a new one.
Once enabled, multiple notebooks can attach to the same Spark session, reducing startup time and optimizing compute usage.
How High Concurrency Mode Optimizes CU Usage
The key advantage of High Concurrency Mode is session reuse. Instead of every notebook launching a new Spark cluster, multiple notebooks can share the same compute allocation.
Example Scenario
Consider a Fabric F4 capacity, which provides:
- 8 Spark vCores
- Bursting up to 24 Spark vCores
Without High Concurrency Mode (Standard Sessions)
- Each notebook runs in its own session.
- A Medium Starter Pool session requires 16 vCores.
- On an F4 capacity, only one notebook session can run at a time.
- Any additional notebooks will hit capacity limits or be queued.
With High Concurrency Mode
- Multiple notebooks share the same Spark session.
- If three notebooks use High Concurrency Mode, they consume the resources of one Spark session instead of three separate ones.
- This allows more notebooks to run concurrently on the same Fabric capacity.
Limitations of High Concurrency Mode
While High Concurrency Mode helps optimize compute resources, it does have some limitations:
- Sessions are user-specific – You cannot share a session between different users.
- Notebooks must have the same default Lakehouse configuration – If different Lakehouses are used, separate sessions will be created.
- Maximum of five notebooks can be attached to a single High Concurrency session.
These constraints mean that while High Concurrency Mode improves efficiency, it does not function as a fully shared cluster across multiple users.
When Should You Use High Concurrency Mode?
Recommended Use Cases
- Running multiple notebooks interactively – Reduces startup time and optimizes CU usage.
- Workloads that involve iterative development – Useful when testing multiple scripts in the same session.
- Efficient use of limited Fabric capacity – Helps when working on smaller SKUs (F2, F4, or F8) where capacity is constrained.
When Not to Use High Concurrency Mode
- Different Lakehouse configurations – If notebooks reference different Lakehouses, separate sessions will be required.
- Multi-user scenarios – Since sessions cannot be shared between users, High Concurrency Mode does not work for team-wide workloads.
| Use Case | Standard Session | High Concurrency Mode |
|---|---|---|
| Running a single notebook | ✅ Best choice | ❌ Not necessary |
| Multiple notebooks running concurrently | ❌ Each creates its own Spark cluster (high CU usage) | ✅ Notebooks share the same compute session |
| Interactivity & performance | ⚠ Slower startup time for each new session | ✅ Fast execution if a session is already active |
| CU optimization on limited capacity | ❌ Higher CU burn | ✅ More efficient CU usage |
Key Takeaways & Best Practices
- Use High Concurrency Mode for interactive workloads to reduce CU consumption and improve execution speed.
- Standard Sessions are the default, but switching to High Concurrency Mode can optimize capacity usage.
- Monitor active sessions to ensure efficient resource allocation.
- Understand the limitations – High Concurrency Mode is user-specific and limited to five attached notebooks per session.
- For production workloads, scheduled jobs, or multi-user environments, standard sessions may still be the better choice.
Final Thoughts & What’s Next
High Concurrency Mode is a powerful yet underused feature in Fabric Spark. It enables better resource utilization, reduces CU costs, and improves execution times, making it a valuable option for interactive and development workloads.
This is Part 3 of my Fabric Spark series. Next, I’ll be demystifying the monitoring of Spark in Fabric, covering:
- Monitor Hub
- Run Details
- Execution Breakdowns
- CU usage in Capacity Metrics
If you found this post helpful, let me know—how do you use High Concurrency Mode in your Fabric Spark workloads?
Pingback: Running Multiple Spark Notebooks on the Same Capacity – justB smart