Spark tuning has a way of chewing up time: you start with something that “should be fine,” performance is off, costs creep up, and suddenly you’re deep in configs, Spark UI, and tribal knowledge trying to figure out what actually matters. That’s why I’m excited to highlight sparkwise, an open-source Python package created by Santhosh Kumar Ravindran, one of my direct reports here at Microsoft. Santhosh built sparkwise to make Spark optimization in Microsoft Fabric less like folklore and more like a repeatable workflow: automated diagnostics, session profiling, and actionable recommendations to help teams drive better price-performance without turning every run into an investigation.

Gil Gerard's departure reminds us that some celebrities aren't just actors; they're the comforting echoes of our past. Buck Rogers was more than a show—it was a place that shaped our childhood optimism.

Microsoft Fabric simplifies Spark workload management but diagnosing performance issues remains challenging. This post introduces the "Job Doctor," an AI tool that analyzes Spark telemetry to identify problems like skew or excessive shuffles, generates human-readable diagnoses, and suggests fixes. The implementation integrates with Azure AI for optimized Spark job management.

As I head to the National for the first time, this is a topic I have been thinking about for quite some time, and a recent video inspired me to put this together with help from ChatGPT’s o3 model doing deep research. Enjoy! Introduction: Grading Under the Microscope Sports card grading is the backbone of the collectibles hobby – a PSA 10 vs PSA 9 on the same card can mean thousands of dollars of difference in value. Yet the process behind those grades has remained stubbornly old-fashioned, relying on human eyes and judgment.

1. Setting the Table Josh, I loved how you framed your conversation with ChatGPT-4o around three crisp horizons — 5, 25 and 100 years. It’s a structure that forces us to check our near-term expectations against our speculative impulses. Below I’ll walk through each horizon, point out where my own analysis aligns or diverges, and defend those positions with the latest data and research. 2. Horizon #1 (≈ 2025-2030): The Co-Pilot Decade Where we agree You write that “AI will write drafts, summarize meetings, and surface insights … accelerating workflows without replacing human judgment.” Reality is already catching up:

There’s a word haunting documents, cluttering up chat messages, and lurking in email threads like an uninvited character from Downton Abbey. That word is whilst. Let’s be clear: no one in the United States says this unironically. Not in conversation. Not in writing. Not in corporate life. Not unless they’re also saying “fortnight,” “bespoke,” or “I daresay.” It’s Not Just Archaic—It’s Distracting In American English, whilst is the verbal equivalent of someone casually pulling out a monocle in a team meeting. It grabs attention—but not the kind you want.

Note: Antonio McDyess is one of my favorite players that no one I know seems to know or remember, so I asked ChatGPT Deep Research to help tell the story of his rise to the cusp of superstardom. Do a YouTube search for McDyess highlights - it’s a blast. Humble Beginnings and Early Promise Antonio McDyess hailed from small-town Quitman, Mississippi, and quickly made a name for himself on the basketball court. After starring at the University of Alabama – where he led the Crimson Tide in both scoring and rebounding as a sophomore – McDyess entered the star-studded 1995 NBA Draft .

Author's note - I have enjoyed playing around with the Deep Research capabilities of ChatGPT, and I had it put together what it felt was the definitive whitepaper on Capacity Management for Microsoft Fabric. It basically just used the Microsoft documentation (plus a couple of community posts) to pull it together, so I'm curious what you think. I'll leave a link to download the PDF copy of this at the end of the post. Executive Summary Microsoft Fabric capacities provide the foundational compute resources that power the Fabric analytics platform. They are essentially dedicated pools of compute (measured in Capacity Units or CUs) allocated to an organization’s Microsoft Fabric tenant.

Keep up with the latest Microsoft Fabric updates, announcements, information, & new features on the Microsoft Fabric blog. Search by category or date published.