About Me
I'm an Associate Director and Data Engineering Capability Lead ANZ with 10+ years of experience spanning data engineering, analytics, and actuarial domains. Based in Adelaide, South Australia, I specialise in Microsoft Fabric, Azure, and modern medallion Lakehouse architectures.
My expertise lies in designing and delivering scalable, metadata-driven data platforms that optimise cloud performance and cost while driving measurable business impact. I write about what I build — from architecture trade-offs to undocumented platform behaviour.
Blog
Content has been referenced by Microsoft Engineering and shared across enterprise Microsoft Teams channels globally.
Delta Table Maintenance in Microsoft Fabric: The Runtime 2.0 Edition
Runtime 2.0 made most of my old maintenance advice the default, but Fabric still doesn't maintain your tables for you. I tested every default, benchmarked incremental liquid clustering against Runtime 1.3, and found a Data Factory incompatibility that breaks Copy upserts into Runtime 2.0 tables.
Fabric IQ: Gold Data and Ontology Aren't Enough
A well-modelled Gold layer and a fine-tuned ontology got every question right — until I asked one the data couldn't support, and the Data Agent guessed a confident, fabricated answer instead of admitting the gap. What fixed it, and why the plumbing alone isn't the whole story.
Delta Table Maintenance in Microsoft Fabric: A 2026 Practitioner's Guide
The definitive guide to Delta table maintenance in Microsoft Fabric — what the documentation says, where to find it, and what your notebooks should actually look like in 2026.
Technical Stack
Platforms & Tools
Languages & Development
Data Architecture
Methodologies
AI & Development Tools
Experience
Associate Director & Capability Lead ANZ - Data Engineering
Promoted to Associate Director and Data Engineering Capability Lead ANZ. Technical lead on a multi-million dollar banking engagement, pioneering and embedding human-directed, local AI agent engineering using Claude within VS Code on the client's codebase.
Senior Consultant & Capability Lead - Data Engineering
Promoted to Senior Consultant and Data Engineering Capability Lead in just over one year. Led the productisation of the Microsoft Fabric Accelerator, formalising patterns from client engagements into a reusable enterprise framework. Played a key role in winning a multi-million dollar banking tender, responding to 400+ requirements and presenting the Fabric Accelerator to CIO and executive leadership.
Consultant - Data Engineering
Built Microsoft Fabric data platforms implementing medallion Lakehouse architecture (Bronze, Silver, Gold). Developed metadata-driven ELT/ETL pipelines with performance optimisation for large-scale workloads. Reduced ingestion costs by 50% through pipeline optimisation. Achieved 80% cost savings by right-sizing capacities and optimising job scheduling. Designed CI/CD processes including Git branching strategy and automated deployment. Collaborated with Microsoft engineers to resolve platform stability issues and product bugs.
Manager - Data & Analytics
Led development of greenfield data analytics platform. Implemented new stack: Fivetran, GitHub, Azure SQL, dbt, Power BI, and Jupyter. Built executive dashboards for revenue, pricing, and budget tracking. Delivered machine utilisation and capacity analytics across 200+ devices spanning all clinics. Automated government KPI and radiologist CPD reporting. Conducted data requirements workshops with executives and clinical stakeholders.
Senior Manager Business Intelligence
Developed and implemented BI strategies aligned with organisational goals. Led cross-functional teams to design and deliver advanced analytics, data warehousing, and data solutions. Streamlined data collection and analysis processes, reducing processing times and improving data accuracy. Implemented new technologies and methodologies to enhance BI capabilities. Built partnerships with business stakeholders to expand BI reach and foster a data-driven culture across the organisation.
Featured Projects
delta-doctor
Open-source library of seven production-ready PySpark notebooks for automated Delta table maintenance on Microsoft Fabric. Covers Lakehouse health scanning, OPTIMIZE gating, VACUUM safety, deletion vector management, and session/property configuration across Bronze, Silver, and Gold layers. Free, Apache 2.0, import directly into any Fabric workspace.
DataEngBytes Adelaide
Co-founded and organise Adelaide's only dedicated data engineering meetup, part of the DataEngBytes national community. First event held May 2026 with ~50 attendees. Responsible for speaker curation, venue, sponsorship, and event delivery.
Microsoft Fabric Accelerator
Production-grade, end-to-end data ecosystem featuring metadata-driven ELT, medallion Lakehouse architecture (Bronze/Silver/Gold), and built-in alerting, auditing, and logging. Built by productising patterns from client engagements into a reusable, enterprise-scale accelerator with cost optimisation and performance tuning capabilities.
Fabric Cost Optimisation Study
Led comprehensive analysis of Microsoft Fabric self-hosted data movement, identifying 10x cost discrepancy compared to Azure Data Factory. Provided actionable feedback to Microsoft PMs influencing product strategy. Achieved 50% cost reduction through pipeline optimisation and 80% savings through capacity right-sizing.
Healthcare Analytics Platform
Built greenfield analytics platform for Benson Radiology with 200+ medical devices. Implemented modern data stack with Fivetran, dbt, Azure SQL, and Power BI. Delivered executive dashboards for revenue tracking, machine utilisation analytics, and automated government KPI reporting.
Certifications
Microsoft Certified: Fabric Analytics Engineer Associate
Education & Awards
Graduate Certificate in Management
Actuarial Exams
Graduate Certificate in Commerce
Bachelor of Journalism
One Young World Delegate
FSC/Deloitte Future Leaders Award - Finalist
Get In Touch
I'm always interested in hearing about new opportunities, collaborations, or discussing data engineering challenges. Based in Adelaide, South Australia. Feel free to reach out!
Email: bradcoles@outlook.com.au
Phone: +61 423 237 043
Location: Aldgate, South Australia