Senior Data Platform Engineer (Microsoft Fabric)

About the role

Software Mind is hiring a Senior Data Platform Engineer in Warsaw for a client building a next-generation risk analysis and scoring platform on Microsoft Fabric. The role owns the medallion data architecture end-to-end, designing and building Bronze, Silver and Gold Delta tables, Fabric Spark feature pipelines, Direct Lake semantic models and physical-layout optimizations. It is described as the highest-utilization engineering role on the project, working alongside a Senior Technical Lead, a Senior DevOps engineer and a mid-level developer.

Key responsibilities

• Design and implement Bronze, canonical Silver and Gold Delta table schemas with PAN-safe physical layout, including statistics allowlist, liquid clustering, retention settings and V-Order optimization.
• Build Fabric Spark notebooks for the feature pipeline (rolling velocity features, merchant-level features, hierarchical rollups, cross-merchant features).
• Author per-flavor Silver adapters, one function per flavor, unioned by name and tested against golden fixtures.
• Build the canonical MERGE DAG (Bronze to Silver) and the Gold materialization pipeline for fact_transaction_risk and fact_merchant_risk, with idempotent merge semantics.
• Design Direct Lake semantic models and test them against representative data volumes.
• Configure OneLake namespace layouts, workspace catalogs and cross-workspace shortcuts, and establish idempotent Delta write patterns using delta-rs from Python containers.
• Handle Fabric capacity sizing (F-SKU selection, load testing, reservation versus pay-as-you-go) and partner with DevOps on Fabric IaC (Terraform capacity provisioning and workspace bootstrap).
• Advise on ClickHouse contingency if Direct Lake latency is insufficient for client-facing UIs.

Required profile

• 5+ years of data engineering experience, with at least 2 years of hands-on Spark development (PySpark preferred).
• Deep knowledge of Delta Lake internals (transaction log, checkpoints, retention, VACUUM, OPTIMIZE, deletion vectors, statistics).
• Experience with Microsoft Fabric workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake and SQL analytics endpoint.
• Ability to communicate architecture clearly in writing and to work autonomously on well-scoped features.

Required skills

• SQL performance tuning for OLAP (Fabric) and OLTP (PostgreSQL) workloads.
• Strong Python data engineering skills (pandas, PyArrow, delta-rs).
• Parquet format expertise (statistics allowlists, row-group sizing, dictionary encoding, column indexes).
• Nice to have: PCI-scoped data platforms, financial services or payments background, Power BI DAX, published data-governance patterns (SCD2, natural-key idempotency), OneLake shortcuts, ClickHouse, Terraform for Azure data resources, and fluid ingestion patterns.

What we offer

• Flexible and remote cooperation options.
• International projects with leading global clients and related travel.
• Language classes, knowledge-sharing initiatives, private healthcare and life insurance, and a multisport card.