Lead PySpark Developer

About the role

DXC Luxoft is hiring a Lead PySpark Developer in Warsaw for a project supporting the Surveillance Enhancement Program (SEP), a multi-year initiative that improves an automated surveillance platform used to detect market misconduct, manipulation and abuse. The project focuses on alert enhancements, data sourcing from FM order and trade lakes, and user interface and workflow improvements to meet regulatory obligations such as Market Abuse Regulation and Dodd-Frank.

Key responsibilities

• Develop software applications based on business requirements and maintain them with enhancements according to project specifications.
• Take part in requirement analysis, design, development, testing and implementation.
• Propose new techniques and technologies, perform unit and user acceptance testing, and ensure deadlines are met.
• Follow coding standards and best practices, and assist junior developers.
• Perform code reviews, review business requirements and recommend changes, and write coding documentation and technical specifications.
• Act as primary contact for development queries, analyse and resolve issues, and develop data pipeline architectures for concurrent processing.

Required profile

• 10+ years of experience in data-intensive PySpark development, window functions, performance optimisation, Spark architecture, salting and column pruning.
• Proven experience leading a development team and as a core Python developer.
• Good understanding of Gen AI.
• Ability to translate business requirements into technical ones and to design concurrent data pipelines.
• English at C1 level (advanced).

Required skills

• Python, PySpark, pandas, numpy, lambda functions, decorators, vector operations on dataframes/series.
• Data loading, transformation and calculation, and DB connectivity.
• Git, service-oriented architecture and integrations with other applications.
• Nice to have: Django, PostgreSQL, Apache Airflow.

What we offer

• Global relocation options and experience in an international environment.
• Annual holiday of 20 or 26 days depending on seniority, plus maternity, parental and paternity leave.
• Training through the Luxoft Training Center, cloud programs and leadership programs.
• Wellness benefits, private healthcare insurance, dental and travel insurance, and a multisport card.
• Internal mobility between projects and accounts.