Lead PySpark Developer

About the role

Luxoft is hiring a Lead PySpark Developer in Wroclaw, Poland, for a project supporting the Surveillance Enhancement Program (SEP). The SEP is a multi-year initiative to improve an automated surveillance platform that identifies potential market misconduct, manipulation and abuse, in line with regulatory obligations such as Market Abuse Regulation and Dodd-Frank. The project covers alert enhancements, data sourcing from FM order and trade lakes, and user interface and workflow improvements.

Key responsibilities

• Develop software applications based on business requirements and maintain them with enhancements according to project specifications.
• Take part in requirement analysis, design, development, testing and implementation.
• Propose new techniques and technologies, review business requirements and recommend changes.
• Perform unit testing and user acceptance testing, and ensure tasks are completed on time.
• Follow coding standards and best practices, write coding documentation and technical specifications.
• Mentor junior developers, perform code reviews and act as the main contact for development queries.

Required profile

• 10+ years of experience in data-intensive PySpark development, window functions, performance optimization, Spark architecture, salting and column pruning.
• Proven experience leading a development team and as a core Python developer.
• Good understanding of Gen AI.
• Ability to translate business requirements into technical ones and to design architectures for concurrent data processing.
• English at C1 level (advanced).

Required skills

• Python, PySpark, pandas, numpy, lambda functions, decorators, vector operations on dataframes.
• Data loading, transformation and calculation, and data pipelines handling batches of 1-10 GB.
• Git, service-oriented architecture and integrations with other applications.
• Nice to have: Django, PostgreSQL, Apache Airflow.

What we offer

• Global relocation and experience in an international environment.
• Annual holiday of 20 or 26 days depending on seniority, plus parental, maternity and paternity leave.
• Training through the Luxoft Training Center, cloud programs and leadership programs.
• Wellness benefits, private healthcare insurance, dental and travel insurance, and internal mobility.