Back to search

Lead Data Engineer

London Interim £650 - £880 per day View Job Description
We are seeking an experienced Lead Data Engineer with deep expertise in Python, Databricks and Spark to lead the delivery of large-scale data engineering initiatives.This is a unique opportunity for a technical leader who enjoys combining hands-on engineering with project delivery, team leadership and stakeholder engagement in a complex enterprise environment.
  • Lead major cloud data initiatives while remaining hands-on technically.
  • Work with Databricks, Python and Spark at enterprise scale.

About Our Client



Our client is a leading international financial institution with a long-established presence across major global markets. Serving a diverse client base that includes corporates, financial institutions and investors, the organisation delivers a broad range of banking, financing and capital markets services.

With significant investment in digital transformation and data-led innovation, the organisation is modernising its technology estate and expanding its enterprise data capabilities. Data plays a critical role in supporting business operations, regulatory obligations, analytics and strategic decision-making, making this an exciting opportunity to contribute to large-scale, business-critical data initiatives within a complex global environment.

Job Description

    • Lead the end-to-end delivery of data engineering projects, ensuring successful outcomes against business objectives, timelines and quality standards
    • Design, develop and optimise scalable data pipelines using Python, Databricks and PySpark
    • Build and enhance ETL/ELT workflows, reusable frameworks and modern data engineering solutions
    • Provide technical leadership and architectural guidance across data platform initiatives
    • Design and govern Lakehouse architectures leveraging Databricks and Delta Lake
    • Manage sprint planning, backlog prioritisation and delivery tracking within Agile teams
    • Identify and mitigate delivery risks, dependencies and technical challenges
    • Troubleshoot complex data, platform and performance-related issues
    • Drive engineering best practices across coding standards, testing, performance optimisation and code reviews
    • Act as a key interface between business stakeholders and engineering teams, translating requirements into scalable technical solutions
    • Collaborate with architects, product owners and senior stakeholders to shape delivery roadmaps and technical direction
    • Lead, mentor and develop a team of data engineers, fostering a culture of ownership and engineering excellence
    • Ensure data quality, governance, reliability and security across data platforms
    • Implement monitoring, logging and alerting capabilities to support operational excellence
    • Champion modern engineering practices, including CI/CD, DevOps and automation
    • Support production environments and drive continuous improvement across data solutions



The Successful Applicant



You will bring:

  • 10+ years' experience in data or software engineering
  • Proven experience as a Lead Data Engineer, Technical Lead, Engineering Lead, or similar hands-on leadership role
  • Strong hands-on expertise in Python
  • Extensive experience with Databricks, including workflows, notebooks and Delta Lake
  • Strong experience with Apache Spark / PySpark
  • Experience building and optimising enterprise-scale ETL pipelines
  • Strong understanding of modern data architecture and Lakehouse concepts
  • Advanced SQL and distributed data systems expertise
  • Experience with cloud platforms such as AWS, Azure or GCP
  • Experience working within Agile delivery environments
  • Excellent stakeholder management and communication skills
  • Ability to balance technical leadership with hands-on delivery responsibilities
  • Experience delivering data engineering solutions within Banking, Financial Services, Capital Markets, Insurance, or other highly regulated enterprise environments would be highly advantageous



Desirable experience includes:

  • Databricks certifications
  • Kafka or Structured Streaming
  • CI/CD and DevOps practices
  • Airflow or Databricks Workflows
  • Delta Lake optimisation techniques
  • Docker and Kubernetes
  • Exposure to machine learning pipelines

What's on Offer

  • Competitive day rate of £650-880 per day Inside IR35 (Umbrella)
  • Initial 6-month contract
  • Hybrid working arrangement - 2/3 days onsite
  • Opportunity to lead strategic data transformation initiatives
  • Access to modern cloud and data engineering technologies
  • Significant influence over technical direction and delivery
  • Collaborative, high-performing engineering environment



Contact
Hannah Kelly
Quote job ref
JN-072026-7064486
Phone number
+44 7989225567

Job summary

Job function
Technology
Subsector
Data Warehousing
Sector
Financial Services
Where
London
Contract type
Interim
Consultant name
Hannah Kelly
Consultant phone
+44 7989225567
Job reference
JN-072026-7064486