Lead Data Engineer
The Lead Data Engineer will be the senior technical authority for Betfred's data platform, setting the architectural direction and engineering standards for the team. This hands-on role will focus on leading the technical delivery of the migration from SQL Server to an AWS lakehouse. The role also involves designing the real-time streaming architecture that underpins automation, business intelligence, and analytics, while mentoring engineers across the team.
- Own the technical architecture, standards and design patterns for the data platform, covering batch and streaming pipelines built on AWS Glue, AWS Lambda, AWS S3 and Redshift.
- Lead the technical delivery of the Data Warehouse migration from SQL Server to AWS, including target modelling, cutover approach, reconciliation strategy and technical risk management.
- Design and build the real-time data architecture using Amazon MSK / Apache Kafka and PySpark Structured Streaming (evaluating Apache Flink where it is the better fit), including event schema governance, exactly-once semantics and stream-to-lakehouse integration.
- Define and embed data quality, validation and observability frameworks, and hold the team to them through design and code review.
- Govern the lakehouse and data warehouse design, ensuring consistency, scalability and sensible cost characteristics across the platform.
- Own platform infrastructure-as-code (Terraform) and CI/CD standards, so environments and pipelines are reproducible and reviewable.
- Monitor and optimise AWS platform spend, making the cost and performance trade-offs explicit when proposing architecture.
- Act as the technical escalation point for complex data issues and performance bottlenecks, leading root-cause analysis and driving permanent fixes.
- Collaborate with data scientists, analysts and other engineering leads to support the deployment of machine learning and advanced analytical solutions.
- Mentor and technically coach data engineers through pairing, review and design sessions, and contribute to technical assessment during hiring.
- Represent data engineering in cross-functional architecture, planning and prioritisation discussions, translating technical trade-offs for senior technical and non-technical stakeholders.
- Embed data security and privacy by design, working with Information Security and the Data Protection team on how customer data is modelled, accessed and retained.
- Deep, hands-on expertise across core AWS data services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR (required).
- Advanced proficiency in Python and SQL, including PySpark for large-scale batch and streaming processing (required).
- Strong command of data modelling and data warehousing concepts (required).
- Production experience designing and operating streaming architectures with Apache Kafka or Amazon MSK (required).
- Hands-on experience with open table formats, such as Apache Iceberg or Delta Lake (required).
- Experience with pipeline orchestration and infrastructure-as-code using Terraform (required).
- Experience acting as technical lead on a significant data platform or migration programme, setting standards others follow (required).
- Track record of mentoring and developing engineers through review, pairing and design guidance (required).
- Ability to diagnose and resolve complex, ambiguous data issues, and to teach others to do the same (required).
- Ability to effectively communicate with both technical and non-technical stakeholders, including senior leadership (required).
- Experience with data security and privacy best practices, and how to build them into platform design (required).
- Experience with Agile development methodologies (required).
- Experience with Apache Flink for stateful stream processing (nice-to-have).
- Experience in migrating data from on-premises systems, such as SQL Server, to cloud environments (nice-to-have).
- Experience in a regulated industry, and awareness of how data platform design intersects with regulatory, compliance and responsible gambling obligations (nice-to-have).
- A competitive rate of pay and pension contribution (£85,000 - £105,000)
- Generous discretionary bonus schemes, incentives and competitions
- An annual leave entitlement that increases with length of service
- Access to an online GP 24/7, 365 days a year for you and your immediate family
- Employee wellbeing support through our Employee Assistance Programme
- Enhanced Maternity & Paternity Pay
- Long Service Recognition
- Access to a pay day savings scheme, financial coach and up to 40% of your earned wage ahead of payday, through Wagestream.
Betfred is a betting and gaming company that has operated for over 50 years, founded by Fred and Peter Done. It runs more than 1,300 betting shops and employs around 9,500 people, combining a large retail estate with online sports betting and gaming. In 2019 it entered the US sports betting market as Betfred Sportsbook, now operating in several states, and has established a presence in South Africa, where it holds a majority stake in the online betting company LottoStar. Headquartered in Warrington, England, Betfred emphasises customer service and responsible gambling across its operations.