Senior Data Engineer
As a Senior Data Engineer, you will be a key driver in our transition from an on-premises data infrastructure to a modern, scalable cloud lake house architecture. Your expertise in SQL querying, query optimization, stored procedures, data modelling, and streaming will support current operations while enabling the transformation to a cloud-first strategy. You will play a critical role in designing and implementing a future-ready data platform, ensuring seamless integration of real-time and batch processing with the latest technologies. Additionally, you will mentor and train junior team members, fostering their development and ensuring the team’s success in this transformative journey.
- Designing, developing, and optimizing robust data pipelines for on-premises and hybrid cloud environments.
- Leading efforts to migrate data and systems from on-premises databases to a cloud lake house architecture.
- Developing scalable data models for cloud-based storage and processing.
- Implementing and managing streaming data solutions using platforms like Kafka or RabbitMQ.
- Collaborating with cross-functional teams to integrate data from multiple sources into a unified data platform.
- Optimizing query performance and database structures to meet evolving business needs.
- Building and maintaining robust ETL/ELT processes to support cloud-native architectures.
- Driving the adoption of modern data lake house technologies such as Delta Lake, Apache Iceberg, or Apache Hudi.
- Mentoring and training junior team members to build their expertise in cloud data engineering.
- Staying updated on emerging cloud and lake house technologies to guide the data architecture evolution.
- Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field (Master’s preferred). (required)
- 5+ years of experience in data engineering or related roles. (required)
- Relevant certifications (e.g., AWS Certified Data Analytics Specialty, Microsoft Certified: Azure Data Engineer, or Databricks Certified Data Engineer). (preferred)
- Strong expertise in SQL, including querying, optimization, and writing stored procedures. (required)
- Proven experience in cloud-based data platforms such as AWS Redshift, Azure Synapse, or Google BigQuery. (required)
- Knowledge of cloud lake house technologies (e.g., Delta Lake, Apache Iceberg, or Apache Hudi). (required)
- Hands-on experience with streaming platforms such as Kafka or RabbitMQ. (required)
- Expertise in data modelling for both on-premises and cloud-based systems. (required)
- Proficiency in Python, Java, or Scala for data engineering tasks. (required)
- Experience with CI/CD practices for data pipelines in a cloud environment. (required)
- Familiarity with data governance and security best practices in cloud environments. (required)
- Strong leadership and mentoring skills, with experience guiding junior team members. (required)
- Experience with big data technologies such as Hadoop, Spark, or Flink, especially in a cloud context. (nice-to-have)
- Knowledge of containerization tools like Docker and orchestration platforms such as Kubernetes. (nice-to-have)
- Expertise in monitoring and observability tools for cloud data pipelines. (nice-to-have)
- Certification in cloud platforms or lake house technologies (e.g., AWS Data Analytics Specialty, Databricks Certified Data Engineer). (nice-to-have)
- Background in implementing real-time analytics and event-driven architectures in the cloud. (nice-to-have)
- Understanding of cost optimization strategies for cloud-based data infrastructure. (nice-to-have)
- Experience in managing hybrid environments during migration from on-prem to cloud. (nice-to-have)
- Hands-on experience with serverless computing in cloud such as Lambda Functions. (nice-to-have)
- Hands-on experience with cloud analytics platforms such as Big Query or Databricks. (nice-to-have)
- Proficiency in C# for general engineering tasks. (nice-to-have)
- Hashicorp Terraform for IaC workloads. (nice-to-have)
- Automation using shell scripts (Powershell, Bash etc). (nice-to-have)
- Comprehensive learning and development programmes.
- Innovative Performance Tool for regular, constructive feedback.
- Employee Assistance programme for you and your family.
- Free Daily Meal.
- Free Massages On-site.
- Free On-Site Gym.
- Group Life Cover.
- Funeral Fund Benefit.
- Financial Services Assistance.
- Employee Assistance Programme.
- Curro School Fees Benefit.
- Income Continuation Benefit.
- Leadership Training.
- Referral Bonus.
- Medical Aid Subsidy.
- Free Sleep Coaching.
- On-site Barista.
- Retirement Annuity Subsidy.
- Team socials.
DigiOutsource is a global iGaming technology and services company and part of Super Group (SGHC), the holding company behind the Betway and Spin brands. Headquartered in Cape Town, South Africa, with offices in London and Portugal, it provides software development, digital marketing, business intelligence, design and communications services that power leading online gaming brands. The company employs around 1,200 people across multiple continents. DigiOutsource was formed from the merger of the ForwardSlash and SpeakUp Communications businesses and operates as a key technology hub within Super Group.