-
SalaryJob Description
Lead Data engineer helps to architect, design, implement, and optimise our data ingestion, transformation pipelines and processes. He or she develops data models, processing pipelines, and back-end services supporting the data science teams, automates processes, builds integrations, and builds analytics.
Desired skills:
· A university degree from a reputed University with a record of academic excellence in Mathematics, Computer Science, Engineering, Physics, or similar.
· 8+ years of relevant experience in Data Engineering, warehousing, ETL, automation, big data and cloud technologies, etc.
· Ability to write clean, scalable, maintainable code in Python. Has a good understanding of software engineering concepts and patterns. Proficiency in other languages like Scala, Java, C#, C++ are an advantage.
· Proven record of building and maintaining data pipelines deployed in at least one of the big 3 cloud ML stacks (AWS, Azure, GCP).
· Hands-on experience with
o Open-source ETL, and data pipeline and orchestration of such tools as Apache Airflow and Nifi. o Large Scale/ Big Data technology, such as Hadoop, Spark, Hive, Impala, PrestoDb, Kafka.
o Fast and distributed data processing in Ray, Dask, DuckDb, Polars, cuDF
o Building performant code in C, C++ and interfaces with python in PyO3.rs, Cython
o Workflow orchestration tools like Apache Airflow. o Containerisation with Docker, deployment on Kubernetes.
o NoSql and graph databases.
o Unix server administration and shell scripting.
· Experience in building scalable data pipelines for highly unstructured data.
· Experience in building DWH and data lakes architectures.
· Experience in working in cross-functional team with software engineers, data scientists and machine learning engineers.
· Experience in working with or leading an off-shore team.
· Proven record of building data science environments deploying ML solutions in at least one of the big 3 cloud ML stacks (Azure/Aws/GCP) and on Kubernetes clusters.
· Excellent written and verbal command of English.
· Strong problem-solving, analytical, and quantitative skills.
· A professional attitude and service orientation with the ability to work with our international teams.
Check Your Resume for Match
Upload your resume and our tool will compare it to the requirements for this job like recruiters do.
Check for Match
It has come to our attention that clients and candidates are being contacted by individuals fraudulently posing as Antal representatives. If you receive a suspicious message (by email or WhatsApp), please do not click on any links or attachments. We never ask for credit card or bank details to purchase materials, and we do not charge fees to jobseekers.