Your mission, roles and requirements:

We are seeking a highly skilled Azure Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, with specific expertise in Azure-based services and Python. You will be responsible for designing, building, and maintaining scalable data pipelines and ensuring the efficient storage and retrieval of data in our Azure cloud environment.

Key Responsibilities:

• Design and implement scalable data pipelines using Azure Data Factory and PySpark.
• Develop and maintain data models in Azure SQL Database, Azure Synapse Analytics, and other Azure data services.
• Work closely with data scientists, analysts, and other stakeholders to understand data requirements and ensure proper data flow.
• Optimize and manage large datasets in Azure, ensuring high availability, performance, and security.
• Monitor and troubleshoot data pipelines, ensuring data integrity and timely availability.
• Continuously improve data processing efficiency by leveraging the latest tools and techniques.
• Document processes and data architectures, ensuring clarity and easy reference for the team.

Must-Have Skills:

• Proficiency in Python for data manipulation and pipeline development.
• Extensive experience with PySpark for large-scale data processing.
• Hands-on experience with Azure Data Factory (ADF) for building data workflows.
• Strong knowledge of Azure SQL Database and Azure Synapse Analytics for data storage and processing.
• Experience in optimizing and managing Azure Databricks environments.
• Familiarity with data modeling, ETL/ELT processes, and data warehousing concepts.
• Strong problem-solving skills and ability to work in a fast-paced environment.

Preferred Qualifications:

• Experience with other Azure services like Azure Blob Storage, Azure Cosmos DB, and Azure Data Lake.
• Knowledge of DevOps practices, including CI/CD pipelines, using Azure DevOps.
• Certification in Azure Data Engineering or related fields.
• Excellent communication skills and ability to collaborate with cross-functional teams.

Education and Experience:

Bachelor’s degree in Computer Science, Information Technology, or a related field.