Job Description

Roles & Responsibilities

Job Description

As a Data Engineer, you will join our data team to participate in strategic data analysis projects and the implementation of large-scale data processing solutions. You will be responsible for the design, optimization, and development of data pipelines on cloud platforms, with a focus on the use of PySpark and Azure Data Factory .

Your main responsibilities will be:

• Design, develop and maintain large-scale data pipelines using PySpark .
• Set up data flows with Azure Data Factory , integrating various data sources and destinations (SQL, Data Lakes, APIs, etc.).
• Ensure the integration of data from various systems in a cloud environment (Azure).
• Optimize the performance of data processing (SQL, PySpark, etc.).
• Collaborating with business teams, data analysts and data scientists to understand needs and propose effective and scalable solutions.
• Participate in defining the data architecture and governance strategy.
• Set up and monitor data processing jobs (error handling, performance management).
• Document the solutions implemented and follow best development practices.
• Participate in the continuous improvement of existing processes and tools.

Desired Candidate Profile

Qualifications

We are looking for a candidate with strong technical expertise in large-scale data management and processing, with the following skills:

Comp tences techniques :

• Mastering PySpark for processing massive data in a distributed environment.
• Significant experience in setting up and managing Azure Data Factory for creating and orchestrating data pipelines.
• Solid experience in relational and non-relational databases (SQL, NoSQL).
• Knowledge of data management tools in the Azure ecosystem (Azure Data Lake, Azure SQL Database, Azure Blob Storage, etc.).
• Practical knowledge of Azure cloud services and their use in data processing solutions (Azure Databricks, Azure Synapse Analytics, etc.).
• Experience with version control tools (Git, Azure DevOps, etc.) and good software development practices.
• Knowledge of Big Data concepts and performance optimization in distributed environments.

Personal skills:

• Autonomy, rigor and attention to detail.
• Excellent ability to solve complex technical problems.
• Good communication skills and ability to work in a team.
• Analytical mindset and ability to understand business needs in order to propose appropriate solutions.

Formation et exp rience :

• Master's degree (Bac +5) in computer science, data science, or an equivalent field.
• Minimum of 3 years of experience in a similar role, with proven experience in development on PySpark and Azure Data Factory .