Job Description

Roles & Responsibilities

Your main tasks will be:

• Design, develop and maintain large-scale data pipelines using PySpark.
• Set up data flows with Azure Data Factory, integrating various data sources and destinations (SQL, Data Lakes, APIs, etc.).
• Ensure the integration of data from various systems in a cloud environment (Azure).
• Optimize the performance of data processing (SQL, PySpark, etc.).
• Collaborating with business teams, data analysts and data scientists to understand needs and propose effective and scalable solutions.
• Participate in defining the data architecture and governance strategy.
• Set up and monitor data processing jobs (error handling, performance management).
• Document the solutions implemented and follow best development practices.
• Participate in the continuous improvement of existing processes and tools.

Desired Candidate Profile

Formation et exp rience :

• Master's degree (Bac +5) in computer science, data science, or an equivalent field.
• Minimum of 3 years of experience in a similar role, with proven experience in development on PySpark and Azure Data Factory.

strong>Comp tences techniques :/strong>

• Mastering PySpark for processing massive data in a distributed environment.
• Significant experience in setting up and managing Azure Data Factory for creating and orchestrating data pipelines.
• Solid experience in relational and non-relational databases (SQL, NoSQL).
• Knowledge of data management tools in the Azure ecosystem (Azure Data Lake, Azure SQL Database, Azure Blob Storage, etc.).
• Practical knowledge of Azure cloud services and their use in data processing solutions (Azure Databricks, Azure Synapse Analytics, etc.).
• Experience with version control tools (Git, Azure DevOps, etc.) and good software development practices.
• Knowledge of the concepts of Big Data and performance optimization in distributed environments.

Personal skills:

• Autonomy, rigor and attention to detail.
• Excellent ability to solve complex technical problems.
• Good communication skills and ability to work in a team.
• Analytical mindset and ability to understand business needs in order to propose appropriate solutions.