Job Description

Roles & Responsibilities

As part of strengthening our Data teams, we are looking for profiles capable of designing, industrializing and optimizing data platforms (batch & real-time) within distributed environments based on Cloudera.

Your responsibilities:

• Development & Industrialization
  • Develop massive data processing pipelines in PySpark (Batch and Real-Time / Streaming modes).
  • Set up real-time streams via Kafka (topics, partitions, schemes, offsets) and ensure ingestion with NiFi.
  • Model and optimize NoSQL schemas, especially on Cassandra (tables, keys, clustering, replication) and Hive.
  • Integrate and transform data from multiple sources (APIs, databases, streams, files).
• Quality, Performance & Reliability
  • Deploy Data Quality mechanisms (controls, monitoring, alerting).
  • Optimize Spark processing (partitioning, tuning, data formats) specifically for distributed architectures.
  • To ensure the supervision and resolution of incidents in a production environment.
• CI/CD & Governance
  • Industrialize developments via CI/CD chains (automated testing, deployments).
  • Documenting flows, models and best practices.
  • Collaborating occasionally with third-party teams on Dataviz topics and collaborating with Data Scientists (particularly on customer segmentation algorithms).
  • Contribute to data governance (catalogue, traceability, security).

Desired Candidate Profile

• Experience: ~4 years of experience in distributed environments and Big Data architectures.
• Architecture: Strong interest and significant experience in On-Premise infrastructures.
• Spark / PySpark: Essential mastery in Batch and Streaming processing.
• NoSQL: Confirmed experience on Cassandra and/or Hive
• Streaming & Ingestion: Very good mastery of Kafka and NiFi
• Nice-to-have / Bonus: Good understanding of the Hadoop/HDFS ecosystem. Cloudera proficiency is a plus but not required. Knowledge of Data Science (clustering/customer segmentation models) and Data Visualization tools.
• Tools & DevOps: Skills in Git, CI/CD (GitLab CI) and an orchestration tool (Airflow, Luigi, Prefect).