Data Engineer

A Data Engineer is a developer tasked with implementing new or updated functionalities on the client's data platform and constructing data pipelines. The Data Engineer works closely with other data engineers, a functional analyst, a solution architect, a technical architect, and a QA engineer, forming an agile team. They leverage their technical expertise in tools such as Confluent Data Platform, Python, Airflow, NIFI, and more to develop functionalities and data pipelines.

On a typical day, a Data Engineer spends most of their time programming and delivering functional code. They also collaborate with team members, validate solutions, review peer code, address operational issues, write documentation, and evaluate and estimate user stories.

Technical Skills and Knowledge

Requirements
- Proficiency in Python
- Familiarity with NIFI (or other ELT tools)
- Experience with Airflow (or similar orchestration tools)
- Knowledge of an API builder (e.g., FastAPI, Flask)

Nice to Have
- Experience with Spark and PySpark
- Familiarity with Cloudera Data Platform (on-premise installation)
- Knowledge of Spark Streaming
- Experience with Kafka and Kafka streams
- Understanding of JDK languages (Java, Scala, Groovy)
- Knowledge of JavaScript (Angular, D3)
- Experience with event processing and JSON
- Familiarity with Apache AVRO & Parquet
- Knowledge of Apache Iceberg
- Understanding of Apache Impala
- Knowledge of Vault Speed
- Familiarity with SingleStore
- (Conceptual) understanding of data warehousing
- (Conceptual) knowledge of data vault
- (Conceptual) understanding of lakehouse architecture

Communication and Practical Skills
- Fluent in Dutch (working language and documentation are in Dutch)
- Ability to tailor communication style and tone to the audience
- Capability to review and incorporate guidelines and best practices into own work
- Independence in taking on and developing a ticket into working code
- Proven experience in writing data pipelines and/or building functionalities for a data platform
- Sincerity and courage necessary for an open feedback culture

Agile and Scrum
- Agile mindset with an understanding of agile principles
- Familiarity with scrum methodology and experience in its application