We are looking for an end-to-end data engineer with experience in the development of data pipeline platforms and in the modelling and querying of data for Business Intelligence purposes. The technical architecture comprises a Teradata and Hadoop platform utilizing Python/Linux batch and also Kafka data ingestion mechanisms. This is a Dev/Ops role where you would be responsible for supporting existing production data pipelines and adhoc BI enhancement requests, while expanding our new Hadoop based data platform.
■Develop, enhance and maintain data pipeline applications and data models
■Trouble-shoot the causes of adhoc daily production failures and provide effective and documented solutions.
■Continuous improvement initiatives in data ingestion performance, ingestion models, data integrity and data availability.
Work with the business in analyzing and documenting new functionality requests and managing the implementation of those within an Agile ownership model.
■B.S. in Computer Science or in related fields.
■More than 3 years’ experience with BI data driven development.
■Expert SQL capability in querying Big Data/ large data sets (Teradata, Hadoop, etc.) to extract BI- insights.
■Programming languages such as Python/Scala/PLSQL/Java.
■Development and operation of data pipeline leveraging big data technologies such as Spark, Map Reduce, Hive, Kafka, Sqoop, NoSQL Databases as well as traditional DB and file based data integration solutions.
■Database development (eg TeraData, Oracle, MySQL, SQLServer, DB2..)
■Shell-scripting languages such as Bash.
■Distributed version control system such as Git.
■Initiative and the ability to work independently and in a team. We are an Agile environment.
■Experience in Teradata and Informatica
■Application development using workflow engines such as Airflow, Oozie, Rundeck
■BI Modelling of data marts using ER hybrid, Kimball, Data Vault methodologies
■Experience in AtScale and Presto
■Operational experience in developing and supporting high availability applications / systems.
■Capability to self-manage and also manage small projects.