Are you passionate in data? Do you like working in a challenging environment where massive volume of data is ingested and processed every day? Are you a continuous learner who wants to learn new tools and technologies in evolving big data and data science technology? Do you have passion in detecting data anomalies in large datasets? Do you expect the best from yourself and those around you?

We are hiring for a Data Engineer for an initiative within the Data Analytics Transformation Team. You will work on creating more scalable pattern fits for teams, building ad hoc pipelines, building repeatable infrastructures.

Job Responsibilities

• Understand complex business requirements
• Design and develop ETL pipeline for collecting, validating and transforming data according to the specification
• Develop automated unit tests, functional tests and performance tests.
• Maintain optimal data pipeline architecture
• Design ETL jobs for optimal execution in AWS cloud environment
• Reduce processing time and cost of ETL workloads
• Lead peer reviews and design/code review meetings
• Provide support for production support operations team
• Implement data quality checks.
• Identify areas where machine learning can be used to identify data anomalies

• 7+ years of experience in programming language Python
• 7+ years of experience in ETL projects
• 7+ years of experience in data engineering projects
• 3+ years of experience with API development (REST API's)

• Strong experience in Python
• Strong experience in big data technologies like AWS EMR, AWS EKS, Apache Spark
• Strong experience with server less technologies like AWS Dynamo DB, AWS Lambda
• Big data tools such as Presto, Spark, etc.
• Must be able to write complex SQL queries
• Experience in performance tuning and optimization
• Experience with CI/CD pipelines
• Agile