Big Data Engineer

RESPONSIBILITIES:

• Engineer reliable data pipelines for sourcing, processing, transforming, enriching and storing data in different ways, using data platform infrastructure effectively
• Ingest and transform data sets from a variety of data sources
• Focus on ingesting, storing, processing, and analyzing large datasets
• Create scalable, high-performance web services for tracking data
• Using Java (Spring Boot framework) for development tasks

REQUIREMENTS:

Must requirements:

• Must have 4+ years of experience at a similar role
• Having hands on experience in Hadoop ecosystem (on-prem) including Spark, HDFS, MapReduce, YARN, …
• Good in programming language Python.
• Experience in monitoring large-scale data processing job (batch-processing, stream processing)
• Having a background of software developing on Java Spring Boot

Good to Have:

• Experience with Hadoop distributions such as Cloudera, HortonWorks, comparison and feasibility
• Experience with Data warehouse and Data management: Data Quality, Data integration
• Experience in ETL, SQL and NoSQL Database
• Experience with SRE, Patching & Automation: Kubernetes or Docker & Containerization
• Experience working with Big Data in a Cloud environment
• Experience in Data API
• Good to have Architecture knowledge or experience