Hadoop Big Data Engineer
Company Name:
Meridian Health Plan
Location:
MI-Detroit
Date Posted:
06th Sep, 2016
Description:
- Install and configure Hadoop components and related utilities
- Develop processes for source data ingestion, transformation, and database loading
- Develop processes for data quality monitoring
- Develop processes to support “Data as a Service (DaaS)”
- File system management, monitoring, support and maintenance for HDFS, KUDU, and HBASE
- Create scalable and high-performance web services for data tracking
- Assisting with data lake folder design
- Recommending and establishing security policies and procedures for the Hadoop environment
- Assisting in the development and implementation of various strategic initiatives.
- Contribute to development of Architecture Policies, Standards and Governance for the Hadoop and Big Data environment
- Research and development with promising leading edge big data technologies
- Participating in data architecture design and review processes, including planning and monitoring efforts, reviewing deliverables, and communicating to management
- Responding to change and engaging in multiple projects simultaneously
- Works with minimal guidance; seeks guidance on only the most complex tasks
Qualification:
- Bachelor’s degree in related field is required
- 7+ years of experience as an IT professional
- 2+ years working with Hadoop (Cloudera or Horton Works or MAPR)
- 3+ years working with data design or database development
- Experience with reporting tools such as Tableau, Qlikview, Datameer, etc would be a plus
- Prior experience in a complex, highly integrated services environment
- Working knowledge of Red Hat LINUX
- Good aptitude in multi-threading and concurrency concepts
- Understanding of and experience developing in Hadoop
- Working knowledge of Kafka, Flume, Hive, Spark, Impala, Sqoop Oozie, Hbase, Zookeeper, HUE
- Expert level SQL knowledge and experience with a relational database
- Working knowledge of Pig Latin, HiveQL, Python or Java
- Substantial understanding of reporting and analytics tools
- Experience working with data lakes
- Pre and post installation of Hadoop software and good understanding of Hadoop ecosystems dependencies
- Implementing data ingress and egress-Facilitating generic input/output, moving bulk data into and out of Hadoop
- Expertise in setting up, configuration and management of data security
- On-going support for various Hadoop environments - DEMO, TEST, UAT, and PROD;
- Familiarity with Cloudera, Kafka, Spark, data lakes, and Flume is desired