Senior Big Data Engineer (Hadoop Admin)

Company Name: Pilot Flying J
Location: Knoxville, TN
Date Posted: 09th Jan, 2018


1.Research, develop, document and modify ETL processes and jobs according to data architecture and modeling requirements and processes set forth by Business Intelligence (BI) Data Architects 
2.Maintain accountability for completing key EDW/Data Lake project activities in a timely and efficient manner; communicate project status and project risks to project leader 
3.Troubleshoot issues including connection, failed jobs, application errors, server alerts, and space thresholds within predefined service level agreements (SLAs); assist less experienced ETL Developers in resolving more complex issues 
4.Proactively maintain and tune BDI to best practices in order to prevent issues 
5.Review and ensure appropriate documentation for all new development and modifications of the ETL processes and jobs 
6.Perform unit testing for solutions developed, and ensure integrity and security of institutional data 
7.Translate business requirements into technical specifications, including the creation of mapping documents to support the ETL development process, and the execution of ETL jobs 
8.Educate customers on BDI technologies 
9.Mentor junior team members 
10.Ensure all activities are in compliance with rules, regulations, policies, and procedures of the organization 
11.Complete other duties as assigned Compensation Based On Experience


Skills Required -

  • Strong T-SQL knowledge and skills required
  • Knowledge of IBM Puredata (Netezza) preferred
  • Knowledge of IBM Information Server (DataStage, QualityStage) preferred
  • Good understanding of current business intelligence and data warehousing tools, able to support the implementation of the tools
  • Familiarity with Hortonworks or similar Data Platform required
  • Experience with the following tools:
  • Spark / Spark-streaming
  • Python
  • Hive
  • Kafka
  • Sqoop
  • Flume
  • HDFS
  • Java
  • MapReduce
  • Understanding of Lambda Architecture principles
  • Experience with Redis preferred