Senior Developer, Big Data
Essential Functions:
• Build real time stream-processing systems, using solutions such as Storm and/or Spark-Streaming
• Design, build and support pipelines of data ingestion ,transformation, conversion and validation
• Design and support effective storage and retrieval using NoSQL databases, such as HBase, Cassandra, MongoDB
• Design and implement best practices for cloud based cluster deployments of Hadoop, Spark, and other BigData eco-system tools
• Develop architecture and design patterns to process and store high volume data sets
• Build continuous integration and test driven development environment
• Research and deploy new tools, frameworks and patterns to build a sustainable big data platform
• Regular, dependable attendance & punctuality
Education/Experience:
• Bachelor's Degree in Computer Science/Engineering required
• 7+ years of experience in large-scale software development
• 4+ years of experience with data analytics and high volume data processing
• Hands-on development experience working on large projects with Hadoop stack (HIVE, Pig, Hadoop streaming, Spark streaming) and MapReduce required
• Experience with messaging & Complex event processing systems such as Kafka, Storm and Spark
• Experience with Machine Learning algorithms a plus
• Statistical analysis with R, Python or similar a plus
• Strong object-oriented design experience, applied use of design patterns, and expert knowledge of the Java, Scala and Python technology platform
• Cluster upgrades and maintenance automation
Communication Skills:
• Strong communication and collaboration skills
Reasoning Ability:
• Strong analytical and problem solving skills, good understanding of different data structures, algorithms and their usage in solving business problems
• Proven ability to work independently; designing, developing and deploying solutions, and to deliver projects on time with minimal direction
Other Skills:
• Innovative and creative, able to champion new ideas and initiatives within an energized, collaborative and entrepreneurial environment
• Ability to visualize data using various web based technologies