Senior Big Data Engineer
- Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
- Implementing ETL processes
- Monitoring performance and advising any necessary infrastructure/process changes
5+ years hands-on programming experience in data engineering – they do not want a senior person.
Solid understanding of CS fundamentals
Proficiency programming in Java
Proficient in another modern programming language: Scala, Python, etc.
Proficient understanding of distributed computing principles
Ability to solve any ongoing issues with operating the cluster/instances
Proficiency with AWS ecosystem or Hadoop v2 ecosystem (Cloudera/MapR/Hortonworks), MapReduce, etc.
Experience with building stream-processing systems, using solutions such as Spark-Streaming, Storm or Kinesis
Experience with integration of data from multiple data sources
Degree in Computer Science, Computer Engineering or similar
Nice to Haves:
Experience with Spark
Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
Knowledge of various ETL techniques and frameworks, such as Flume
Experience with various messaging systems, such as Kafka or RabbitMQ
Experience with Big Data ML toolkits, such as SparkML, scikit-learn, H2O, etc.