how to parse logs files in hadoop


2 Answer(s)


I came to know open source software like Elastic Search and Kibana, https://www.elastic.co/products/kibana I have not used it that much, it seems you can configure the server names and log fie location, and may be some custom filters. You can get all that analytics out of the log files, I think you need LogStach, you can explore it all
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04

Hi Vadivel, Thanks for the info.I have used ELK for certain things like sending the data in json format and plotting it on kibana. It can do chart plotting and aggregation like min, max, average. As per my knowledge it needs data needs to be structured before Kibana can make use of it. In my case I need to find error in completely unstructured data and raise tickets based on that. Even if unstructured data can be shown on kibana, someone has to query for the errors. I need to read more if there is any automated reporting based on certain filters on unstructured data otherwise it can't solve my problem.

Thanks again.

I am still trying to find out what are the use cases of ELK, Solr and Splunk vs the hadoop ecosystem we have been learning.

-Sushil