Cluster Hadoop Setting and stress test



0

Hi Michael,

I have some questions which need your expertise/exp help.

(1) Regarding to the cluster setting, there are 6 core file - core-site.xml, hdfs.xml, slave,masters,etcs.

Do you have any use case/ scenarios from begining to advance (real-life) level and actual setting to those files ?

(2) Have you exp in doing stress test on the hadoop system,  can you share the Infrastructure/ system conf and the statistic result ? eg. No of files/ data size per mil-sec the Name/Data node handling, No of DataNode, any hadoop KPI measurement, fail reason ?

Thanks

Regards

 


2 Answer(s)


0

Hi Kenny,

It would be very hard for me to specifically answer your question as my usual job role is a big data developer and not administration.

1) The core configuration files all have defaults and changing the values of the file is not baseed on some science but on per-cluster base. Remember that they are other factors that affect the performance of a cluster that you have listed including node resources, network configuration and the likes. In a real world instance, there resource vary from cluster to cluster. This also should affect the config settings values.

2.I don't have any experience in stress testing as it is not my role in any team so far and even in the future. I am ultimately a developer.

 

Regards


0

sorry about that , Michael.

I will go through the this  session material and will have some questions which need your help.

 

Thanks

 

Regards

 

Kenny

 

Your Answer

Click on this code-snippet-icon icon to add code snippet.

Upload Files (Maximum image file size - 1.5 MB, other file size - 10 MB, total size - not more than 50 MB)

Email
Password