Apache pig local mode



0

Hi Team. I have a question regarding pig local mode (Pig -x local) if my hadoop deamon is not running and we run pig in local mode its working fine and simlilar command which runs in map reduce mode work as well which suggest single JVM concept for pig local, my question is when we exec pif query and after parsing->Optimizer->Compiler it creates series of map reduce job which then takes help of MR to run on hDFS in local we don't have hadoop running and no MR, how it workes in thid case


2 Answer(s)


0

Hi Nitesh,

Pig local mode: All scripts are run on a single machine without requiring Hadoop MapReduce and HDFS. This can be useful for developing and testing Pig logic. If you’re using a small set of data to developer or test your code, then local mode could be faster than going through the MapReduce infrastructure.

Local mode doesn’t require Hadoop. When you run in Local mode, the Pig program runs in the context of a local Java Virtual Machine, and data access is via the local file system of a single machine. Local mode is actually a local simulation of MapReduce in Hadoop’s LocalJobRunner class.

MapReduce mode (also known as Hadoop mode): Pig is executed on the Hadoop cluster. In this case, the Pig Script gets converted into a series of MapReduce jobs that are then run on the Hadoop cluster. LOcal and Distributed mode of pig

If you have a terabyte of data that you want to perform operations on and you want to interactively develop a program, you may soon find things slowing down considerably, and you may start growing your storage. Local mode allows you to work with a subset of your data in a more interactive manner so that you can figure out the logic (and work out the bugs) of your Pig program.

After you have things set up as you want them and your operations are running smoothly, you can then run the script against the full data set using MapReduce mode.

Hope this answers your query.


0

Thanks Abhijit