Question
Explain the process of spilling in MapReduce?
Answers
The output of a map task is written into a circular memory buffer (RAM). The default size of buffer is set to 100 MB which can be tuned by using mapreduce.task.io.sort.mb property. Now, spilling is a process of copying the data from memory buffer to disc when the content of the buffer reaches a certain threshold size. By default, a background thread starts spilling the contents from memory to disc after 80% of the buffer size is filled. Therefore, for a 100 MB size buffer the spilling will start after the content of the buffer reach a size of 80 MB.

Note: One can change this spilling threshold using mapreduce.map.sort.spill.percent which is set to 0.8 or 80% by default.   Your Comment




  • Contributors: *,
More Software Questions..
What is the inputsplit in map reduce software?
What is software configuration management?
What Is Java Api For Xml-based Rpc (jax-rpc)?
How can you implement fine-grained auditing?
What is IBM’s simple explanation for Big Data’s four critical features?
What is static synchronized method in JDBC API? Give an example?
What does the NULLIF function do?
What happens if a start method is not invoked and the run method is directly invoked?
Should we override finalize method
what is the difference between mysql_fetch_array and mysql_fetch_object?
How will XML affect my document links?
Why to use Style Sheets?
What are Filters in MVC?
Can you explain Application layer in OSI model?
How to define new testplan attributes?
What are the minimum system requirements to run Photoshop? Is it possible to run Photoshop over linux?
Which oracle package is used to manage the oracle lock management services?
What is Latch Up? Explain Latch Up with cross section of a CMOS Inverter. How do you avoid Latch Up?
What is marker interface?
What types of partitioning are there for BW?


Search
Can you Answer!!
  • Q What is the usage of Django-admin.py and manage.py?
  • Q Which is the softest?
  • Q Do i really need to learn javascript?
  • Q Briefly explain the steps for creating a Power Point presentation
  • Q   What do you mean by term “Annuity”?
  • Q What are the destination target options?
  • Q What is the effect of addition of zinc in copper? what is the use of 70/30 brass ?
  • Q Advantages of migrating to vb.net?
  • Q Explain cross-validation.?
  • Q Which of the following is the MOST effective technique for providing security during data transmission? A. Communication log B. Systems software log C. Encryption D. Standard protocol
  • Q Define ugs class in wimax qos