By | 09/22/2013



MapReduce is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster.[1]

A MapReduce program comprises a Map() procedure that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a Reduce() procedure that performs a summary operation (such as counting the number of students in each queue, yielding name frequencies). The “MapReduce System” (also called “infrastructure”, “framework”) orchestrates by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, providing for redundancy and fault tolerance, and overall management of the whole process.

The model is inspired by the map and reduce functions commonly used in functional programming,[2] although their purpose in the MapReduce framework is not the same as their original forms.[3] Furthermore, the key contribution of the MapReduce framework are not the actual map and reduce functions, but the scalability and fault-tolerance achieved for a variety of applications by optimizing the execution engine once.

MapReduce libraries have been written in many programming languages, with different levels of optimization. A popular open-source implementation is Apache Hadoop. The name MapReduce originally referred to the proprietary Google technology and has since been genericized.


‘MapReduce’ is a framework for processing parallelizable problems across huge datasets using a large number of computers (nodes), collectively referred to as a cluster (if all nodes are on the same local network and use similar hardware) or a grid (if the nodes are shared across geographically and administratively distributed systems, and use more heterogenous hardware). Computational processing can occur on data stored either in a filesystem (unstructured) or in a database (structured). MapReduce can take advantage of locality of data, processing data on or near the storage assets to decrease transmission of data.

“Map” step: The master node takes the input, divides it into smaller sub-problems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node.

“Reduce” step: The master node then collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve.

MapReduce allows for distributed processing of the map and reduction operations. Provided each mapping operation is independent of the others, all maps can be performed in parallel – though in practice it is limited by the number of independent data sources and/or the number of CPUs near each source. Similarly, a set of ‘reducers’ can perform the reduction phase – provided all outputs of the map operation that share the same key are presented to the same reducer at the same time, or if the reduction function is associative. While this process can often appear inefficient compared to algorithms that are more sequential, MapReduce can be applied to significantly larger datasets than “commodity” servers can handle – a large server farm can use MapReduce to sort a petabyte of data in only a few hours.[citation needed] The parallelism also offers some possibility of recovering from partial failure of servers or storage during the operation: if one mapper or reducer fails, the work can be rescheduled – assuming the input data is still available.

Another way to look at MapReduce is as a 5-step parallel and distributed computation:

  1. Prepare the Map() input – the “MapReduce system” designates Map processors, assigns the K1 input key value each processor would work on, and provides that processor with all the input data associated with that key value.
  2. Run the user-provided Map() code – Map() is run exactly once for each K1 key value, generating output organized by key values K2.
  3. “Shuffle” the Map output to the Reduce processors – the MapReduce system designates Reduce processors, assigns the K2 key value each processor would work on, and provides that processor with all the Map-generated data associated with that key value.
  4. Run the user-provided Reduce() code – Reduce() is run exactly once for each K2 key value produced by the Map step.
  5. Produce the final output – the MapReduce system collects all the Reduce output, and sorts it by K2 to produce the final outcome.

Logically these 5 steps can be thought of as running in sequence – each step starts only after the previous step is completed – though in practice, of course, they can be intertwined, as long as the final result is not affected.

In many situations the input data might already be distributed (“sharded”) among many different servers, in which case step 1 could sometimes be greatly simplified by assigning Map servers that would process the locally present input data. Similarly, step 3 could sometimes be sped up by assigning Reduce processors that are as much as possible local to the Map-generated data they need to process.

Logical view

The Map and Reduce functions of MapReduce are both defined with respect to data structured in (key, value) pairs. Map takes one pair of data with a type in one data domain, and returns a list of pairs in a different domain:


The Map function is applied in parallel to every pair in the input dataset. This produces a list of pairs for each call. After that, the MapReduce framework collects all pairs with the same key from all lists and groups them together, creating one group for each key.

The Reduce function is then applied in parallel to each group, which in turn produces a collection of values in the same domain:

Reduce(k2, list (v2))list(v3)

Each Reduce call typically produces either one value v3 or an empty return, though one call is allowed to return more than one value. The returns of all calls are collected as the desired result list.

Thus the MapReduce framework transforms a list of (key, value) pairs into a list of values. This behavior is different from the typical functional programming map and reduce combination, which accepts a list of arbitrary values and returns one single value that combines all the values returned by map.

It is necessary but not sufficient to have implementations of the map and reduce abstractions in order to implement MapReduce. Distributed implementations of MapReduce require a means of connecting the processes performing the Map and Reduce phases. This may be a distributed file system. Other options are possible, such as direct streaming from mappers to reducers, or for the mapping processors to serve up their results to reducers that query them.


The prototypical MapReduce example counts the appearance of each word in a set of documents:[4]

function map(String name, String document):
  // name: document name
  // document: document contents
  for each word w in document:
    emit (w, 1)

function reduce(String word, Iterator partialCounts):
  // word: a word
  // partialCounts: a list of aggregated partial counts
  sum = 0
  for each pc in partialCounts:
    sum += ParseInt(pc)
  emit (word, sum)

Here, each document is split into words, and each word is counted by the map function, using the word as the result key. The framework puts together all the pairs with the same key and feeds them to the same call to reduce, thus this function just needs to sum all of its input values to find the total appearances of that word.

As another example, imagine that for a database of 1.1 billion people, one would like to compute the average number of social contacts a person has according to age. In SQL such a query could be expressed as:

SELECT age AS Y, AVG(contacts) AS A
FROM social.person GROUP BY age ORDER BY age

Using MapReduce, the K1 key values could be the integers 1 through 1,100, each representing a batch of 1 million records, the K2 key value could be a person’s age in years, and this computation could be achieved using the following functions:

function Map is
    input: integer K1 between 1 and 1100, representing a batch of 1 million social.person records
    for each social.person record in the K1 batch do
        let Y be the person's age
        let N be the number of contacts the person has
        produce one output record <Y,N>
end function

function Reduce is
    input: age (in years) Y
    for each input record <Y,N> do
        Accumulate in S the sum of N
        Accumulate in C the count of records so far
    let A be S/C
    produce one output record <Y,A>
end function

The MapReduce System would line up the 1,100 Map processors, and would provide each with its corresponding 1 million input records. The Map step would produce 1.1 billion <Y,N> records, with Y values ranging between, say, 8 and 103. The MapReduce System would then line up the 96 Reduce processors by performing shuffling operation of the key/value pairs due to the fact that we need average per age, and provide each with its millions of corresponding input records. The Reduce step would result in the much reduced set of only 96 output records <Y,A>, which would be put in the final result file, sorted by Y.


The frozen part of the MapReduce framework is a large distributed sort. The hot spots, which the application defines, are:

  • an input reader
  • a Map function
  • a partition function
  • a compare function
  • a Reduce function
  • an output writer

Input reader

The input reader divides the input into appropriate size ‘splits’ (in practice typically 16 MB to 128 MB) and the framework assigns one split to each Map function. The input reader reads data from stable storage (typically a distributed file system) and generates key/value pairs.

A common example will read a directory full of text files and return each line as a record.

Map function

The Map function takes a series of key/value pairs, processes each, and generates zero or more output key/value pairs. The input and output types of the map can be (and often are) different from each other.

If the application is doing a word count, the map function would break the line into words and output a key/value pair for each word. Each output pair would contain the word as the key and the number of instances of that word in the line as the value.

Partition function

Each Map function output is allocated to a particular reducer by the application’s partition function for sharding purposes. The partition function is given the key and the number of reducers and returns the index of the desired reduce.

A typical default is to hash the key and use the hash value modulo the number of reducers. It is important to pick a partition function that gives an approximately uniform distribution of data per shard for load-balancing purposes, otherwise the MapReduce operation can be held up waiting for slow reducers (reducers assigned more than their share of data) to finish.

Between the map and reduce stages, the data is shuffled (parallel-sorted / exchanged between nodes) in order to move the data from the map node that produced it to the shard in which it will be reduced. The shuffle can sometimes take longer than the computation time depending on network bandwidth, CPU speeds, data produced and time taken by map and reduce computations.

Comparison function

The input for each Reduce is pulled from the machine where the Map ran and sorted using the application’s comparison function.

Reduce function

The framework calls the application’s Reduce function once for each unique key in the sorted order. The Reduce can iterate through the values that are associated with that key and produce zero or more outputs.

In the word count example, the Reduce function takes the input values, sums them and generates a single output of the word and the final sum.

Output writer

The Output Writer writes the output of the Reduce to the stable storage, usually a distributed file system.

Distribution and reliability

MapReduce achieves reliability by parceling out a number of operations on the set of data to each node in the network. Each node is expected to report back periodically with completed work and status updates. If a node falls silent for longer than that interval, the master node (similar to the master server in the Google File System) records the node as dead and sends out the node’s assigned work to other nodes. Individual operations use atomic operations for naming file outputs as a check to ensure that there are not parallel conflicting threads running. When files are renamed, it is possible to also copy them to another name in addition to the name of the task (allowing for side-effects).

The reduce operations operate much the same way. Because of their inferior properties with regard to parallel operations, the master node attempts to schedule reduce operations on the same node, or in the same rack as the node holding the data being operated on. This property is desirable as it conserves bandwidth across the backbone network of the datacenter.

Implementations are not necessarily highly reliable. For example, in older versions of Hadoop the NameNode was a single point of failure for the distributed filesystem. Later versions of Hadoop have high availability with an active/passive failover for the “NameNode.”


MapReduce is useful in a wide range of applications, including distributed pattern-based searching, distributed sorting, web link-graph reversal, term-vector per host, web access log stats, inverted index construction, document clustering, machine learning,[5] and statistical machine translation. Moreover, the MapReduce model has been adapted to several computing environments like multi-core and many-core systems,[6][7][8] desktop grids,[9] volunteer computing environments,[10] dynamic cloud environments,[11] and mobile environments.[12]

At Google, MapReduce was used to completely regenerate Google’s index of the World Wide Web. It replaced the old ad hoc programs that updated the index and ran the various analyses.[13]

MapReduce’s stable inputs and outputs are usually stored in a distributed file system. The transient data is usually stored on local disk and fetched remotely by the reducers.


Lack of novelty

David DeWitt and Michael Stonebraker, computer scientists specializing in parallel databases and shared-nothing architectures, have been critical of the breadth of problems that MapReduce can be used for.[14] They called its interface too low-level and questioned whether it really represents the paradigm shift its proponents have claimed it is.[15] They challenged the MapReduce proponents’ claims of novelty, citing Teradata as an example of prior art that has existed for over two decades. They also compared MapReduce programmers to Codasyl programmers, noting both are “writing in a low-level language performing low-level record manipulation.”[15] MapReduce’s use of input files and lack of schema support prevents the performance improvements enabled by common database system features such as B-trees and hash partitioning, though projects such as Pig (or PigLatin), Sawzall, Apache Hive,[16] YSmart,[17] HBase[18] and BigTable[18][19] are addressing some of these problems.

Greg Jorgensen wrote an article rejecting these views.[20] Jorgensen asserts that DeWitt and Stonebraker’s entire analysis is groundless as MapReduce was never designed nor intended to be used as a database.

DeWitt and Stonebraker have subsequently published a detailed benchmark study in 2009 comparing performance of Hadoop’s MapReduce and RDBMS approaches on several specific problems.[21] They concluded that relational databases offer real advantages for many kinds of data use, especially on complex processing or where the data is used across an enterprise, but that MapReduce may be easier for users to adopt for simple or one-time processing tasks.

Google has been granted a patent on MapReduce.[22] However, there have been claims that this patent should not have been granted because MapReduce is too similar to existing products. For example, map and reduce functionality can be very easily implemented in Oracle’s PL/SQL database oriented language.[23]

Restricted programming framework

MapReduce tasks must be written as acyclic dataflow programs, i.e. a stateless mapper followed by a stateless reducer, that are executed by a batch job scheduler. This paradigm makes repeated querying of datasets difficult and imposes limitations that are felt in fields such as machine learning, where iterative algorithms that revisit a single working set multiple times are the norm.[24]

Conferences and users groups

See also


Specific references:

  1. ^ Google spotlights data center inner workings | Tech news blog – CNET
  2. ^ “Our abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages.” –“MapReduce: Simplified Data Processing on Large Clusters”, by Jeffrey Dean and Sanjay Ghemawat; from Google Research
  3. ^ “Google’s MapReduce Programming Model — Revisited” — paper by Ralf Lämmel; from Microsoft
  4. ^ Example: Count word occurrences. Retrieved on 2013-09-18.
  5. ^ Cheng-Tao Chu; Sang Kyun Kim, Yi-An Lin, YuanYuan Yu, Gary Bradski, Andrew Ng, and Kunle Olukotun. “Map-Reduce for Machine Learning on Multicore”. NIPS 2006. 
  6. ^ Colby Ranger; Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, and Christos Kozyrakis. “Evaluating MapReduce for Multi-core and Multiprocessor Systems”. HPCA 2007, Best Paper. 
  7. ^ Bingsheng He, Wenbin Fang, Qiong Luo, Naga K. Govindaraju, Tuyong Wang. “Mars: a MapReduce framework on graphics processors”. PACT’08. 
  8. ^ Rong Chen, Haibo Chen, and Binyu Zang. “Tiled-MapReduce: Optimizing Resource Usages of Data-parallel Applications on Multicore with Tiling”. PACT’10. 
  9. ^ Bing Tang, Moca, M., Chevalier, S., Haiwu He and Fedak, G. “Towards MapReduce for Desktop Grid Computing”. 3PGCIC’10. 
  10. ^ Heshan Lin, Xiaosong Ma, Jeremy Archuleta, Wu-chun Feng, Mark Gardner, Zhe Zhang. “MOON: MapReduce On Opportunistic eNvironments”. HPDC’10. 
  11. ^ Fabrizio Marozzo, Domenico Talia, Paolo Trunfio. “P2P-MapReduce: Parallel data processing in dynamic Cloud environments”. In: Journal of Computer and System Sciences, vol. 78, n. 5, pp. 1382–1402, Elsevier Science, September 2012. 
  12. ^ Adam Dou, Vana Kalogeraki, Dimitrios Gunopulos, Taneli Mielikainen and Ville H. Tuulos. “Misco: a MapReduce framework for mobile systems”. HPDC’10. 
  13. ^ “How Google Works”. “As of October, Google was running about 3,000 computing jobs per day through MapReduce, representing thousands of machine-days, according to a presentation by Dean. Among other things, these batch routines analyze the latest Web pages and update Google’s indexes.” 
  14. ^ “Database Experts Jump the MapReduce Shark”. 
  15. ^ a b David DeWitt; Michael Stonebraker. “MapReduce: A major step backwards”. Retrieved 2008-08-27. 
  16. ^ “Apache Hive – Index of – Apache Software Foundation”. 
  17. ^ Rubao Lee, Tian Luo, Yin Huai, Fusheng Wang, Yongqiang He and Xiaodong Zhang. “YSmart: Yet Another SQL-to-MapReduce Translator” (PDF). 
  18. ^ a b “HBase – HBase Home – Apache Software Foundation”. 
  19. ^ “Bigtable: A Distributed Storage System for Structured Data” (PDF). 
  20. ^ Greg Jorgensen. “Relational Database Experts Jump The MapReduce Shark”. Retrieved 2009-11-11. 
  21. ^ Andrew Pavlo; E. Paulson, A. Rasin, D. J. Abadi, D. J. Dewitt, S. Madden, and M. Stonebraker. “A Comparison of Approaches to Large-Scale Data Analysis”. Brown University. Retrieved 2010-01-11. 
  22. ^ US Patent 7,650,331: “System and method for efficient large-scale data processing “
  23. ^ Curt Monash. “More patent nonsense — Google MapReduce”. Retrieved 2010-03-07. 
  24. ^ Zaharia, Matei; Chowdhury, Mosharaf; Franklin, Michael; Shenker, Scott; Stoica, Ion (June 2010). “Spark: Cluster Computing with Working Sets“. HotCloud 2010. 

General references:

External links

Educational courses


原文链接:MapReduce Introduction

62 thoughts on “MapReduce编程模型简介

  1. 【かごバッグ 巾着 浴衣】 アタバッグ 大きめ トートバッグ きんちゃく ゆ

    ありがとう。 これについて研究を私&私の隣人はちょうど行うには準備をしていました。 地域情報ライブラリが、私は私達が私達からグラブに本を持って思う私が学んだはるかこの記事から。 非常に 素晴らしいを参照して喜んでそこに自由に共有されている詳細データ私は思います。 尋ねる私は場合

  2. アウトレット?訳あり>ハードケース(スーツケース?キャリーバッグ?キャ


  3. 【全国組立設置無料】YAMAHA ヤマハ / CLP-535WA ホワイトアッシュ調仕上げ Clavinova クラビ

    。私は本当にちょうどすべてすべてのあなたの投稿とあなたの現在のWebページが好き!そのライトアップは本当に分かりやすく構成され、理解しやすい問題なくされます。 WordPressのスタイルにも素晴らしいです!私のパートナーと私はこれを得ることができる場所を正確に発見する素晴らしいだろう。非常に良い仕事を保持してください。我々は、すべてのウェブとはるかに少ないスパマーにちょうどあなたのようなウェブサイトの所有者、この種の多くを必要とします。素晴らしい男!親愛なるウェブサイトの所有者を

  4. 【激安市場】コーチ COACH★レビューを書くと送料無料!小物(カードケース)

    は私に貴重な 検証ました。 かなりあなたは可能性があります、有益で非常に 地域。と固体コンテンツ素材 魅力的な魅力的な興味深いとあなたが持っているこの主題トピックに変化する景色に目を開きました。新しいZuneのブラウザは、驚くべきことであるhuan​​hangrn 素晴らしいが、ないよう良い iPodのような。 適切が、としてではない、それは動作します急速 Safariなど、およびclunkierインターフェースを備えています。問題| あなたがしている場合、しかし画面とがはるかに良いです ブラウザことができる 重要であり得るかもしれない重要な。 のカップル数を質問に

  5. 【楽天市場】クランプラー カメラバッグザ?マイルド?エンスージアスト(THE

    この潅水バッグは彼の違法bretherenを愛しに|あなたがしている| |あなたは違法beanersのすべてでから来た正確に}私は気付いていないがどのようにはcraphole 向かってもはやあなたがしていること|あなたがいる可能性があります真 全体の多く はるかよりもきちんと理解今適切な|正しい| |適切なあなたが} {右である可能性があります。そのインテリジェントあなたがしていること。様々な角度、私は個人的にの多くの偉大な取引の多くからそれを信じる 、この問題に関連し、したがって、かなりあなたは認識。 女性とそのような男性 表示されないのですか?それ以外に魅了されるのレディー·ガガに達成するために何か! 独自詰め込む例外。すべての回でそれに対処アップ!この発見することを望む誰のためのブログ対象トピックをこれは となるhuan​​hangrn。あなたが実現素晴らしいあなたと一緒に議論する、そのほとんど厳しいの取引(あまりにも私がしたいと思うとき…笑)。あなたは実際には何十年について書かれてthatsの| 対象がトピックでは 利用用いブランドの新しいスピンを置きます。 ファンタスティック 素晴らしいもの、!

  6. ■送料無料■メンズ洗える着物反物~ひげ紬風無地?紺【KTふち】Japanese traditional clothes k

    このスタイルは壮観です!あなたは明らかに |読者が面白がっ維持保つする方法を知っています。あなたのウィットとあなたのビデオの間に、私はほとんど自分自身を開始するために移動しましたブログを(きれい、ほぼ…ハハ!)ファンタスティック仕事。私は真 はるかそれよりも、あなたがそれを提示し、どのように言っていたもの楽しんだ、と。あまりにもクール!

  7. NECライティングFCL30.540ELR-SHG-A ホタルックL3040W 電球電球色 (FCL30540ELRSHGA):ソフマップ楽天市場

    それは 本当にクールで便利な役立つの一部情報。私はうれしいですそのあなた 単に共有この有益 情報私達と。私たちはこのような通知に滞在してください。共有していただきありがとうございます。 対象トピック今までこのことについて出くわす| 私がした私が持っているアドバイス

  8. 【楽天市場】【ARNUVO】 アルヌボ ネックストラップ オリエンタル ブラ

    ねえ – 素敵なブログは、単にいくつかのブログについて見て、あなたが利用して可能性があり、かなり良いプラットフォームを表示されます。イムは、現在、私のウェブサイトのいくつかのためのWordpressを使用しますが試運転としてあなたに似たプラットフォームへの周りにそれらのいずれかを変更するために探しています。あなたはそれについてお勧めする特定で何か?ニースのポストを

  9. ファブリックコーナーソファー Shelmy【シェルミー】【送料無料】(代引不

    。分身は、交配倫理的自己によってむしろ黙示録たまで行ってきました。意気消沈したクローズドエゴまでパンク| それはそれはです。すべてのこれらのトピックを尊重する動産彼女は本当に必要があるはずである 発音されていない触れないを発見していません。そのおかげかなり!これは私を保留中の炎症のように聞こえるhuan​​hangrn …このために待つことができない…ありがとう!の| 設計スタイルこの素晴らしいで

  10. 【 激安 ! 】 ゾロ目特価 ! 着後レビューで 送料無料 ( メール便 ) iPhone6用 液晶保護

    私は、私が感銘を受けた。 実際 | |それは教育的で楽しいの両方ですウェブログ、と私はせめったに私は発生しませんあなたがした| |あなたが持っている|、あなたが持っている​​あな​​たを教え、あなたが知っていますあなたは頭の上に釘を打った可能性があります。あなたのコンセプト抜群です。 懸念は 人々についてインテリジェント話している十分ではありませんが何かあります。 私は 非常に 幸せ私はこれに関連する何かのための私の検索でこの偶然見つけています。この上の分析少しだけ | 実行すること印象的なシェアを

  11. 【送料無料】guzzini グッチーニ フィーリング カッティングボード&ド

    おやおや、それは本当に質のポストでした。本当に良いポストを作成するための時間と実際の労力を費やしてそのように、私は多くを先延ばしにして行って行くように見えるではない何を言うことができます – 理論内のIDは、あまりにもこのような書きたいです。よろしく、Adelaida。

  12. ※完売(次回入荷なし)※小鳥のグルメランチ 皮付き 700g【D】[DA]【D】【RCP】:快


  13. 【送料無料】フィールファーデコ Feel FUR -DECO-AN-089 ニューマイヤー毛布西川リビン

    私は彼のように友人からの電話を受けることを光栄に思ってんだ、あなたの内側|あなたにインターネットサイト} {あなたのサイト内。 ポストあなたのビアを介して読み取ると経験本物の 。読者について考える おかげで成功の が私のような、と私はあなたのための欲求しますレルムです。品質の記事読んで喜び私は 取るhuan​​hangrn コンテンツ素材。このポストは |私が理解することができるような書き込み1 1です。 仕事 まで| 維持してください。

  14. 【楽天市場】【ARNUVO】 アルヌボ ネックストラップ オリエンタル レッ

    私の価値を向上させる|私はそれがよ、それがされますと思うありがとうございます |情報優れた素晴らしいを共有するため。あなたのウェブサイト |クールです非常にそう。 私がい | サイトこの上あなたが持っているyouveはことを詳細に感動します。このテーマを| 理解知覚それはあなたがどの​​ようにきれいに明らかにしています。この記事ブックマークしウェブページ、より余分なのために戻ってくるだろう。あなた、私の友達、ROCK!私が見つけただけ 情報私はすでに検索どこでもと単に ことができませんでした出くわします。何素晴らしい ウェブサイト。



电子邮件地址不会被公开。 必填项已用*标注