NetworkWordCount失败之Spark Master URLs

By | 01/19/2015
这篇文章写的

Spark Streaming 和 Spark Master URLs

Spark任务在提交时,需要指定 --master 参数,表明该spark任务提交到哪个集群。

这个参数有以下几种可能的值:

  • local

    Run Spark locally with one worker thread (i.e. no parallelism at all).

  • local[K]

    Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine).

  • local[*]

    Run Spark locally with as many worker threads as logical cores on your machine.

  • spark://HOST:PORT

    Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default.

  • mesos://HOST:PORT

    Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk://….

  • yarn-client

    Connect to a YARN cluster in client mode. The cluster location will be found based on the HADOOP_CONF_DIR variable.

  • yarn-cluster

    Connect to a YARN cluster in cluster mode. The cluster location will be found based on HADOOP_CONF_DIR.

如果没有指定该参数,如:

bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999

此时默认 --masterlocal .

这看起来毫无问题,但我在一台1核1G的主机上测试该样例,却永远无法成功。 原因这位老兄已道出:

Note

I experienced exactly the same problems when using SparkContext with “local[1]” master specification, because in that case one thread is used for receiving data, the others for processing. As there is only one thread running, no processing will take place. Once you shut down the connection, the receiver thread will be used for processing.

也就是说,spark streaming在启动时会启动两个线程: receving threadprocessing data thread .

local 模式下,spark streaming只启动了与CPU核数相同个数的线程(在我的例子中, 只有一个线程),无论如何是无法测试成功的;而在其它开发主机(4核主机)则完全不会出现此问题。

这涉及到了spark的线程模型。更深入的文档我还尚未找到。如果你有更深入的理解,欢迎探讨。

参考:

 

发表评论

电子邮件地址不会被公开。 必填项已用*标注