NetworkWordCount失败之Spark Master URLs

By | 01/19/2015

Spark Streaming 和 Spark Master URLs

Spark任务在提交时,需要指定 --master 参数,表明该spark任务提交到哪个集群。


  • local

    Run Spark locally with one worker thread (i.e. no parallelism at all).

  • local[K]

    Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine).

  • local[*]

    Run Spark locally with as many worker threads as logical cores on your machine.

  • spark://HOST:PORT

    Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default.

  • mesos://HOST:PORT

    Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk://….

  • yarn-client

    Connect to a YARN cluster in client mode. The cluster location will be found based on the HADOOP_CONF_DIR variable.

  • yarn-cluster

    Connect to a YARN cluster in cluster mode. The cluster location will be found based on HADOOP_CONF_DIR.


bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999

此时默认 --masterlocal .

这看起来毫无问题,但我在一台1核1G的主机上测试该样例,却永远无法成功。 原因这位老兄已道出:


I experienced exactly the same problems when using SparkContext with “local[1]” master specification, because in that case one thread is used for receiving data, the others for processing. As there is only one thread running, no processing will take place. Once you shut down the connection, the receiver thread will be used for processing.

也就是说,spark streaming在启动时会启动两个线程: receving threadprocessing data thread .

local 模式下,spark streaming只启动了与CPU核数相同个数的线程(在我的例子中, 只有一个线程),无论如何是无法测试成功的;而在其它开发主机(4核主机)则完全不会出现此问题。




3,728 thoughts on “NetworkWordCount失败之Spark Master URLs

  1. Jefaltentda

    pharmacie auchan angouleme therapie realite virtuelle therapie de couple comment Г§a se passe , pharmacie de garde yonne therapie de couple st-jean-sur-richelieu therapies cellulaires .

  2. Josephnit

    [url=]synthroid 0.125[/url] [url=]plaquenil generic name[/url] [url=]hydroxychloroquine 0.5[/url] [url=]augmentin drug[/url] [url=]escrow pharmacy canada[/url] [url=]viagra man[/url]

  3. preargy

    iPSC CDMD1003 deletion of exons 46 51 were differentiated to cardiomyocytes and treated with AO targeting exon 45 skipping, H45A, alone or in combination with dantrolene A and B or Rycal ARM210 B side effects of tamoxifen