问题排查:haproxy会发送大量的RST包给客户端

By | 06/17/2020

现象

客户反馈在负载均衡器上能看到大量的 RST 包,如下截图:

大量 RST 包的现象

分析

跟踪其中一个流,如下截图:

跟踪其中一个异常流

仔细分析,有如下特征:

  1. 这是一个完整的 HTTP 请求过程,所有交互都已经在发送 RST 前正常完成了
  2. 只有在连接完成,准备做连接关闭的四次挥手过程中,才会有 LB 往客户端发 RST 包的现象。

于是,进一步对比了正常的流,如下截图:

跟踪其中一个正常流

发现连接其实是能够正常关闭的。

分析到这,不得其解,转而去分析 RST flag 的原因。

FIN VS. RST

FIN RST
Gracefully terminates the connection. Abruptly tells the other side to stop communicating.
Only one side of conversation is stopped. The whole conversation is stopped.
No data loss. Data is discarded.
Receiver of FIN keeps communicating till it wants to. Receiver has to stop communication.

可能发RST的情况:

One reason a device will send a RST is in response to receiving a packet for a closed socket.

由此得知,RST 包发送的原因,很可能是因为这个 socket 已经关闭了。

于是转而去看图二的异常流,发现了端倪:四次挥手过程中,第二个包(ACK)距第三个包(FIN)的时间很长,接近五分钟。

第二个包和第三个包之间的状态是FIN_WAIT_2,该状态的超时设置是 120s。

root@i-e538fkrw:~# sysctl -a | grep fin_wait
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
root@i-e538fkrw:~#

结论

当四次挥手超时时,LB会把socket回收,后续如果再收到往这个socket的发包,就会响应 RST。

参考

3,917 thoughts on “问题排查:haproxy会发送大量的RST包给客户端

  1. bitcoincasino

    I’m writing on this topic these days, bitcoincasino, but I have stopped writing because there is no reference material. Then I accidentally found your article. I can refer to a variety of materials, so I think the work I was preparing will work! Thank you for your efforts.

    Reply
  2. Pneutle

    Merck s Zostavax vaccine is about 50 percent effective, the AP reported side effects of clomid in men Hormonal imprinting is a basic biological phenomenon which was first observed, named, and defined by us more than 30 years ago

    Reply
  3. PeterPibly

    видео плетение волос с короткими волосами. Где получить кредит webmoney моментально? женщина вихорошала двух мужчин

    Reply
  4. meeneCape

    Time trends in the use of associated treatments in patients undergoing mastectomy for breast cancer, depicting the trends in rates of use of postmastectomy radiation therapy RT, bilateral mastectomy, lymph node surgery, and breast reconstruction among the breast cancer patients in our sample from the MarketScan claims database priligy and viagra combination The effects of levonorgestrel on endometrial receptivity are unclear, and the relevance of this mechanism to the therapeutic efficacy of levonorgestrel is contentious

    Reply
  5. casinosite

    I’ve been searching for hours on this topic and finally found your post. casinosite, I have read your post and I am very impressed. We prefer your opinion and will visit this site frequently to refer to your opinion. When would you like to visit my site?

    Reply
  6. https://sopharmsn.com/

    An intriguing discussion is worth comment. I do believe that you need
    to publish more about this subject matter, it might not be a taboo subject but generally people don’t discuss these issues.
    To the next! Best wishes!!

    Reply
  7. ed meds online

    Hi would you mind letting me know which webhost you’re using?
    I’ve loaded your blog in 3 different browsers and I must say this blog loads
    a lot faster then most. Can you recommend a good web hosting provider at a fair price?

    Thank you, I appreciate it!

    Reply
  8. Freree

    Рынок Поднебесной сегодня представляет крупный подбор первоклассного и привлекательного газооборудованья которое целиком и полностью можно рассматривать именно как аналог более дорогих из Европы и США. Прекрасный вариант, например, в целях введение личного производственного бизнеса. По этой причине многие начинающие бизнесмены задаются вопросом, каким образом отгрузить оборудование из Китайской Народной Республике.

    Отписаться от рассылки

    Reply
  9. kneency

    betagan fosamax spc In a government shutdown, spending for functions consideredessential, related to national security or public safety, wouldcontinue along with benefit programs such as Medicare healthinsurance and Social Security retirement benefits for seniors brand cialis online

    Reply