项目场景：
基于 SpringCloud 构建的微服务系统
问题描述
生产环境 SpringCloud Gateway 偶发 Connection has been closed BEFORE send operation 异常
reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
at reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:59)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.web.filter.reactive.ServerHttpObservationFilter [DefaultWebFilterChain]
*__checkpoint ⇢ io.sentry.spring.jakarta.webflux.SentryWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ HTTP GET "/product/products/public/hint" [ExceptionHandlingWebHandler]
Original Stack Trace:
at reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:59)
at reactor.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:295)
at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:73)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:311)
at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
原因分析：
官方解释: means that we were able to obtain the connection from the pool and the connection was alive but before sending the request the remote peer closed the connection 根据官方说法，该异常是因为 gateway 在向下游微服务发送请求前链接被关闭了。而链接被关闭有两种可能，客户端关闭链接或下游微服务端关闭链接。针对此两种情况分别进行模拟测试，发现两种情况均可复现此异常。异常复现后进一步排查出现问题的项目，发现后端微服务配置了 “socket READ_TIMEOUT”（undertow）；当客户端请求到达 gateway 后，转发到 undertow 时，如果 undertow 触发 READ_TIMEOUT 限制就会主动断开与 gateway 的链接；正是因为此行为导致 gateway 发生 Connection has been closed BEFORE send operation 异常。（具体底层逻辑仍在整理中，如需了解过程细节请留言）
解决方案：
将 slowpost 防护提前到 gateway 或者 nginx 来实现。
相关文章
发表评论