We are seeing these errors on the client side sporadically.
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: channel closed at io.grpc.Status.asRuntimeException(Status.java:532) at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:398) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at (ThreadPoolExecutor.java:1128) at (ThreadPoolExecutor.java:628) at (Thread.java:834) Caused by: java.nio.channels.ClosedChannelException: null at io.grpc.netty.Utils.statusFromThrowable(Utils.java:166) at io.grpc.netty.NettyClientHandler.onConnectionError(NettyClientHandler.java:474) at io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:641) at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:225) at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:53) at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:157) at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:141) at io.grpc.netty.NettyClientHandler.createStream(NettyClientHandler.java:543) at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:312) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:708) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:791) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:701) at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1026) at io.netty.channel.AbstractChannel.write(AbstractChannel.java:288) at io.grpc.netty.WriteQueue$AbstractQueuedCommand.run(WriteQueue.java:174) at io.grpc.netty.WriteQueue.flush(WriteQueue.java:112) at io.grpc.netty.WriteQueue.access$000(WriteQueue.java:32) at io.grpc.netty.WriteQueue$1.run(WriteQueue.java:44) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: java.nio.channels.ClosedChannelException: null at io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:955) at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:863) at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1378) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:708) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:791) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:701) at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeadersInternal(DefaultHttp2FrameWriter.java:528) at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeaders(DefaultHttp2FrameWriter.java:268) at io.netty.handler.codec.http2.Http2OutboundFrameLogger.writeHeaders(Http2OutboundFrameLogger.java:60) at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:208) ... 22 common frames omitted Caused by: java.io.IOException: Connection reset by peer at (Native Method) at (SocketDispatcher.java:47) at (IOUtil.java:113) at (IOUtil.java:58) at (IOUtil.java:50) at (SocketChannelImpl.java:466) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:405) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:928) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:356) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895) at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1383) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:749) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:741) at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:727) at io.grpc.netty.NettyClientHandler.sendPingFrame(NettyClientHandler.java:646) at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:318) ... 17 common frames omitted Both the server and the client run in Kubernetes with the server being a ClusterIP service. The client's channel builder looks like this:
NettyChannelBuilder.forAddress("some-service", 8090) .nameResolverFactory(DnsNameResolverProvider.asFactory()) .defaultLoadBalancingPolicy("round_robin") .idleTimeout(60000, TimeUnit.MILLISECONDS) .usePlaintext(); The client sends requests in a burst of 4-5 (concurrently using project-reactor) every 5 minutes. We see failures happening once or twice a day. We have tried setting both keepAlive and idleTimeout on separate occasions but nothing seems to be working. We have retries defined for UNAVAILABLE and DEADLINE_EXCEEDED status codes but not UNKNOWN since that could mean retrying on potentially non-retriable errors. Is there a way to fix this on the client side without having to retry on UNKNOWN errors?
Client gRPC version: 1.20.0 Server gRPC version: 1.20.0 Netty Version: 4.1.39.Final
2Related questions 1 grpc-java server shuts down immediately 1 Getting IllegalStateException when starting the grpc server 2 grpc gives "INTERNAL: Connection closed with unknown cause" Related questions 1 grpc-java server shuts down immediately 1 Getting IllegalStateException when starting the grpc server 2 grpc gives "INTERNAL: Connection closed with unknown cause" 3 Channel closed when running simple example with TLS 0 gRPC Java Client - hasNext during onNext? 2 gRPC "call is closed" exception when calling onNext on server 20 gRPC: Random CANCELLED exception on RPC calls 10 Channel ManagedChannelImpl was not shut down properly 0 Java gRPC: exception from client to server 8 gRPC client failing with "CANCELLED: io.grpc.Context was cancelled without error" Load 7 more related questions Show fewer related questions
Reset to default
