Monday, June 27, 2011

Cassandra Hector Broken Pipe Recovery


I found tons of these exceptions and a posting that this was a known
issue in 0.7.0-18 (?). Unfortunately I was running 0.7.0-28. This
happened roughly when I added 2 more nodes to the cluster. While
pelops picks them up hector does not and throws these exception
forever. Has somebody experienced this already and has a clue?

2011-04-30 00:00:48,640 INFO [pool-3-thread-1]
me.prettyprint.cassandra.connection.NodeAutoDiscoverService - using
existing hosts [10.0.78.42(10.0.78.42):9160, 10.0.78.40(10.0.78.40):
9160, 10.0.78.41(10.0.78.41):9160]
2011-04-30 00:00:48,640 ERROR [pool-3-thread-1]
me.prettyprint.cassandra.connection.NodeAutoDiscoverService -
Discovery Service failed attempt to connect CassandraHost
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:
147)
at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:
156)
at org.apache.cassandra.thrift.Cassandra
$Client.send_describe_keyspaces(Cassandra.java:1019)
at org.apache.cassandra.thrift.Cassandra
$Client.describe_keyspaces(Cassandra.java:1009)
at
me.prettyprint.cassandra.connection.NodeAutoDiscoverService.discoverNodes(NodeAutoDiscoverService.java:
81)
at me.prettyprint.cassandra.connection.NodeAutoDiscoverService
$QueryRing.run(NodeAutoDiscoverService.java:56)
at java.util.concurrent.Executors
$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask
$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
150)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:
136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:
145)



I did'nt add the auto discovery service until *.22 iirc, so that may
have been a typo referring to *.28. Try it with the latest release and
let me know how it goes. Another thought - are you using
authentication per chance?



2 comments:

  1. https://github.com/rantav/hector/pull/295

    ReplyDelete
  2. Thanks for all the advice. We have had the same pipe break on us twice now. I just want to be ready for if or when it happens again.

    http://www.goldsealplumbing.com/leak-repair.html

    ReplyDelete