Cassandra Broken Pipe Exception

Hi all,
I am using hector-0.7.0-18 and i meet this problem: 
I try to use hector with failover configuration:Cluster cluster =
HFactory.getOrCreateCluster("clusterName",
"10.18.130.100:9160,10.18.130.201:9160");
every thing is OK before one of the connection's Socket meet the
SocketException: Broken pipe.
This exception was wrapped by the thrift in an TTransportException,
I check the hector's source FailoverOperator.java and i found the
function "operate(Operation<?> op)" catch this exception by catch
(Exception e) without calling the invalidate() which was used to make
sure the client was ok. "operate" simply throw this Exception to the
caller
i have to restart the application to avoid this endless loop..
is there any way to remove the broken client from the client pool? or
any other suggestion?
here is the exception
2010-10-28 14:53:55,524 - INFO  [pool-1-thread-12:CasFactory@93][] -
init the cluster with configure:10.18.130.100:9160,10.18.130.201:9160
2010-10-28 14:53:55,525 - DEBUG [pool-1-
thread-12:CassandraClientPoolByHostImpl@87][] - Borrowing client from
CassandraClientPoolImpl<10.18.130.201(10.18.130.201):9160>
2010-10-28 14:53:55,525 - DEBUG [pool-1-
thread-12:CassandraClientPoolByHostImpl@92][] - Just before borrow:
CassandraClientPoolImpl<10.18.130.201(10.18.130.201):
9160>&maxActive=50&maxIdle=-1&lifo=true&minEvictableIdleTimeMillis=1800000& timeBetweenEvictionRunsMillis=-1&blockedThreadCount=1&liveClientsFromPool.s ize=2&numActive=0&numIdle=2
2010-10-28 14:53:55,525 - DEBUG [pool-1-
thread-12:CassandraClientPoolByHostImpl@98][] - Client
CassandraClient<10.18.130.201:9160-4> successfully borrowed from
CassandraClientPoolImpl<10.18.130.201(10.18.130.201):9160>
(thread=pool-1-thread-12)
2010-10-28 14:53:55,525 - DEBUG [pool-1-thread-12:FailoverOperator@186]
[] - Performing operation on 10.18.130.201:9160; retries: 2
2010-10-28 14:53:55,525 - ERROR [pool-1-thread-12:FailoverOperator@146]
[] - Cannot retry failover, got an Exception
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
        at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.jav a:
147)
        at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:
156)
        at org.apache.cassandra.thrift.Cassandra
$Client.send_set_keyspace(Cassandra.java:455)
        at org.apache.cassandra.thrift.Cassandra
$Client.set_keyspace(Cassandra.java:444)
        at
me.prettyprint.cassandra.service.FailoverOperator.operateSingleIteration(Fa iloverOperator.java:
192)
        at
me.prettyprint.cassandra.service.FailoverOperator.operate(FailoverOperator. java:
99)
        at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(Ke yspaceServiceImpl.java:
123)
        at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSuperSlice(Key spaceServiceImpl.java:
473)
        at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSuperSliceQuery
$1.doInKeyspace(ThriftMultigetSuperSliceQuery.java:61)
        at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSuperSliceQuery
$1.doInKeyspace(ThriftMultigetSuperSliceQuery.java:55)
        at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMea sure(KeyspaceOperationCallback.java:
20)
        at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspac e.java:
58)
        at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSuperSliceQuery.execute (ThriftMultigetSuperSliceQuery.java:
54)
        at my code...
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:
153)
        at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
        at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.jav a:
145)
        ... 20 more
2010-10-28 14:53:55,526 - DEBUG [pool-1-
thread-12:CassandraClientPoolByHostImpl@168][] - Maybe releasing
client CassandraClient<10.18.130.201:9160-4>. is aready Released?
false
2010-10-28 14:53:55,526 - ERROR [pool-1-thread-12:TThreadPoolServer
$WorkerProcess@259][] - Error occurred during processing of message.
me.prettyprint.hector.api.exceptions.HectorException:
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
        at
me.prettyprint.cassandra.service.FailoverOperator.operate(FailoverOperator. java:
149)
        at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(Ke yspaceServiceImpl.java:
123)
        at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.multigetSuperSlice(Key spaceServiceImpl.java:
473)
        at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSuperSliceQuery
$1.doInKeyspace(ThriftMultigetSuperSliceQuery.java:61)
        at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSuperSliceQuery
$1.doInKeyspace(ThriftMultigetSuperSliceQuery.java:55)
        at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMea sure(KeyspaceOperationCallback.java:
20)
        at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspac e.java:
58)
        at
me.prettyprint.cassandra.model.thrift.ThriftMultigetSuperSliceQuery.execute (ThriftMultigetSuperSliceQuery.java:
54)
        at my code
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
        at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.jav a:
147)
        at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:
156)
        at org.apache.cassandra.thrift.Cassandra
$Client.send_set_keyspace(Cassandra.java:455)
        at org.apache.cassandra.thrift.Cassandra
$Client.set_keyspace(Cassandra.java:444)
        at
me.prettyprint.cassandra.service.FailoverOperator.operateSingleIteration(Fa iloverOperator.java:
192)
        at
me.prettyprint.cassandra.service.FailoverOperator.operate(FailoverOperator. java:
99)
        ... 15 more
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:
153)
        at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
        at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.jav a:
145)
        ... 20 more




This is a known issue and has been addressed in the "concurrent"
branch on github.
This branch will be merged back in to master at some point in the next
several days.
Some additional background:
http://groups.google.com/group/hector-users/browse_thread/thread/3aaf...