Whats this? IOException: Write end dead

I like Java very much. Nevertheless, there are things I cannot stand in Java. Sure, and one of them are undocumented exceptions from the Java API, that nobody can explain.

I will talk about “Write end dead” IOExceptions. What does this mean? Why did this happen? Even if googling around, it is difficult to find a clear answer.

This article is a full explanation to understand this exception and shows how to fix your code. Just continue reading…

What happens

You run your Java application and get:

java.io.IOException: Write end dead
    at java.io.PipedInputStream.read(PipedInputStream.java:244)
    at java.io.PipedInputStream.read(PipedInputStream.java:305)
    ...

Even worse, this exception is not deterministic! You program may sometimes run perfectly, and sometimes crash.

What to blame

PipedInputStream and PipedOutputStream are very sensitive to threads.

“Write end dead” exceptions will arise when you have:

  1. A PipedInputStream connected to a PipedOutputStream and
  2. The ends of these pipe are read/writen by two different threads
  3. The threads finish without closing their side of the pipe.

If you encapsulate the PipedInputStream into a BufferendInputStream and/or encapsulate the PipedOutpuStream into a BufferedOutputStream, then the problem will get even more non-deterministic.

What to fix

When using PipedInputStream and PipedOutputStream, now consider:

  • A thread that writes to a stream should always close the OutputStream before terminating.

In fact, when you read the JavaDoc for PipedInputStream, you will find an advice:

“If a thread was providing data bytes to the connected piped output stream, but the thread is no longer alive, then an IOException is thrown.”

Unfortunately, the JavaDoc does explain that this IOException is a “Write end dead” exception.

Also, consider, to avoid other types of problems:

  • The PipedInputStream should always be read by the same thread. The PipedOutputStream should always be written by the same thread.
  • Of course, the thread of the PipedInputStream should not be the same as the one of the PipedOutputStream, or unexpected deadlock might happen.
  • In general, any InputStream or OutputStream should never be shared among different threads.

What to know, but that is easily fogotten

Pipes are naturally intended to be accessed by two threads, where one thread writes (produces) data to the pipe, and the other reads (consumes) the data.

Those threads may be running at different speeds. If the producer is faster than the consumer, the pipe will buffer some amount of data. But if even the buffer gets full, then the producer is put to sleep until the consumer reads more bytes and frees again some space on the buffer.

If the consumer is faster than the producer, then the buffer will become empty and the consumer will be put to sleep, until the producer writes enough bytes to the pipe.

When the producer has nothing more to write to the pipe, then it should close the stream. The consumer will be signaled with a special error when the pipes becomes empty.

When the consumer does not want to receive data anymore, then it should close the stream. The producer will be interrupted with a special error the next time it tries to write to the pipe.

Example

Suppose this typical example of producer/consumer:

The producer class:

public class Producer extends Thread {
    int interval;
    int iterations;
    OutputStream os;
    byte data[] = { 'a', 'b', 'b' };

    public Producer(int interval, int iterations, OutputStream os) {
        this.interval = interval;
        this.iterations = iterations;
        this.os = os;
    }

    @Override
    public void run() {
        try {
            System.out.println("Producer started");
            for (int i = 0; i < iterations; i++) {
                System.out.println("Produce...");
                System.out.flush();
                os.write(data);
                synchronized (this) {
                    this.wait(interval);
                }
            }
            os.close();
            System.out.println("Producer finished");
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

The consumer class:

public class Consumer extends Thread {
	int interval;
	InputStream is;
	static final int max = 10;
	byte data[] = new byte[max];

	public Consumer(int interval, InputStream is) {
		super();
		this.interval = interval;
		this.is = is;
	}

	@Override
	public void run() {
		try {
			System.out.println("Consumer started");
			int amount;
			while ((amount = is.read(data)) >= 0) {
				String s = new String(data, 0, amount);
				System.out.println(MessageFormat.format("Consumed {0} bytes: {1}", amount, s));
				synchronized (this) {
					wait(interval);
				}
			}
                        is.close();
			System.out.println("Consumer finished");
		} catch (IOException e) {
			e.printStackTrace();
		} catch (InterruptedException e) {
			e.printStackTrace();
		}
	}
}

The main method:

public class Tester implements Runnable {
	public void run() {
		PipedOutputStream os = new PipedOutputStream();
		PipedInputStream is = new PipedInputStream();

		try {
			is.connect(os);
		} catch (IOException e) {
			// This is not possible, as piped streams
                        // are being used correctly.
			assert false;
		}

		Producer producer = new Producer(5, 10, os);
		Consumer consumer = new Consumer(5, is);
		producer.start();
		consumer.start();
	}

	public static void main(String[] args) {
		Tester tester = new Tester();
		tester.run();
	}
}

This will produce something like (order of output varies from execution to execution):

Producer started
Produce...
Consumer started
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 6 bytes: abbabb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Producer finished
Consumer finished

If the lines:

os.close()

and

is.close()

are removed, then the streams will not closed. That means that the producer thread will finish while the consumer thread is still running and trying to read. The consumer will be interrupted with a “Write end dead” exception.

The output of the broken implementation:

Producer started
Produce...
Consumer started
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 6 bytes: abbabb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Produce...
Consumed 3 bytes: abb
Producer finished
java.io.IOException: Write end dead
	at java.io.PipedInputStream.read(PipedInputStream.java:244)
	at java.io.PipedInputStream.read(PipedInputStream.java:305)
	at java.io.InputStream.read(InputStream.java:89)
	at Consumer.run(Consumer.java:23)

19 Responses to Whats this? IOException: Write end dead

  1. Thanks a lot for sharing this. I have been wrestling my Piped*Streams for hours now, until I found this post – which explained it all.

  2. I had an issue where I have a “startup” thread create a bunch of worker objects like:

    Worker worker = new Worker();
    try {
    worker.initialize();
    }
    catch …. { }

    Then I place the worker on a thread pool, and when the pool’s thread tried to read from the pipedInputStream, I got the very instructive “write end dead” message.

    Once I changed the initialization of the pipes to the worker thread, it (and by extension I) was happy.

  3. Andy Cohen says:

    Great explanation! Thanks!

  4. Corey says:

    Thanks for the explanation! Saved me a lot of grief.

  5. Ali says:

    > A thread that writes to a stream should always close the OutputStream before terminating.

    Thanks, that was really helpful!

  6. Daniel Hardman says:

    Bless you! I’d been gritting my teeth and debugging for a couple hours when I found your post — all is now clear!

  7. Daniel says:

    Great post. Saved me a lot of time. Thanks!

  8. Josh says:

    ahh… now I can move forward!

  9. twb says:

    Thanks man!

  10. Guy says:

    Thank’s a lot, very clear . good illustrations . i spend a day before reaching to your advice .

  11. Sergio Moreno says:

    Thank you for the post. I spend a few hours to make this work fine.

  12. Martin says:

    This is the most useful post I have ever seen. Thank you !!!

    Problem has been solved by:

    >> A thread that writes to a stream should always close the OutputStream before terminating.

  13. Guy says:

    Thank you!

    This post was really helpful :)

  14. Pingback: Flaws with PipedInputStream/PipedOutputStream | PHP Developer Resource

  15. Mike Hummel says:

    Hm! Excuse I’m not ready to accept that a stream should be written / read only from one thread. There are a couple of situations that’s necessary. I understand that only one thread should be ‘halted’ on read / write. But do not understand why another thread is not allowed to do the same in order. There are two solutions: 1. A reader queue, 2. The second calling thread get an exception as long as another is waiting for the call to return.

  16. Christoph Brändle says:

    I agree with Mike Hummel in saying that there are reasons where there can be more than one producer. I have an application with one consumer thread acting as a “concentrator”, and several producer threads. At the beginning, I also got “write end dead” errors in the consumer in some situations. From another thread, I learned that the pipe “remembers” the last thread that was writing to the queue, and if this former producer thread exits even if the thread has nothing to do with the pipe at all any more, the consumer gets this “write end dead” error.
    I then implemented the following hack: after reading the data, the consumer just executes “os.flush()”, even if this thread never writes to the output pipe.
    I am not absolutely sure that this is the final solution, but at least until now I did not get “write end dead” errors any more.

  17. Ranjit Aneesh says:

    I had a problem with write end dead that I could not fix even after taking all the care as mentioned here. Finally I figured out that there is a “Java Feature restriction” with the way PipedInputStream and PipedOutputStream are designed. In short they are not designed for reading / writing by Threads external to the JVM. See http://ranjitaneesh.blogspot.in/2014/08/pipes-in-java.html for more details.

  18. ericjs says:

    It would seem it is also possible to get a “Read end dead” from (I think) similar circumstances.

  19. Pingback: Flaws with PipedInputStream/PipedOutputStream | Ziada

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 390 other followers

%d bloggers like this: