Learned something about bash and iterating over piped output

Michael Torrie torriem at gmail.com
Fri Sep 16 20:38:16 MDT 2016

Yesterday I was trying to write a script that processed output from
another process, line by line, and had problems where some of the output
would just disappear and not get processed by the subprocess I was calling.

I had a vague recollection that I'd encountered this problem before
while working at BYU and ended up dumping the output to a file and then
processing that because I couldn't figure out what was going on.  TL;DR
version: any subprocess that consumes input from standard in will
consume the input you are trying to read in the parent while loop.

Anyway, my script was trying to chop up an m4a file into individual
chapter files, using ffmpeg.  Here's the relevant part of the script in
its simplest form:

ffmpeg -i "$1" 2>&1 | grep "Chapter #" | while read chapter; do
	number=$(echo $chapter | cut -f2 -d' '| cut -f2 -d':')
	number=$(printf "%02d" $number)
	starttime=$(echo $chapter | cut -f 4 -d' ' | cut -f1 -d',')
	endtime=$(echo $chapter | cut -f 6 -d' ')
	length=$(echo $endtime - $starttime | bc -l )
	echo "$newname $number | $starttime | $endtime | $length"
	ffmpeg -i "$1" -acodec copy -vn -movflags faststart -ss "$starttime" -t
"$length" "$newname ${number}.m4a"

If I commented out the inner ffmpeg command it ran fine and I saw every
line of output from the first command.  But with it in there, I got
weird errors about bad command parsing from ffmpeg.  Even turning on set
-x I couldn't see where it was going wrong (I thought it was a problem
with the arguments being passed).  Finally I figured out that ffmpeg,
though it's a command-line batch utility, apparently reads commands from
standard-in while it's running. And the output from the first ffmpeg was
actually ending up going to the standard in of the child ffmpeg process.
 So the fix was to just do:
	echo "" | ffmpeg -i "$1" -acodec copy -vn -movflags faststart -ss
"$starttime" -t "$length" "$newname ${number}.m4a"

That kept ffmpeg happy and the while loop kept reading lines from the
original process.  What a strange bash gotcha!  Thought I would share
this with you all in case any of you encounter this in your bash
scripting. The "while read line; do ; loop" idiom is very common in
bash, and handy too, as long as I understand that behavior.


More information about the PLUG mailing list