Redirecting output from compiler/linker in Visual Studio 2005

I ran into an "interesting" (i.e. very annoying) problem today. I was in the process of figuring out a way to know if two Windows executables or DLLs are different. Just comparing them byte-for-byte does not work; the output cl.exe is not identical even if the incoming source code is.

So I tried to use dumpbin /disasm and compare the two disassembly listing, which worked nicely. I then wrote a little Ruby script to add as a post-build step in Visual Studio:

require 'digest/sha1'
require 'tempfile'

inputfile = ARGV.shift
IO.popen("dumpbin /disasm #{inputfile}") do |io|
Tempfile.open(File.basename(inputfile)) do |tempio|
 while io.gets
   if $_ =~ /.*Dump of file.*/
     next
   else
     tempio.write($_)
   end
 end
 tempio.rewind
 File.open(inputfile + ".sha1", "w") do |outio|
   outio.puts Digest::SHA1.hexdigest(tempio.read)
   puts "Wrote SHA1 signature to #{outio.path}"
 end
end
end

but to my surprise, dumpbin insisted on passing its output to the Visual Studio output window instead of through stdout and into my Ruby program. WTF?!

I started to dig. What could possibly cause dumpbin.exe to bypass stdout/stderr when run from inside Visual Studio? I fired up Process Explorer during build and studied the environment variables of cl.exe. Is there any magic going on here? The following environment variables caught my eye:

VCBuildHelper_CancelEvent
VCBuildHelper_Command
VS_UNICODE_OUTPUT

Knowing that Google Is Your Friend, I got a few hits, which gave me a hint about Unicode output being handled differently depending cl.exe is executed outside or inside Visual Studio. It turns out that people has compiler wrappers such as STLFILT which went mad when the output from the compiler disappeared.

Was it really so difficult for Microsoft to anticipate this problem? Piping output from a compiler or linker isn't a very odd thing to do. And how necessary is it to be able to have Unicode error messages? And why didn't they fix stdout/stderr to be able to handle Unicode? -- which would have been The Right Solution to the problem in the first place.