InstallShield

I'm starting to regret that I recommended to my manager that we use InstallShield to build our installers. InstallShield does have a few nice features, such as allowing side-by-side installations of the same product, but from a developers point of view, it is so awkward to work with that I get frustrated each time I try to do something. The list of annoyances is getting rather long:

• The COM automation interface is not complete: there is no way to create a new InstallShield project, certain attributes on components are not accessible, and you can add "previous media" in order to create an update installer. This forced me to manually patch the XML project file.
• Silent operation is clunky at best, and just non-functional at worst. You need to twiddle around with response files, non-existing error reporting, weird command line flags, and there is no guarantee that it works. Why not have one single flag which runs the installation using all default values? And why do I need to supply a response file for uninstalling? Just remove the damn product!
• InstallScript: this is a Pascal-ish language which just makes my brain hurt. There is no design behind it, data-structures have been added in an ad-hoc manner, and it's just painful to use. At least to people like me who are used to more modern programming languages such as Ruby and Python. (Actually, I'd choose just about any language before InstallScript: Perl, Java, Delphi, C, C++, Lisp, Smalltalk, you name it. Well, maybe not Cobol.)
• There are several types of installers you can create: InstallScript-based, MSI-based, and a mixture of them. (And a bunch of others as well.) I would like to have different user manuals depending on which installer type you're using. The current manual is one monolith which is unclear all over the place about which options, functions, methods, etc. that applicable to each project type.

Oh, well. Now I'll stop whining and get back to work.

CruiseControl

Sometimes you stumble across a tool you wish you had started to use several years ago. A tool which solves (or at least simplifies) a lot of the problems you have when you're developing software. CruiseControl is such a tool; and I had had it on my radar for a couple of years but never gotten around to actually start using it until a few weeks ago.

CruiseControl (or one of its cousins: CruiseControl.NET, CruiseControl.rb) is a tool used to support Continuous Integration, a software development practice which is characterized by frequent commits (usually at least once a day), and regular automated builds several times per day. We haven't gotten around to adopting CI completely, but CC is still a great framework to take care of your automatic build. Here are some highlights:

• It has nice web dashboard for monitoring the state of the builds. It tracks the time each build takes, and when a build is in progress it estimates time left based on the previous build.
• It is easy to configure.
• It can automatically trigger builds when commits are made to the source repository.
• It can automatically send mails to all users which have participated in the last build (i.e. have commited code since the last successful build). If you want it to it can send mail to everyone for every build, but with several builds per day that quickly becomes annoying.

If you aren't using CruiseControl for your automatic building, you should do so. Now.

GMail and IMAP

Finally the people at Google have gotten around to letting people access their GMail accounts via IMAP, and finally my account has been enabled too (it took a few days). I've wanted this almost since the very beginning (for some 2 years ago), and I gather from the reactions around the web that I'm not alone. Google on "gmail imap" and you get some 6 million hits, many (of course) referring to GMail now getting IMAP support, but a sizable amount of hits are from before this and talk about how to get IMAP support for GMail in other ways.

Half-Life 2: Episode Two

I finally finished Episode Two yesterday. This time, I set the difficulty level to "hard" directly. When I've played the other Half-Life games, I've always started on "medium" (or even "easy", I think), but this time I decided to take the difficult route the first time through. And Valve really makes you earn your kills.

A fun thing about Episode Two (and Team Fortress 2) is there is a set of "achievements" (or sub-missions); such as breaking every web cache in the caves, killing all the antlion grubs, etc. There is also a fun achievement involving the gnome you find at the very beginning of the game.

The first really tough part was trying to kill two Antlion Guards. You don't have enough ammo (even if you've fully stocked on the shotgun and the MP5) to take them out, so you'll have to be a little creative with explosive barrels and other things you can shoot at them. To make things more fun you have the antlions to worry about too.

The second fun part was the helicopter battle where you desperately try to find a rocket launcher just to realize that there isn't any. The point is to take it out by taking the mines it drops and throw them back at the helicopter. Ah, the joy of crawling around and being shot at by helicopters!

Then there is the Hunters. The first time you encounter them is when Alyx is injured, but the first time you have to take one out is later on when you and Alyx are attacked by a pack of them in a farm house. Also a fun scene, where you have to be a little creative with the gravity gun.

Then there is the final battle. You've reached the White Forest rocket base, but it's being attacked by Striders. And not one, two, or three, but at least 8 or 9. I didn't keep count. This is where the real fun begins. And to make it even more challenging, each Strider is escorted by 2 or even 3 Hunters.

This game is nothing less than a masterpiece. And you haven't really played it until you've played it with the difficulty set to "hard".

Xcerion

Xcerion is a new Swedish startup which is working on a "internet operating system", something which runs entirely inside your webbrowser. The market-speak on their website gives the impression that it will revolutionize the computer industry and bring down Google and Microsoft and the same time. That is, of course, not true. They will be absorbed by either long before that happens.

I tried to get some hard details about what they're actually doing, but details are scarce, and any information I could get hold of is full of market-speak. Things like the following sounds like something Dilbert's boss would say:

"Given that everything is defined in XML creates enormous synergies and unleashes the innate power of XML,"

I'm not sure why they push XML so hard. XML is an underlying code/data representation, and will not cause any magic to happen by itself. XML may very well be the best choice of a representation, but it is not a silver bullet.

The fact that all applications are written in XML rings a big, fat, warning in my head. We've heard several claims over the last 20 years about new technologies which will render "traditional programming" obsolete, but we're still here writing code in C++, Java, Erlang, Ruby, Python, etc. What's the common denominator of the 20 most popular programming languages today? Their source code is written using text editors. Yes, some text-editors are most advanced than others, but we're still writing code. Practical, general purpose point-and-click programming won't be here for years.

XML is not designed to be human-editable. This forces you to have top-notch graphical editors on top of everthing, and that is not an easy task. Many have tried and failed (the Eclipse plugin.xml editor is a good example; it is not a bad editor, but it doesn't scale very well for non-trivial XML).

But then again, if Xcerion can deliver on their promises and replace the traditional desktop OS with a system which is light-weight, fast, runs everywhere, and fun to program (sort of like on the old Amiga days), I'm all for it. But I can't help but feeling sceptical about the whole thing.

Xcerion, please prove me wrong.

San Andreas

What do you play when you're waiting for Half-Life 2: Episode Two to be released, when Team Fortress 2 takes forever to load, and when you've completed Half-Life 2 with difficulty level set to "hard"? Well, you can always dig up Grand Theft Auto: San Andreas, and try to get a little closer to those 100%. After cruising around in San Fierro for a while and destroying a motorcycle I found my way to the Blood Bowl race, and decided to give it a try. It's quite hard, and after 20 attempts or so I managed to hold on long enough for the other cars to destroy themselves, making it a little easier to get to the checkpoints.

The lesson here is: there is always something to do in San Andreas.

Writing a toolchain plugin for CDT (or: things not to use XML for)

I'm currently in the process of implementing support for a compiler toolchain for Eclipse/CDT. This means writing a large chunk of XML code describing how the toolchain works (what tools there are, what kind of inputs they take, what options they take, what build configurations are available, and so on). A typical standard thing to do when you want to use CDT with a compiler other than GCC.

Eclipse/CDT is very flexible when it comes to supporting other kind of toolchains; anything which can be broken down to a series of external programs which which interact to produce output from a given set of inputs. You can implement a toolchain for creating PDF files from LaTeX input, although this has little to do with C/C++ development. This great flexibility is actually the first warning flag. Yes, flexibility is a good thing, but the user's have to be able to wield it as well.

Most of the toolchain is defined in the plugin.xml. This is where all tools, configuration, projects type, options, etc. are defined. This is actually the second warning flag. My plugin.xml is at 2000 lines and counting, and I haven't really done anything complicated. A bunch of options, two different project types, two build configurations, and 4 tools (c-compiler, c++-compiler, linker, assembler). Yes, I could probably break out much of this in separate plugins, but that seems a little overkill.

CDT has a system for inheritance among elements in the toolchain definition file. I can define a common base class for my C and C++ compilers, and define C++-specific options in a subclass. What CDT does is basically to paste a primitive inheritance system on top of XML. It works like this:

<option
  id="com.foo.bar.option.parentOption"
  isAbstract="true"/>
<option
  id="com.foo.bar.option.subOption"
  superClass="com.foo.bar.option.parentOption"
  ... />

This may sound great at a first look, but it quickly turns out to be hopelessly inadequate. It's ok to do small things this way, but when you have three different tools with similar set of options you need a way to be able to share these in a more flexible manner.

One of my favorite programming principles is DRY: Don't Repeat Yourself. While writing CDT toolchain definitions you have to break this rule so many times, it's almost physically painful. For example: each XML element has to have a globally unique identifier, and it has to be specified in full in the element itself. In the plugin.xml file which contains my toolchain definition, 25% of the lines includes the reverse-DNS name ("com.foo.bar.plugin", for example) used to make sure that the identifier are globally unique.

There is an option for my compiler which takes 20-30 enumerable values (the target cpu type). To specify this, I have to write an XML element for each of these values:

<enumeratedOptionValue
  command="--cpu FOOBAR1"
  id="com.foo.bar.plugin.toolchain.compiler.generic.option.cpu.FOOBAR1"
  name="FOOBAR1"/>

Instead of allowing me to have a simple list of all cpu types and generate each option, I have to manually (cut-n-paste) expand the enumeration value definitions.

I think it would be better to write the toolchain definition in Java instead. The idea of defining things in the plugin's plugin.xml file is to let Eclipse know things about plugins without having to load the plugin itself, but does this include details about which options my compiler takes? If plugin.xml just defined the entry point classes for my toolchain in the plugin.xml file, I could define the toolchain by creating the appropriate Java objects. This would allow me to use the full power of Java's inheritance system and I can store option metadata (such as the number of available cpu types) in a way most suitable for my needs.

Heroes

I've been catching up on the first season of Heroes (the first seasons
airs here in Sweden on TV4 twice a week), and I must say that it is
the most intriguing TV-series I've watched since the first season of
24. The combination of a compelling story arc and superheroes (without
tights!) makes for very enjoyable TV, especially when most other
programs are some form of reality-tv (home makeover shows, American
Idol (or the swedish equivalent), etc.) which I really can't stand.

And I like the more or less subtle references to Star Trek (there are
more than just Hiro doing the Vulcan salute).

In his father's footsteps

Fantastic photo!

MFC Sucks. Badly.

I've done a fair amount of GUI-programming over the last 10 years, and there are a lot of toolkits out there of varying quality. None of them stand out as "excellent" in my eyes, but there is one which stands out as particularly lousy. MFC.

MFC is a patchwork of old (and in many cases non-existing) design, backward-compatibility fixes, API limitations, bad documentation, and bugfixes. Just to take a few examples:

• The OnVScroll() method which is called when a window's vertical scrollbar changes only gets the position as a 16-bit integer. 32-bit integers are supported, but you need to call GetScrollInfo() to get it.
  
• OnVScroll() gets a pointer to a CScrollBar control, but this pointer is unusable (and in most cases NULL). Instead, you need to call GetScrollInfo() to get a reference to the scrollbar.
  
• CListCtrl can be used as a "virtual listview", i.e. you specify the number of lines and a callback which is invoked each time the view needs the data for a given row. Useful when you need a large number of rows and fetch data on demand (such as displaying memory contents). However, it refuses to use more than 100 million elements. And, yes, this is then power-of-ten kind of million, not the power-of-two you might have expected in case the widget uses a part of the 32 bit index integer for tagging or such. And all documentation on this, including web articles and code samples consider 100 million to be "huge" and "enormous". If you have a 32-bit integer as index, why can't I have 2^32 entries in the view.
  
• You can't set CListCtrl's vertical position by setting the position on the scrollbar.
  
• If you want to change colors in individual cells in a CListCtrl(), you must implement your own drawing function from scratch. There is no way to enhance the list control's own implementation. You might as well reimplement the entire list control from scratch (which many people seem to have done.)
  
• There is no way (short of black magic and sacrificing green goats) to get CListView to use another implementation of CListCtrl. There are lots of variants of CListCtrl on the web, but exchanging the one ClistView uses is easier said than done.
  
• I needed to add a text-display window below my CListView which was to display details about the selected row. This required 4 (!) extra classes with ~500 LOC. Compare this to XMonad, a tiling X11 window manager implemented in about the same amount of code. But they used Haskell, not MFC. ;-)
  

Yet Another Annoying Thing About Visual Studio

I'm getting more and more annoyed by Visual Studio for every day I use it. Today's annoying thing is broken .sbr files.

Visual Studio has the annoying habit of leaving broken .sbr files around. I'm not sure when or why they appear, possibly when a build is interrupted. When bscmake at a later point tries to read these broken files, it croaks and causes the entire build to fail.

Every single other compiler in the world has the good sense of not leaving incomplete files on disk when exiting. But not Microsoft's. And it's not like this is a new bug; a quick check on google showed that this bug has been there at least since 2000.

Redirecting output from compiler/linker in Visual Studio 2005

I ran into an "interesting" (i.e. very annoying) problem today. I was in the process of figuring out a way to know if two Windows executables or DLLs are different. Just comparing them byte-for-byte does not work; the output cl.exe is not identical even if the incoming source code is.

So I tried to use dumpbin /disasm and compare the two disassembly listing, which worked nicely. I then wrote a little Ruby script to add as a post-build step in Visual Studio:

require 'digest/sha1'
require 'tempfile'

inputfile = ARGV.shift
IO.popen("dumpbin /disasm #{inputfile}") do |io|
Tempfile.open(File.basename(inputfile)) do |tempio|
 while io.gets
   if $_ =~ /.*Dump of file.*/
     next
   else
     tempio.write($_)
   end
 end
 tempio.rewind
 File.open(inputfile + ".sha1", "w") do |outio|
   outio.puts Digest::SHA1.hexdigest(tempio.read)
   puts "Wrote SHA1 signature to #{outio.path}"
 end
end
end

but to my surprise, dumpbin insisted on passing its output to the Visual Studio output window instead of through stdout and into my Ruby program. WTF?!

I started to dig. What could possibly cause dumpbin.exe to bypass stdout/stderr when run from inside Visual Studio? I fired up Process Explorer during build and studied the environment variables of cl.exe. Is there any magic going on here? The following environment variables caught my eye:

VCBuildHelper_CancelEvent
VCBuildHelper_Command
VS_UNICODE_OUTPUT

Knowing that Google Is Your Friend, I got a few hits, which gave me a hint about Unicode output being handled differently depending cl.exe is executed outside or inside Visual Studio. It turns out that people has compiler wrappers such as STLFILT which went mad when the output from the compiler disappeared.

Was it really so difficult for Microsoft to anticipate this problem? Piping output from a compiler or linker isn't a very odd thing to do. And how necessary is it to be able to have Unicode error messages? And why didn't they fix stdout/stderr to be able to handle Unicode? -- which would have been The Right Solution to the problem in the first place.

What happens during ExitProcess()

"The Old New Thing" is an excellent blog if you want to know things about Windows which is not in the documentation. Such as how ExitProcess works:

"Exiting is one of the scariest moments in the lifetime of a process. (Sort of how landing is one of the scariest moments of air travel.)"

The quest for a real build system, part 1

I've been looking for a "real" build system for quite some time, something which allows me to correctly and completely specify the relationsships between all elements of my program in a compact and easy-to-maintain sort of way. I didn't think it would be so hard, but apparently it is.

Take Visual Studio, for an example. If I build an executable in directory A and also want to copy it to B, the only way to do that (except for possibly creating a special project for the task) is to add a post-build step which copies the executable from A to B. However, there is no way to make Visual Studio guarantee that the executables actually is copied. If the copy-step fails, it will never be executed again unless the executable itself is rebuilt. Since Visual Studio does not know (or care) about what happens in a post-build step, it is impossible to know if all post-build steps in a project actually succeeded. In other words, post-build steps are useless for anything related to the actual build process.

SCons does this right. Its specification files are Python programs and you can manipulate nodes in the dependency graph exactly as you wish. Given that you've specified your dependency graph correctly, you can actually run scons and it will rerun exactly those commands necessary to bring your targets up-to-date. As a result, you really don't need a "clean" command (even if there is one).

Unfortunately, SCons has serious performance issues and does not scale well for very large systems. But sometimes I'm willing to pay a lot of performance for actually doing correct builds.

I've been using CMake at work for some time, and it has some nice features which makes it the best there is at the moment. The fact that it can generate Visual Studio project files, while still giving command-line people like me a Makefile is a big plus.