Idea: background building of Visual Studio C++ projects

Sometimes I come up with crazy ideas about programs or scripts that could do magic.

Since I don't have time or knowledge to implement them all, I decided at least describe them here. Hopefully somebody else finds them useful and decides to make them a reality. In this case, I will be the first to help. When you think of something long enough, it gets more and more details, single blog post cannot describe them all but just some key points only.

CPP background compiler

Ok, today's idea is about compiling source files. I often wish that Visual Studio compiled projects on a background, while you editing them. Of course, the changed files would need to be recompiled. Updating headers would require many cpp files to be recompiled... So what? Those files will take their time anyway, when you press "Build". But if you have most of the files compiled on a background, building the project from IDE would be much faster, since only the recently changed files would need to be compiled and only linkage would left.

Simple and stupid approach: build it all

Just pick up the solution or project and run a batch file which would constantly call devenv with /build switch to attempt to build the project. Make that process idle priority, so it would not affect other programs much.

Visual Studio doesn't like much if you're saving source files during compilation. If that is a cpp file which was saved, then during next build request, it should figure out that obj file has creation time less than cpp and recompile it. Not a big deal, but this shall work only if my following assumptions are true:

1. Compiler loads entire cpp file into memory before compiling it (including preprocessing)
2. Obj file creation time equals to the time when file was started to be written on disk, not when it was closed
3. The time between cpp was loaded into compiler's memory and the time obj file was opened on disk is negligibly small (or even better — obj file is opened before cpp file was read from the disk.

Of course, modifying header may cause more recompilations, since Visual Studio must figure out what cpp files depend on it and recompile them. I don't see much problems, as soon as the assumptions above are ok.

Well, almost no problems. If you update stdafx.h this means you should recompile stdafx.cpp. And if you're recompiling stdafx.cpp this means you should recompile all cpps in the project. I am not sure Visual Studio /build is smart enough to figure that out, so this is better to be tested.

One small thing:

Synchronization with Visual Studio IDE

Having another instance of compiler writing some obj file will prevent Visual Studio to compile that file from IDE. This means we should synchronize IDE with our background script. Practically this means we should stop or kill that background IDE before building project from IDE.

I see this could be done by creating a macro, which would call pskill.exe (written by Mark Russinovich from Sysinternals) before actually starting a build from IDE. To avoid running background compilation script manually after IDE build finishes, we could run it from project post-build event. However this will only work if IDE compilation was successful. Otherwise — alas, I don't see how to call a macro after build finishes with error. Maybe creating IDE add-on would help?

Ok, that seems not too efficient to build an entire project on background, since it will do scanning all of the project files and sometimes that can be a lot. The synchronization problems with IDE make it worse too.

How about we compile only cpps?

We could run compiler itself (cl.exe) on all cpp files of the project, just like IDE does. It would require to pick up project options from vcproj file, also we would need to clean the objs after compiler options were updated in vcproj (and don't forget about IDE environment variables). That seems like more difficult to do, but still nothing too special.

This way IDE would do the linkage and all post-build stuff and our background script would only compile the cpps as they get updated.

Synchronization would be less required, since the background script would do nothing after all cpps are compiled. So every time you start the build, it could just wait until the background script finishes compiling all the recently updated cpps and then start IDE build (presumably only linkage would left at this point). As a signal that he is done, the background script could create an empty file with predefined name in project directory, which could be easily checked by macro for existence. Also, this macro would need to call "SaveAllFiles" manually so background script would not wake up after IDE saves the modified source files before starting a build.

The only thing which makes complicates this approach more is that background script should watch header files too and recompile all affected cpps (having stdafx in mind too), i.e. reproduce more of the IDE logic. Having many nested include files makes this task not so trivial. However still doable.

Sounds familiar?

If we can do pretty much what IDE does with the builds, maybe we could leave it only editing of files and projects and avoid synchronization problems just by relying on background compilation scripts?

We could remap standard shortcuts so they pass commands to our "engine" and even redirect it's output to IDE build windows probably. If that cannot be done with macros, maybe add-on could work instead.

Another thing I often desperately need in Visual Studio is queueing the projects compilation. Usually this is required when you have big solution and few projects you want to be built. You start "Build this project only" on one project and then wait patiently until it finishes, then you run nother project the same way... Of course you can create build configuration and group such projects there. But sometimes it is just "this and that" and not worth creating a configuration for a single case.

So, at this point having a command "add to the building queue" would be perfect, but there is no such thing in IDE (at least in Visual Studio 7.1 aka 2003). But if we had a script-engine capable of building projects, adding some queueing to it would be pretty simple and convenient.

This is pretty close actually to having own IDE for C++ projects, but that is another big idea. =)

Let me know if you implement some of these approaches. Of course, if I ever do this, I will write about it here. Maybe having a plan and design in form of a blog post will make this come true sooner — we'll see…