Feature #23

Support building multiple files within a project with cl.exe concurrently

Added by Steven Craft over 9 years ago. Updated over 9 years ago.

Status:NewStart date:07/20/2010
Priority:NormalDue date:
Assignee:-% Done:


Target version:-


I have been trailing Incredibuild, and have noted with their technology it automatically builds a single visual studio project in a multicore manner. Initially when I hooked JamPlus up to Incredibuild I found that while the project did manage to build, it come up with a lot of warnings about pdb files being essentially corrupt. So I emailed Incredibuild support, and then told me to add a magic property to my cl compiler definition which fixed the issue. Since then it seems to building fine.

Based on this, I thought it might be worth while finding out what magic they are performing, and seeing whether it would be something that could be applied to JamPlus, as it does improve build times quite a lot for large projects when multiple cores are available to perform the build.



#1 Updated by Joshua Jensen over 9 years ago

Could it be the /MP option? The problem with that option is that JamPlus is no longer in control of firing off processes anymore.

#2 Updated by Steven Craft over 9 years ago

I'm not sure; my tests with Incredibuild are all done via JamPlus, the only change I made to allow Incredibuild to build multiple files from the same project is to remove the semaphore that normally prevents concurrent compilation within the same project, and then enable their magic option (their system intecepts all file system calls, and probably other calls to, and you tell it which exes to intecept, and provide it with a bunch of options for each exe). For what it is worth, the cl tool is setup as:

<Tool Filename="cl.exe" AllowRemote="true" DeriveCaptionFrom="lastParam" VCCompiler="True" IdentifyTaskOutput="true" />

It is worth noting that with their technology, you can build multiple files, concurrently, within the project, on the same machine.

I've been trying to track down what they do, it seems to involve having a PDB file for each parallel cl.exe instance. I have disabled the semaphore locally for cl.exe, and am building without Incredibuild, but I am unable to recreate any problems at present. Do you know the best way to recreate the problem that lead to the introduction of the semaphore in the first place?


#3 Updated by Arseny Kapoulkine over 9 years ago

The "magic option" involves having a PDB file for each separate agent, which is hard to setup manually. If you want to work with IncrediBuild, the best option I found is to use /Z7 command-line switch instead of /Zi or /ZI. This gets rid of intermediate PDB files completely by embedding debug information inside the object files. The final PDB (generated by linker) is still present, of course, but link is usually local anyway.

You lose Edit-and-Continue with this, but everything else works fine.

Also available in: Atom PDF