Makefile conversion project

Added by Chris Chapman almost 11 years ago

Okay then, I've been through the Jambase from top to bottom and side to side, time to ask for some help.

I'm converting over our old makefile build system into JamPlus, with a view to replacing it altogether (and putting it to work on our asset build process, but that's for another day).

So our setup is fairly clean. It consists of a multiple libraries in rough groups, each built with their own makefile, and tied together in a tree by some lua scripts. E.g.:

core/core
core/Input
core/Renderer
game/game

Where core & game are groups, and core, Input and Renderer are individual libraries. They all follow a similar pattern, so if I can crack the core library I can get them all working. In all the following examples, assume we're building on wii, for the debug configuration.

Libraries have a /include folder for all of their external headers, and a /src folder for all the .cpp files. The source folder is organised hierarchically into directories, with .cpp files at all levels. E.g.:

core/src
+ core/src/AABB.cpp
+ core/src/Vector.cpp
core/src/integral
+ core/src/integral/Vector.cpp
core/src/wii
+ core/src/wii/Timer.cpp
core/src/win32
+ core/src/win32/Timer.cpp

For each library, we have a list of source directories we want to build, and we want to compile all .cpp/.c files within those directories. So we would be building src, src/integral and src/wii. So if we're adding files to the project, they will be pulled into the library automatically if they are in one of the existing directories, new directories obviously need manually added to the makefile. To counter this, we have a list of excluded source files which won't be built/depended on.

The library file we want to build into core/lib/wii/debug/libCore.a.

Intermediate object files are built into the core/obj/wii/debug folder, and crucially using a subdirectory structure matching the source tree. The subdirectories approach is key because otherwise core/src/Vector.cpp would generate obj/wii/debug/Vector.o, and then core/src/integral/Vector.cpp would also generate the same .o file.

With makefiles this is pretty straightforward - we generate a list of source files from our list of source directories. Then we do a pattern replace in that list and swap src/.cpp* for obj/wii/debug/%.o - for example obj/wii/debug/integral/Vector.o from src/integral/Vector.cpp. The dependency rules are written in such a way that obj/%.o depends on src/%.cpp, and so the % pattern is the relative subdirectory plus file root (e.g. integral/Vector) which we can easily plug into our compile rules to generate the .o file correctly.

I've added parts to the Jambase to support the Wii (Codewarrior based) and PSP (GCC based) platforms/compilers, basically mirroring the MinGW compiler usage and setting it up so that the compiler is selected automatically when compiling for those platforms. Otherwise it's pretty untouched. Also I've got a (relatively) simple jamfile for the core library which sets up include directories, and uses GLOB to get the list of files to build. I've attached it for reference, although it's currently pared down to build just two files for testing.

So anyway, my current issues:
1) The biggy: MultiCppCompile deliberately flattens out the object files such that src/Vector and src/integral/Vector will produce the same .o file. This is implicit all through the function I think, there are several places where $(sources:S=$(SUFOBJ):D=) or similar is used to infer the .o target from the input source. Obviously the :D= is stripping off the source path, so any input path information is lost. It's not as simple as removing that construct though, as it introduces other issues and doesn't help matters. There is an _OutputAsTree setting near the top, but I'm guessing it's a relic because it's not referenced anywhere else. There is some commented out stuff in there, but I don't think its going to fix my problems.

Anyway, the behaviour I see is that each input .cpp file gets passed individually to batched_C++, except for files which occur in more than one source directory. Those are grouped together (presumably because they are grouped by flags, and both files generate equivalent -o \"$(_obj:T)\ sections, while all other source files have unique object files).

I've tried a bunch of things involving tinkering with the _obj creation, none of which have really worked properly, because essentially I'm coding in the dark as I don't quite get how the existing logic is working.

2) Closely related to 1: The two new compilers handle it differently - one complains that you're not allowed to specify multiple input files when using -o, the other uses the passed in _obj file for the first source file, but then compiles the second file and drops the object for it into the /current directory/. So clearly the multiple source files in a single compile command is going to have issues with these compilers. That said, I wouldn't be fussed about the speed loss if I could get correctness, so one source/object file per compiler call would be fine for me. Would I be right in saying the the Objects rule has been removed in favour of the MultiCppCompile rule?

3) Simpler problem: I want the .o files in one folder, but the .a file in another. I can get this to happen by specifying ./lib/wii/debug/libCore everywhere in the Jamfile I currently specify just libCore. I'm sure there must be a better/nicer way of doing this.

All that said - I'm very impressed with the speed and flexibility so far. The makefiles we use aren't really extensible enough to cope with our asset building as well, and they're about at the limit of make's functionality as is.

Jamfile.jam - Simple Jamfile for the core library (566 Bytes)


Replies (13)

RE: Makefile conversion project - Added by Joshua Jensen almost 11 years ago

I have checked in a sample in source:tests/compile_outputastree/ which illustrates how you can get your files building into the obj/ and lib/ directories, as you specified above. I think it behaves according to your description. Let me know if it doesn't, and we'll talk about it some more.

In a nutshell, I have restored MultiCppCompile's OutputAsTree support. It can be turned on via:

CppCompileOptions outputastree ;

You'll find reference to that in source:tests/compile_outputastree/Jamfile.jam. Also in that file is a helper rule to replace SubDir in your Jamfiles:

rule ProjectSubDir TOKENS : SUBNAME : PROJECT_NAME
{
    SubDir $(<) : $(>) ;
    LOCATE_TARGET = $(SUBDIR)/obj/$(PLATFORM)/$(CONFIG) ;
    OutputPath $(PROJECT_NAME) : $(SUBDIR)/lib/$(PLATFORM)/$(CONFIG) ;
    OutputPostfix $(PROJECT_NAME) : ;
}

LOCATE_TARGET is set similar to yours. OutputPath puts the target into the correct directory. OutputPostfix removes the default .$(CONFIG) postfix.

I think this fully solves #1 and #3. Let me know if this solves #2, also.

(I'll be releasing the JamPlus 0.2 RC3 after I get your feedback and when I have most of the documentation done. I'm getting close on the documentation. In the meantime, I have attached a Jam.exe to unzip over a JamPlus 0.2 RC2 binary build.)

Thanks for all the information and your willingness to report issues in JamPlus.

Josh

jam-081112-exe.zip - Grab the 0.2 RC2 binaries and unzip this on top and run with latest Git pull (135 KB)

RE: Makefile conversion project - Added by Joshua Jensen almost 11 years ago

My Internet connection died before I could answer the rest of #2.

It turns out that the MinGW code in the Jambase has a side effect that automatically causes none of the files to be batched to the compiler.

else if $(COMPILER) = mingw
{
    MFLAGS on $(_src) = $(_flags) "-c -o \"$(_obj:T)\"" ;
    UseCommandLine $(_obj) : $(_flags) ;
}

It works due to the $(_obj:T) in MFLAGS. When specified, it makes every set of flags different from every other set of flags, since every _obj name is different. In that case, the batching code will split every file into a unique compiler call.

To be clear, the old Perforce Jambase should continue to work without issue, although some users may find a need to set PATHDELIM_OLDSTYLE = 1. JamPlus internally formats all paths to use forward slashes. If you need the Objects rule, you can resurrect it from there.

For JamPlus, though, the Objects rule doesn't handle batching of files to the compiler, nor can it, really. It was removed. Anything that was removed is up for debate to bring back, although I would want to make sure it integrated into the JamToWorkspace facility.

Josh

RE: Makefile conversion project - Added by Chris Chapman almost 11 years ago

Bam - that certainly goes much further. The new ProjectSubDir rule takes care of dropping the output in the appropriate lib folder, and the outputastree functionality tackles the hierarchical source files issue. Mostly. ;-) Once I'd updated with your patches, the output files do get sorted out by sub-directory, and I've now generated some nice clean Jamfiles for each project (core, Input, Renderer, etc.), and on their own these all build nicely.

I think the mostly issue might be down to how I'm using the jam stuff though, so here's some more context. Let's pull back a bit to see my project hierarchy in full. We have:

/art
/software
/software/core
/software/core/core
/software/core/Renderer
...etc...
/software/game
/software/pong
/software/solitaire

And yes, pong and solitaire are our test titles. So pong depends on game, which depends on core/core, core/Input, etc; your basic layers of engine.

I've dropped a jamfile in each of the folders (including the root, because some day I'd like to include /art in the jam building tree). Previously we'd combined a little lua script with the makefiles such that if you did lua build.lua in any of the folders, all of the children would be built too. In Jam this looks like it's easily built in by having Jamfiles in the root folders like:

SubDir TOP software core ;

SubInclude TOP software core core ;
SubInclude TOP software core Audio ;

So if I'm in /software/core/core and I run jam -s PLATFORM=wii, the core library will be built. If I'm in /software/core and run jam -s PLATFORM=wii, then the core, Input, Renderer, etc. libraries will all be built. Which is great, just what I want. Also, if I'm in /software/core and run jam -s PLATFORM=wii libCore, then I can select which library I want to build. All good so far.

The glitch comes in the output directory handling. If I'm in /software/core/core, the list of source files looks like:

./src/Vector.cpp
./src/integral/Vector.cpp

which will generate object files as follows:
./obj/wii/debug/src/Vector.cpp
./obj/wii/debug/src/integral/Vector.cpp

If I'm in /software/core and I do the build, the source files are
./core/src/Vector.cpp
./core/src/integral/Vector.cpp

which will generate object files as follows:
./obj/wii/debug/core/src/Vector.cpp
./obj/wii/debug/core/src/integral/Vector.cpp

So basically running jam from a different folder in the hierarchy will likely cause a full rebuild.

So that's the problem, but I'm not sure it's not just a side effect of how I'm setting up the hierarchy. Because, what I've also got are the game projects (e.g. pong). I'd like to be able to build just pong when I'm running within the pong folder. I've a Jamfile for pong that looks like:

ProjectSubDir TOP software pong : : pong ;

local SOURCE_DIRECTORIES = 
    ./source
    ./source/$(PLATFORM) 
    ;

local COREDIR = ../core ;
local GAMEDIR = ../game ;
local INCLUDE_DIRECTORIES = 
    ./include
    ./source
    $(GAMEDIR)/include
    $(COREDIR)/lua-5.1.1/include
    $(COREDIR)/movie/include
    $(COREDIR)/font/include
    $(COREDIR)/Renderer/include
    $(COREDIR)/Audio/include
    $(COREDIR)/Input/include
    $(COREDIR)/core/include
    ;

SOURCE_EXTENSIONS ?= *.cpp *.c ;

local SOURCE_FILES = [ GLOB $(SUBDIR)/$(SOURCE_DIRECTORIES) : $(SOURCE_EXTENSIONS) ] ;

LinkLibraries pong : 
    libGame 
    libLua libFreeType libZLib 
    libOgg libTheora libVorbis 
    libFont libRenderer libInput libAudio libMovie libCore
    ;

IncludeDirectories pong : $(INCLUDE_DIRECTORIES) ;
CppCompileOptions outputastree ;

Application pong : $(SOURCE_FILES) ;

Again - this works as advertised, but only if I run from the /software folder. If I run from the /software/pong folder, the jam file knows nothing about the core or game libraries, and complains about not knowing how to make libCore et al.

So I've lost a little bit of functionality, but the speed and efficiency gains are still pretty good, and I can rejig our working build process to always go to the correct folder and selectively choose targets (e.g. run cd /software && jam pong). I want to do a bit of work on the setup too - it doesn't feel so nice to have wedged in the proprietary compiled stuff in the Jambase file - if possible I'd like to keep that clean and have it pull in the compiler specific information and local rules (e.g. the ProjectSubDir rule you provided is obviously a local thing to our setup, but to get clean Jamfiles I've had to put it into the Jambase). That way when we get to a clean working solution I can share the local setup files on the appropriate newsgroups where people who are properly NDAed can just download them and the latest JamPlus release and everything slots into place without hand editing files.

RE: Makefile conversion project - Added by Joshua Jensen almost 11 years ago

Some quick thoughts, as I'm on my way out right now, and I'll come back with some more detailed ones, if necessary, later.

You can name TOP anything you want. All the examples say TOP, but I don't find it very descriptive.

The ProjectSubDir rule should exist in a Jamrules.jam in the directory immediately above software/. Then you don't have to modify the Jambase. You should also put the CppCompileOptions line in there.

I would recommend SubIncluding the Jamfiles you need in pong/ and solitaire/. They can then exist in a standalone fashion, as you desire. You could put them in the Jamrules, too, if you wanted, although I would recommend more of the approach done by the JamPlus PopCap example.

In fact, I'd be willing to bet that, like the PopCap example, you could write a single rule to represent your game applications and embed it in the Jamrules:

rule GameApplication TARGET
{
    SubInclude TOP software core core ;
    SubInclude TOP software core Audio ;

    local COREDIR = ../core ;
    local GAMEDIR = ../game ;
    local INCLUDE_DIRECTORIES = 
        ./include
        ./source
        $(GAMEDIR)/include
        $(COREDIR)/lua-5.1.1/include
        $(COREDIR)/movie/include
        $(COREDIR)/font/include
        $(COREDIR)/Renderer/include
        $(COREDIR)/Audio/include
        $(COREDIR)/Input/include
        $(COREDIR)/core/include
    ;

    SOURCE_EXTENSIONS ?= *.cpp *.c ;

    local SOURCE_FILES = [ GLOB $(SUBDIR)/$(SOURCE_DIRECTORIES) : $(SOURCE_EXTENSIONS) ] ;

    LinkLibraries $(TARGET) : 
        libGame 
        libLua libFreeType libZLib 
        libOgg libTheora libVorbis 
        libFont libRenderer libInput libAudio libMovie libCore
    ;

    IncludeDirectories $(TARGET) : $(INCLUDE_DIRECTORIES) ;

    Application $(TARGET) : $(SOURCE_FILES) ;
}

And then in software/pong/:

GameApplication pong ;

One final recommendation would be to use the SubInclude facility to name your Jamfiles:

SubInclude TOP software core core : core ;          # Expects to load a core.jam, instead of Jamfile.jam.
SubInclude TOP software core Audio : Audio ;        # Expects to load an Audio.jam, instead of Jamfile.jam.

In my later response, I'd love to talk about how to split out the Jambase into separate platforms and configurations. I had a way of doing this before, but it wasn't clean. I'd like to find a way to clean it up and make it easy to drop in new compilers/platforms/configs.

Josh

RE: Makefile conversion project - Added by Chris Chapman almost 11 years ago

Man, such a long reply to type up I missed your second post.

Yes I did think that there was no way the batching would work with that code as it was, but I assumed that there was some nuance to the batching code I hadn't spotted. I'm guessing that the batching functionality is fairly compiler specific, as it will depend on the ability of the compiler to specify output object files sensibly for a given batch of input files. If you can do multiple <source>/<object> style pairings on the command line then it's pretty straightforward, otherwise it will need to fall back on a less efficient solution.

I've been concentrating on getting the console compiles up and running (because we do win32 compiles through manually built DevStudio solutions), but when I get a chance I'll make sure that the win32 compiles can operate with the same set of Jamfiles.

Cheers,
C

RE: Makefile conversion project - Added by Chris Chapman almost 11 years ago

Just a quick note to say things are progressing, but I've had to put it to one side for now. Definitely feels like the big stumbling blocks are gone now, and it's just a case of refining what we've done. Got the Win32 compile working (had to tweak Jambase to detect VS Express versions, I think that can be rolled back into the main version). Things on my list to do:

1) Find some way to re-root the object file generate to a specific folder such that the output-as-tree functionality generates the same set of object files regardless of where the jam build was initiated from
2) Re-rig our recursive structure and split out into Jamfile.jam (which builds a project and all it's children) and <project>.jam which does the local project only. That will give us what we need with respect to local building and being able to build the entire tree.
3) Put SubIncludes into the application projects so that they build their dependencies, even if built locally. I'm a bit worried what will happen when the same project gets SubIncluded twice if building from the base of the tree, but I'll have to experiment and see.

Sadly it doesn't work to put the ProjectSubDir rule in the Jamrules file, because the Jamrules file doesn't get pulled in unless you use SubDir, but we replaced SubDir with ProjectSubDir, which can't be found because it's not read the Jamrules in yet. Catch-22. That said, what was really missing was the bit of knowledge that ProjectSubDir encapsulated, which was that OutputPath was the way to control the output location, not LOCATE_TARGET. So I've rolled that logic into my LibraryProject and ApplicationProject rules, and have gone back to using regular SubDir.

RE: Makefile conversion project - Added by Joshua Jensen almost 11 years ago

I think I have a patch for the VS Express stuff coming from someone. I have VS2008 Professional, so I'm not sure what the Express version has going.

Let's talk about the things you've mentioned above, and then I'll offer a bit of alternate advice.

1) I'm not sure I understand what you are talking about here. Could you elaborate?

2) Is the goal to change the working directory to, say, core/Renderer/ and be able to just run 'jam' without any arguments, as opposed to running 'jam Renderer'?

3) SubInclude will only include a given Jamfile once.

Ack. That's true about the ProjectSubDir rule. I had the example one sitting in Jamfile.jam, and I always ran from the root. I didn't get bit by the SubDir rule. :(

Okay, here's my advice. JamToWorkspace can take your Jam build tree and output a completely out-of-source build tree. Instead of putting your obj/ and lib/ directories in the middle of your source tree, those can be redirected completely out of source. Other things are set up during the JamToWorkspace process, too, such as the header cache for build speed. When you don't want the build tree anymore, you just destroy it. It isn't necessary to search for stray files in the source tree.

It is possible in an out of source build to have more than one build tree, too, with different defines. When we made Pathstorm, we had an out of source build tree for each game portal Pathstorm was to be published on, since each had different requirements and often we would have to enable certain #defines to meet the requirements. It's very useful.

All said, I'm here to help you get your build system going the way you would like it, hopefully evidenced by the conversation above.

Let me know.

Josh

RE: Makefile conversion project - Added by Chris Chapman almost 11 years ago

Quadruple post combo! Sorry.

Express versions are free, that's about all they have going for them to be honest. But in 3 years of developing the only thing I've missed about the Professional editions are macros, and I get them from Notepad++. But that's by-the-by.

1) Currently if I build the core library from say /software, then change to /software/core and build again, it will rebuild everything (because it can't find the object files built previously - I detail this further in the start of my second post in this thread). Not a show-stopper, but irritating. Basically the object folder structure generated by outputastree needs to be rooted at $(SUBDIR), and right now it's rooted at the current folder.

2) Exactly. My first instinct was to name files core.jam, Input.jam, etc., but if you run jam with no arguments it's still looking for Jamfile. I wouldn't mind jam core, but the best syntax I can get is jam -s JAMFILE=core.jam which to me is too much typing, given that we'll already have to append -s PLATFORM=wii and -s CONFIG=release.

3) Great

All that said though, because of 3, we don't get much from naming the sub-folder Jamfiles explicitly. So I've stuck with Jamfile.jam in each sub-folder now. For the game titles, which need to reach back up the hierarchy for libraries, we SubInclude using TOP. So regardless of whether we build from the root
or the title folder, everything gets pulled in as needed, which is just what we wanted.

I'm almost convinced by the out-of-source build tree; that's specifically why each library has a lib and an obj folder so as not to pollute the source code folders with intermediates. That said, I'd like to stick as close to the structure we've got just now, as I don't think any of the integration issues we're working through here will be helped too much by it.

So aside from the annoyance of 1), we're almost done converting our code-base over, with just tool building left. Next I'd like to get our asset building process working through Jam, but that's another thread I think!

RE: Makefile conversion project - Added by Joshua Jensen almost 11 years ago

1) I'm a little confused. In the ProjectSubDir rule posted above, LOCATE_TARGET reads:

LOCATE_TARGET = $(SUBDIR)/obj/$(PLATFORM)/$(CONFIG) ;

That appears rooted at $(SUBDIR). I just updated the source:tests/compile_outputastree/ code (in Git) to better test what you're talking about. It appears to work according to your description. If you could modify it to show the broken case, I'm sure we can quickly fix it.

2) What comes immediately to mind here is this (untested... I should throw together a sample for this also): Keep the main body of the Jam information in Core.jam. Make a Jamfile.jam. Add to it:

include Core.jam ;
Depends all : Core ;

Then, when you run Jam by itself in the Core directory, only Core will build.

Josh

RE: Makefile conversion project - Added by Chris Chapman almost 11 years ago

A-ha! You're right, it did work as it was, and it's the right shape for our projects. But I've tracked where things goes awry. If the liba/Jamfile.jam file looks like:

SubDir TOP liba : : liba ;
ProjectSubDir liba ;

SRCS =
        $(SUBDIR)/treea/treeb/deepfile.cpp
        $(TOP)/outer/outer.cpp
;

Library liba : $(SRCS) ;

Then you'll see the issue. Basically if you specify your source files relative to the Jamfile, the paths are generated correctly relative. But if you try to use the $(SUBDIR) or $(TOP) variables, the output object files end up in different places depending on where you started the build from.

What I can't tell from this is why the compile_outputastree example works. I switched to the habit of specifying my sources explicitly with $(SUBDIR)/ as a prefix because otherwise my sources weren't getting found when I compiled from other folders. Perhaps this is because I'm using globbing to gather up source files. Here's a snippet of my current setup which may illuminate.

Jamrules:

rule LibraryProjectFromSources TARGET : SOURCE_FILES : INCLUDE_DIRECTORIES
{
    IncludeDirectories $(TARGET) : $(INCLUDE_DIRECTORIES) ;

    LOCATE_TARGET = $(SUBDIR)/obj/$(PLATFORM)/$(CONFIG) ;
    OutputPath $(TARGET) : $(SUBDIR)/lib/$(PLATFORM)/$(CONFIG) ;
    OutputPostfix $(TARGET) : ;

    Library $(TARGET) : $(SOURCE_FILES) ;
}

rule LibraryProject TARGET : SOURCE_DIRECTORIES : INCLUDE_DIRECTORIES : SOURCE_EXTENSIONS
{
    SOURCE_EXTENSIONS ?= *.cpp *.c ;

    local SOURCE_FILES = [ GLOB $(SUBDIR)/$(SOURCE_DIRECTORIES) : $(SOURCE_EXTENSIONS) ] ;

    LibraryProjectFromSources $(TARGET) : $(SOURCE_FILES) : $(INCLUDE_DIRECTORIES) ;
}

core Jamfile:

SubDir TOP software core core ;

local SOURCE_DIRECTORIES = 
    ./src
    ./src/Loading 
    ./src/Saving 
    ./src/memory 
    ./src/integral 
    ./src/$(PLATFORM) 
    ./src/$(PLATFORM)/Loading 
    ./src/$(PLATFORM)/Saving
    ;

local INCLUDE_DIRECTORIES = 
    ./include
    ;

LibraryProject core : $(SOURCE_DIRECTORIES) : $(INCLUDE_DIRECTORIES) ;

Specifically the

local SOURCE_FILES = [ GLOB $(SUBDIR)/$(SOURCE_DIRECTORIES) : $(SOURCE_EXTENSIONS) ] ;
line. If I omit the $(SUBDIR)/, the globbing takes place relative to the current folder; so works when I'm in the library's directory, but doesn't work from a parent folder.

RE: Makefile conversion project - Added by Joshua Jensen almost 11 years ago

Okay, adding $(SUBDIR)/ in front of treea/treeb/deepfile.cpp causes the output file to be weird. I have altered bin/Jambase.jam to change a full path's drive letter, such as c:/dir/, to the drive letter followed by two dashes: c--/dir/.

However, you don't really want to use full paths like you are doing. There are a number of reasons for that. The output directory is one of them. The network cache won't work with full paths like that. The gristed target name is also long and machine specific. There are others, too, if I thought a bit harder.

If you grab latest in the Git repository, you'll notice I have fleshed out the compile_outputastree example even more. Your specific case is illustrated in libc/Jamfile.jam and libb/Jamfile.jam. Note that I remove $(SUBDIR)/ from the front of the globbed files. That gives us a relative path.

The compile_outputastree Jamfiles also handle building from any directory with a Jamfile in it. Output files are put in the right location.

Josh

RE: Makefile conversion project - Added by Chris Chapman over 10 years ago

The latest Jambase does indeed fix the problem where the output/intermediate files were ending up in different places depending on where jam was run from, which tidies up the last real issue I'd had. Thanks!

I've also added the SOURCE_FILES = [ Match $(SUBDIR)/(.*) : $(SOURCE_FILES) ] ; line to my Jamrules in the few places where I glob files, so that the source file lists are stripped down to a minimal form. Although with the updated Jambase that wasn't strictly necessary to get things working, but I'm sure it is better to have a minimal rather than a full path being used.

Next task: art asset building! :-)

RE: Makefile conversion project - Added by Joshua Jensen over 10 years ago

Let us know how the art asset builds go. I have done this kind of work before, but it is hard to post a generalized solution when everyone's pipelines are different.

(1-13/13)