JamPlus 3.0 branch take two

Added by Joshua Jensen about 5 years ago


I have recently pushed a branch called 'jamplus3'. This branch contains changes for the next major evolution of JamPlus, something I've been working on over the past year. In particular, it greatly enhances the multiple toolchain support.

I have taken suggestions and some ideas of my own and made the toolchain support even better, I hope.

I open this up for discussion. What do you think of the new toolchain support? What other features, breaking, if necessary, would you like to see?



Missing rules

Jam now makes a good attempt to automatically find rules that have not been specifically loaded by the script via IncludeModule or include or a similar mechanism. It does this by splitting the rule name into its components (separated by periods) and then searching the disk for a file of the component name. Each component file is loaded and tested for the rule name.

If the script asks for C.Application, Jam loads modules/c.jam and looks for a C.Application rule. It finds the C.Application rule and executes it.

If the script calls CopyFile, Jam will look in modules/copyfile.jam for the CopyFile rule.

If the script wants to execute the rule Assets.Build.Something, the following happens:

  1. modules/assets.jam is loaded. A test is performed for the rule Assets.Build.Something. If it is found, it is executed.
  2. modules/assets/build.jam is loaded. A test is performed for the existence of the rule Assets.Build.Something.
  3. modules/assets/build/something.jam is loaded. A test is performed for the existence of the rule Assets.Build.Something.
  4. Assuming the search made it this far, the rule Assets.Build.Something does not exist. Now, Jam begins looking for a more generic rule called Assets.Build.Something.*.
  5. If that is not found, Assets.Build.* is searched for.
  6. If that is not found, Assets.* is searched for.
  7. If any are found, $(__MISSING_RULE_COMPONENTS) is filled in with the string list Assets Build Something. $(__MISSING_RULE_SCAN_LIST) is filled in with the rules that were searched for and can be used for error reporting.

As a result of this new rule searching facility, a number of the rules in the provided C modules have undergone some renames:

  • c/dotnet.jam
    • C.UseDotNet -> C.DotNet
    • C.ReferenceDirectories -> C.DotNet.ReferenceDirectories
    • C.StrongName -> C.DotNet
  • c/mfc.jam
    • C.UseMFC -> C.MFC
  • c/midl.jam
    • C.MidlFlags -> C.Midl.Flags
    • C.MidlCompiler -> C.Midl.Compiler
  • c/mysql.jam
    • C.UseMySQL -> C.MySQL
  • c/openssl.jam
    • C.UseOpenSSL -> C.OpenSSL
  • c/qt.jam
    • C.UseQt -> C.Qt
    • $(QT_FOUND) -> C.Qt.Found
  • c/win32resource.jam -> c/rc.jam
    • C.ResourceCompiler -> C.Rc
    • C.RcFlags -> C.Rc.Flags
    • C.RcDefines -> C.Rc.Defines
    • C.RcIncludeDirectories -> C.Rc.IncludeDirectories
  • c/wxwidgets.jam
    • C.UseWxWidgets -> C.WxWidgets

Extended Lua Support:

Calling IncludeModule luasupport ; loads a small Lua script that makes Lua->Jam interop much nicer.

An example of its usage from a real world working asset build can be found in samples/assetbuild-luasupport/.

New Rule-Based Toolchain Support:

The Jambase.jam file will auto-launch a toolchain called (on Windows) win32/release. (I'd really like to not have the Jambase auto-launch anything in case the user is not using a C-based toolchain, but then it takes away from Jam's ease of use.) The user can override the default toolchain by passing the C.TOOLCHAIN=platform/config key/value pair. For instance, to build win64 debug for the C toolchain, you would specify C.TOOLCHAIN=win64/debug on the Jam command-line.

jam C.TOOLCHAIN=win64/debug

Additionally, as a backward compatibility feature, passing PLATFORM=abc and CONFIG=defg will result in a toolchain called abc/defg.

jam PLATFORM=abc CONFIG=defg

The rule C.Toolchain TOOLCHAIN_SPEC allows the current toolchain to be changed. The format of TOOLCHAIN_SPEC (win32/release) is slash-separated and the format is determined by the underlying Jamfiles that are executed. Additionally, at the end of the TOOLCHAIN_SPEC, you can add key=value options by preceding the option with an \@ sign. See samples/toolchains-helloworld/Jamfile.jam for usages.

C.Toolchain win32/release ;
C.Toolchain win32/=mingw
C.Toolchain ==debug

C.ToolchainSpecKeys is used as before, but it now parses a / separator to assign the same value to multiple keys. For instance, C.ToolchainSpecKeys C.PLATFORM/PLATFORM assigns the value to both C.PLATFORM and PLATFORM.


The majority of the files residing in the bin/modules/c-compilers/ directory have been moved to bin/modules/c/toolchain/'. The @bin/modules/c-compilers/ directory tree has been removed.

The order of toolchain file loads is as follows when Toolchain win32/release is called:

  1. The C.Toolchain rule is requested. The rule is found missing, and Jam loads bin/modules/c.jam to find it.
  2. Jam then attempts to execute a rule called C.Toolchain.win32.release. The MissingRule support kicks in and ends up executing rule C.Toolchain.win32.* from bin/modules/c/toolchain/win32.jam.
  3. Toolchain spec keys are added for C.PLATFORM/PLATFORM and C.CONFIG/CONFIG. For the toolchain win32/release, the following assignments are made: C.PLATFORM=win32, PLATFORM=win32, C.CONFIG=release, CONFIG=release
  4. Within C.Toolchain.win32.*, auto detection of the compiler is performed. First, the latest Visual Studio version is detected. Failing that, TDM/GCC is detected then MinGW. Assuming Visual Studio detection, C.Toolchain.vc.win32.Detect from bin/modules/c/toolchain/vc/win32.jam is called.
  5. Next, C.Toolchain.vc.win32-release is called to set up some optimization flags and #defines.

samples/fakeps3/jam/c/toolchain/fakeps3.jam shows an example of handling multiple configurations and architectures within a single file.

jam C.TOOLCHAIN=fakeps3/debug/ppu

Be aware that the Jam variable called C.ACTIVE_TOOLCHAIN has been renamed to C.ACTIVE_TOOLCHAIN_GRIST.

Instead of faking 'derived' classes of compiler hierarchies within Jam (that never worked well before), settings are now added to $(C.COMPILER_SUITE_SYMBOL) with the rule names that should be called. More than one rule name can be attached per setting; the listed rules will executed in order. The names of the settings are as follows: C.ApplicationFromObjects_CleanIntermediates, C.ApplicationFromObjects_LinkFlags, C.ApplicationFromObjects_PostBuild, C.ApplicationFromObjects_Setup, C.C++Exceptions, C.LibraryFromObjects_LibFlags, C.LinkPrebuiltLibraries, C.MultiCppCompile_PreCompile, C.MultiCppCompile_PchDeps, C.MultiCppCompile_PostCompile, C.MultiCppCompile_SetupFlags, C.RuntimeTypeHelper, C.SharedLibraryFromObjects_CleanIntermediates, C.SharedLibraryFromObjects_ExportLib, C.SharedLibraryFromObjects_LinkFlags, C.SharedLibraryFromObjects_PostBuild, C.SharedLibraryFromObjects_RegServer, C.SharedLibraryFromObjects_UnRegServer.

Miscellaneous updates:

  • For jam --workspace more than one IDE type can be specified comma separated to --gen, and all listed IDEs will have project files written.
  • Visual Studio files use backslashes.
  • For iOS, the minimum SDK version is handled correctly.
  • iOS 8.1 support has been added.
  • The workspace project hierarchy has changed around for greater clarity in the out of source build directory.
  • All samples and tests have been updated with the new toolchain support.
  • UseModule has been completely removed. Use IncludeModule instead.
  • IncludeModule now searches $(CWD)/jam for additional modules. The search order is: $(SUBDIR) $(CWD) $(CWD)/jam $(JAM_MODULES_USER_PATH) $(JAM_MODULES_PATH)
  • toolchain= is a synonym for TOOLCHAIN= on the Jam command-line.
  • C.SubDirCcFlags, C.SubDirC++Flags, and C.SubDirHdrs have been removed.

Replies (3)

RE: JamPlus 3.0 branch take two - Added by Joshua Jensen about 5 years ago

Miscellaneous updates

  • Fix building from within Xcode.
  • Fix building of the UpdateWorkspace project from within Visual Studio.
  • Support `C.Inherits` for IDE project generation.
  • Route Visual Studio 2013 through Windows Kits 8.1 instead of 8.0.
  • Add Visual Studio 2015 compilation support.
  • Account for a situation during Visual Studio IDE generation where a project moves to a different location on the disk but does not change names.

Although they've been pushed for a while, you can use one of the `bootstrap*` files from the root of the JamPlus distribution to do a complete build of JamPlus after running `git submodule update --init`.

RE: JamPlus 3.0 branch take two - Added by Joshua Jensen about 5 years ago

A number of updates have been made to the jamplus3 branch in the past month and are briefly documented below:

  • Add JAM_EXECUTABLE_PATH variable for use within Jamfiles, so the user can launch a second copy of the Jam executable.
actions screenoutput C.ios._WebServer { $(JAM_EXECUTABLE_PATH:C) --webserver $(OUTPUT_PATH) }
  • Add CLEAN.NOOP variable to disable file deletions during the cleaning pass.
  • C module:
    • C.GetLinkTargets did not work due to a typo.
    • Support inheriting from C.Library and non-C.Library 'libraries' when using C.Inherits.
    • Enhance `C.Toolchain` with the ability to override toolspec options. For example, the following will override `C.ARCHITECTURE` to equal `armv7s`. Previously, it would have ignored the `armv7s`.

      C.Toolchain ipad/==armv7s

    • Visual C++:
      • For Visual C++, keep .exp/.ilk/.lib/.map/.pdb files during the clean up phase.
      • Use VS2013 64-bit executables if they exist.
    • iOS:
      • Rename the platforms in the following ways:
        • ipad -> ios
        • ipadsimulator -> iossimulator
        • iphone -> ios
        • iphonesimulator -> iossimulator
      • Default the iOS UIDeviceFamily info.plist entry to both iPhone and iPad. Override by calling C.ios.BundleInfo TARGET : uidevicefamily : PLATFORMS where PLATFORMS is an array containing ipad and/or iphone.
      • iOS Build Environment for Windows:
        • Remove the executable extension for iOS apps when building on Windows.
        • Assign settings to C.ACTIVE_TOOLCHAIN_* instead of C.ACTIVE_TOOLCHAIN_TARGET
        • Proper bundle generation.
        • Support iOS Build Environment certificate chains. Pass the certificate chain as the third parameter to C.CodeSign. Example: C.CodeSign : "iPhone Developer" : "ios_development.cer:AppleIncRootCertificate.cer:AppleWWDRCA.cer:my_key.key:" ;
      • Default iOS architectures to armv7 instead of armv7s.
      • Set CFBundleVersion on the active toolchain target instead of setting it globally and affecting other toolchain builds.
      • Add C.ProvisionFile rule. C.Provision attempted to detect whether PROFILE_ID was a profile identifier or whether it was a file. The detection was dodgy in some circumstances, so @C.ProvisionFile always expects a file, no detection needed.
      • Fix the dependecies for bundle generation.
      • Add C.Inherits support for frameworks for iOS and Mac builds.
      • Add a locally hosted on-demand web server, accessible during a build's C.ios.WebServer rule.
      • Refactor macosx-shared.jam into ios-shared.jam. Rename a number of rules.
        • C.FrameworkDirectories -> C.ios.FrameworkDirectories
        • C.LinkFrameworks -> C.ios.LinkFrameworks
        • C.WeakLinkFrameworks -> C.ios.WeakLinkFrameworks
        • C.MacOSX_SDK -> C.ios.SDK
        • C.MinimumOSVersion -> C.ios.MinimumOSVersion
        • C.BundleInfo -> C.ios.BundleInfo
        • C.SetTaskAllow -> C.ios.SetTaskAllow
        • C.CodeSign -> C.ios.CodeSign
        • XIB -> ios.XIB
        • C.ProvisionFile -> C.ios.ProvisionFile
        • C.Provision -> C.ios.Provision
        • C.BundleArchive -> C.ios.Archive
      • Add C.ios.Lipo TARGET : LINK_TARGETS : OUTPUT_PATH rule to combine multiple builds into a single output executable.
      • Add C.ios.Bundle TARGET : TOOLCHAINS rule to separately create a bundle with a lipo'ed executable given a list of TOOLCHAINS. Bundles can still be created by passing bundle to the C.Application rule, but those bundles can only contain a single architecture executable. An example is in samples/ios/simple-multiple-architectures.
      • Renamed C.ios.BundleArchive to C.ios.Archive TARGET : OUTPUT_PATH.
      • Add C.ios.WebServer rule. Examples are in samples/ios/simple and samples/ios/simple-multiple-architectures. When used, an HTTPS web server will be launched to locally serve your .ipa to your device. Local certificates are generated.
    • Mac OS X:
      • Refactor macosx-shared.jam.
      • C.FrameworkDirectories -> C.macosx.FrameworkDirectories
      • C.LinkFrameworks -> C.macosx.LinkFrameworks
      • C.WeakLinkFrameworks -> C.macosx.WeakLinkFrameworks
      • C.BundleInfo -> C.macosx.BundleInfo
      • C.SetTaskAllow -> C.macosx.SetTaskAllow
  • Jambase:
    • Provide a reasonable default in SubDir. If the user has not specified ALL_LOCATE_TARGET, then a default ALL_LOCATE_TARGET is filled in using $(CWD)/.build/ as its root. In order for this to not take effect, merely assign ALL_LOCATE_TARGET to a valid path before the first SubDir rule is called.
    • Allow SubInclude to use the default SubDir path if none is specified.

RE: JamPlus 3.0 branch take two - Added by Joshua Jensen almost 5 years ago

Another month brings more updates to the jamplus3 branch. As of now, I am calling this branch suitable for production.

If anyone needs them, I have Jamfiles for SDL, Cocos2D-x, and Moai before the CMakeLists reorg.

  • Handle empty directory deletion in clean_unused_files(). Previously, directories would only be cleaned up if there was a file present requiring clean up, too. Now, any empty directories that should not exist in the CLEAN chain will be removed.
  • Bootstrapping:
    • Add various bootstrapping scripts in the root JamPlus directory for building 32-bit or 64-bit executables with the Lua dependencies for Windows, Mac, or Linux.
  • C module:
    • Add C.ForcePublic rule for targets that have no public includes or libraries
    • Add rule C.ConfigureFile. This rule handles files generated from *.in files.
    • iOS:
      • iOS Build Environment for Windows:
        • Delete the lipo generated files before running the iOS lipo on Windows
        • Fix some code signature timestamp issues.
    • Linux:
      • Fix the linux32 and linux64 toolchains.
    • Mac OS X:
      • Make macosx behave like iOS builds when bundling
  • Jambase:
    • Disallow creation of a .build directory in rule SubDir when one already exists via JamToWorkspace
  • JamToWorkspace:
    • Fix a bunch of C.Inherits issues that were preventing projects from showing in the workspace or crashing jam --workspace
    • Change the output location of the intermediates to be PLATFORM/_intermediates_
    • Fix Xcode generation of projects when using C.ios.Bundle
    • Remove the ugly 'clean' targets in Xcode