JamPlus 3.0 branch

Added by Joshua Jensen over 2 years ago

(This information is deprecated. See the JamPlus 3.0 branch take two topic for the most recent updates.)

Hello.

I have recently pushed a branch called 'jamplus3'. This branch contains changes for the next major evolution of JamPlus, something I've been working on over the past year. In particular, it greatly enhances the multiple toolchain support.

I open this up for discussion. What do you think of the new toolchain support? What other features, breaking, if necessary, would you like to see?

Thanks.

Josh

Extended Lua Support:

Calling IncludeModule luasupport ; loads a small Lua script that makes Lua->Jam interop much nicer.

An example of its usage from a real world working asset build can be found in samples/assetbuild-luasupport/.

New Toolchain Support:

The Jambase.jam file will auto-launch a toolchain called (on Windows) c/win32/release. The user can override the default toolchain by passing the TOOLCHAIN=c/platform/config key/value pair. For instance, to build win64 debug for the C toolchain, you would specify TOOLCHAIN=c/win64/debug on the Jam command-line.

jam TOOLCHAIN=c/win64/debug

Additionally, as a backward compatibility feature, passing PLATFORM=abc and CONFIG=defg will result in a toolchain called c/abc/defg.

jam PLATFORM=abc CONFIG=defg

A new rule, Toolchain TOOLCHAIN_SPEC, allows the current toolchain to be changed. Unlike JamPlus 2.x, the toolchains can be something other than C language stuff. The format of TOOLCHAIN_SPEC (c/win32/release) is slash-separated where the word before the first slash is the toolchain category. Everything the follows is parsed by the category's C._Toolchain rule and is category-dependent. See samples/toolchains-helloworld/Jamfile.jam for usages.

The rule C.IncludeModule MODULE_NAME replaces uses like IncludeModule c/MODULE_NAME.

C.ToolchainSpecKeys is used as before, but it now parses a / separator to assign the same value to multiple keys. For instance, C.ToolchainSpecKeys C.PLATFORM/PLATFORM assigns the value to both C.PLATFORM and PLATFORM.

C.ToolchainHelper is used to load modules from bin/modules/toolchains/c/_helpers. C.ToolchainHelperOnce will load the specified file only once, whereas C.ToolchainHelper will load it at every request.

The majority of the files residing in the bin/modules/c-compilers/ directory have been moved to bin/modules/toolchains/c/'. The @bin/modules/c-compilers/ directory tree has been removed.

The order of toolchain file loads is as follows when Toolchain c/win32/release is called:

  1. The category of toolchain is parsed from the front of the toolchain spec (c/win32/release). In this case, it is c, and bin/modules/toolchains/c/_helpers/c.jam is loaded.
  2. Jam will attempt to load bin/modules/c/win32-release.jam. If that fails, Jam will attempt to load bin/modules/c/win32.jam. If that fails, Jam falls back to loading bin/modules/c/_default_.jam.
  3. Toolchain spec keys are added for C.PLATFORM/PLATFORM and C.CONFIG/CONFIG. For the toolchain c/win32/release, the following assignments are made: C.PLATFORM=win32, PLATFORM=win32, C.CONFIG=release, CONFIG=release
  4. Next, C.ToolchainHelper $(C.PLATFORM) is called. For c/win32/release, this loads bin/modules/c/_helpers/win32.jam.
  5. For the C toolchain, auto detection of the compiler is performed. For c/win32/release, the latest Visual Studio version is detected. Failing that, MinGW is detected. C.COMPILER_SUITE is set to vc for a successful Visual Studio detection.
  6. Next, C.ToolchainHelper $(C.COMPILER_SUITE)-$(C.PLATFORM)-$(C.CONFIG) is called. For c/win32/release with a successful Visual Studio detection, this loads bin/modules/c/_helpers/vc-win32-release.jam which sets up some optimization flags and #defines.

There are no examples at the moment for a toolchain category other than c, but it would be easy to create one. If you wanted a toolchain category called assets, you would create a toolchains/assets/_helpers/assets.jam file and fill in the blanks.

samples/fakeps3/toolchains/c/fakeps3.jam shows an example of handling multiple configurations and architectures within a single file.

Be aware that the Jam variable called C.ACTIVE_TOOLCHAIN has been renamed to C.ACTIVE_TOOLCHAIN_GRIST.

Instead of faking 'derived' classes of compiler hierarchies within Jam (that never worked well before), settings are now added to $(C.COMPILER_SUITE_SYMBOL) with the rule names that should be called. More than one rule name can be attached per setting; the listed rules will executed in order. The names of the settings are as follows: C.ApplicationFromObjects_CleanIntermediates, C.ApplicationFromObjects_LinkFlags, C.ApplicationFromObjects_PostBuild, C.ApplicationFromObjects_Setup, C.C++Exceptions, C.LibraryFromObjects_LibFlags, C.LinkPrebuiltLibraries, C.MultiCppCompile_PreCompile, C.MultiCppCompile_PchDeps, C.MultiCppCompile_PostCompile, C.MultiCppCompile_SetupFlags, C.RuntimeTypeHelper, C.SharedLibraryFromObjects_CleanIntermediates, C.SharedLibraryFromObjects_ExportLib, C.SharedLibraryFromObjects_LinkFlags, C.SharedLibraryFromObjects_PostBuild, C.SharedLibraryFromObjects_RegServer, C.SharedLibraryFromObjects_UnRegServer,

Miscellaneous updates:

  • For jam --workspace more than one IDE type can be specified comma separated to --gen, and all listed IDEs will have project files written.
  • Visual Studio files use backslashes.
  • For iOS, the minimum SDK version is handled correctly.
  • The workspace project hierarchy has changed around for greater clarity in the out of source build directory.
  • All samples and tests have been updated with the new toolchain support.

Replies (5)

RE: JamPlus 3.0 branch - Added by John Brandwood over 2 years ago

It sounds like you've made some great changes!

I've only just been upgrading to jamplus-nextgen, so this will take some time to get used to.

On the plus side, I really like the idea of what you've done with the toolchains ... I've been wanting to move our asset building out of custom batch files, so that will be great ... when I can figure it out.

Moving to jamplus-nextgen, I've been confused by where to put platform-specific settings, such as C.CC and C.C++, so I'll be interested to see what the startup process looks like now.

For instance, in jamplus-nextgen, the processing seems to be ...


incl Jambase.jam
--incl modules/c-autodetect.jam
----incl modules/c.jam
----incl modules/c-compilers/configs/myplatform-myconfig.jam
------incl modules/c-compilers/configs/_myplatform.jam
--------incl modules/c-compilers/c-mycompiler-autodetect.jam
--------incl modules/c-compilers/c-mycompiler.jam
--------incl modules/c-compilers/myplatform-mycompiler.jam
----------incl modules/c-compilers/myplatform.jam
--------incl modules/c-compilers/configs/_mycompiler.jam
------incl modules/c-compilers/configs/_Global_DEBUG.jam
incl Jamfile.jam

Now ... what I don't understand here, is where to set up C.CC and C.C++.

They're set to generic settings for the compiler in "_mycompiler.jam" destroying any settings that I put in "myplatform-mycompiler.jam" ... and the logic of this flow confuses me, I just can't figure out the purpose of "_mycompiler.jam"!

The flow itself is subtly different in the MSVC-based, APPLE-based and LINUX-based toolchains, making things even more interesting.

I look forward to taking a look at jamplus3 over the next few days.

John

RE: JamPlus 3.0 branch - Added by John Brandwood over 2 years ago

I've compiled the latest jamplus3 from today's sources and am have a couple of initial questions/problems.

1) Intellisense is broken on win32/win64/xbox360 builds ... and I can't fix it anymore.

This is because the MSSDK_STDHDRS aren't written out with the Visual Studio workspace, and so intellisense can't find them.

I could fix this on jamplus-nextgen by modifying DumpJamTargetInfo.jam (line 144 in jamplus3) ...
hdrs += $(SUBDIRHDRS) $(STDHDRS) ;
needs to be
hdrs += $(SUBDIRHDRS) $(STDHDRS) $(C.STDHDRS:Z=$(C.COMPILER_SUITE_SYMBOL)) ;

This doesn't work anymore because it looks like the workspace builds shortcut the compiler setup, and things like C.STDHDRS and C.COMPILER_SUITE_SYMBOL seem to never be initialized.

Can you please tell me the thinking behind this change in behavior, and how I can get around it and get the SDK paths added to the workspace again?

2) Which brings me to a question ... what is the thinking behind having both STDHDRS and C.STDHDRS?

clang and gcc both set STDHDRS, and then it's never used anywhere except in DumpJamTargetInfo.jam on line 144.

msvc sets C.STDHDRS, and it's used heavily in vc-shared.jam to do a
SET INCLUDE=$(C.STDHDRS:\\J=;)
before calling the compiler/linker/etc, rather than adding the include directories to the command line.

But at c.jam line 366, in C._MultiCppCompile, there's
collectHdrs += $(postHDRS) $(C.STDHDRS) ;
which seems to sets the directories redundantly ... but only for multi-compile.

Is there some thinking behind these apparent inconsistencies that I'm missing?

The normal msvc setup also defines MSSDK_STDHDRS, but it's global on win32/win64, and local-only on xbox360. Is MSSDK_STDHDRS supposed to be something that I can refer to?

Thanks,

John

RE: JamPlus 3.0 branch - Added by Joshua Jensen over 2 years ago

I have just pushed another update to the jamplus3 branch.

If there is anything to be done, let me know, and I'll get right on it.

Missing rules

Jam now makes a good attempt to automatically find rules that have not been specifically loaded by the script via IncludeModule or include or a similar mechanism. It does this by splitting the rule name into its components (separated by periods) and then searching the disk for a file of the component name. Each component file is loaded and tested for the rule name.

If the script asks for C.Application, Jam loads modules/c.jam and looks for a C.Application rule. It finds the C.Application rule and executes it.

If the script calls CopyFile, Jam will look in modules/copyfile.jam for the CopyFile rule.

If the script wants to execute the rule Assets.Build.Something, the following happens:

  1. modules/assets.jam is loaded. A test is performed for the rule Assets.Build.Something. If it is found, it is executed.
  2. modules/assets/build.jam is loaded. A test is performed for the existence of the rule Assets.Build.Something.
  3. modules/assets/build/something.jam is loaded. A test is performed for the existence of the rule Assets.Build.Something.
  4. Assuming the search made it this far, the rule Assets.Build.Something does not exist. Now, Jam begins looking for a more generic rule called Assets.Build.*.
  5. If that is not found, Assets.* is searched for.
  6. If either is found, $(__MISSING_RULE_COMPONENTS) is filled in with the string list Assets Build Something.

As a result of this new rule searching facility, a number of the rules in the provided C modules have undergone some renames:

  • c/dotnet.jam
    • C.UseDotNet -> C.DotNet
    • C.ReferenceDirectories -> C.DotNet.ReferenceDirectories
    • C.StrongName -> C.DotNet
  • c/mfc.jam
    • C.UseMFC -> C.MFC
  • c/midl.jam
    • C.MidlFlags -> C.Midl.Flags
    • C.MidlCompiler -> C.Midl.Compiler
  • c/mysql.jam
    • C.UseMySQL -> C.MySQL
  • c/openssl.jam
    • C.UseOpenSSL -> C.OpenSSL
  • c/qt.jam
    • C.UseQt -> C.Qt
    • $(QT_FOUND) -> C.Qt.Found
  • c/win32resource.jam -> c/rc.jam
    • C.ResourceCompiler -> C.Rc
    • C.RcFlags -> C.Rc.Flags
    • C.RcDefines -> C.Rc.Defines
    • C.RcIncludeDirectories -> C.Rc.IncludeDirectories
  • c/wxwidgets.jam
    • C.UseWxWidgets -> C.WxWidgets

Miscellaneous

  • With John's help, the Visual Studio Intellisense values are now filled in properly.
  • UseModule has been completely removed. Use IncludeModule instead.
  • IncludeModule now searches $(CWD)/jam for additional modules. The search order is: $(SUBDIR) $(CWD) $(CWD)/jam $(JAM_MODULES_USER_PATH) $(JAM_MODULES_PATH)
  • toolchain= is a synonym for TOOLCHAIN= on the Jam command-line.
  • C.SubDirCcFlags, C.SubDirC++Flags, and C.SubDirHdrs have been removed.

RE: JamPlus 3.0 branch - Added by John Brandwood over 2 years ago

I can confirm that Intellisense is working fine now, thanks.

My win32/win64 builds are now working, but I don't have time right now to port over my other platform compiler settings ... that'll have to wait a few days.

For anyone else building jamplus3 ... please note that you'll need to get the new "jamplus3build" branch of luaplus instead of the "nextgen" branch.

RE: JamPlus 3.0 branch - Added by Joshua Jensen about 2 years ago

Running one of the bootstrap files in the root directory will automatically do the right thing with the LuaPlus submodule.

(1-5/5)