How to generate source files?

Added by Andreas Fredriksson about 11 years ago

I'm trying to get a little source file generator working but looking at the Jambase it seems the rules for user objects from the vanilla Jambase are gone.

AFAICT the MultiCppCompile rule (which is used for Application, Library, etc) just assumes its arguments are either c/c++ sources or objects so there doesn't seem to be a way to add custom actions for file extensions anywhere without hacking the Jambase.

If possible it would be desirable for MultiCppCompile to filter its source list and run the "old" UserObject path for those files it cannot directly compile so that stuff like yacc, flex and other source file generators can work with Jamplus.

Replies (6)

RE: How to generate source files? - Added by Joshua Jensen about 11 years ago

Your request is interesting, as I was never fond of the old UserObject mechanism in the original Jambase. I found it inflexible and unruly to work with. A better version existed in the Crystal Space Jam build system (under the GPL license, I believe).

Still, perhaps it is the right thing to do. In the majority of cases, behavior could be properly applied. Instead of running the ResourceCompiler rule explicitly (my preferred route), a UserObject-like system could do it automatically.

I recently added a rule for handling .def files. I suppose the .def could be automatically handled, too. There is nothing special there.

    TARGET = [ _retrieveActiveTargetName $(TARGET) ] ;
    local grist = $(TARGET) ;
    SOURCES = $(SOURCES:G=$(grist)) ;
    Depends $(TARGET) : $(SOURCES) ;
    LinkFlags $(TARGET) : /DEF:"$(SOURCES:T)" ;

DefFile MyProject : MyProject.def ;

Let me see what I can come up with.


RE: How to generate source files? - Added by Andreas Fredriksson about 11 years ago

Hi Joshua,
I certainly see how explicitly attaching results to some top-level target could work for a few designated files such as resource files and maybe the odd Bison file where they are easily distinguishable from other basic source files.

In the case I'm battling though there are 196 files to be run through a custom code generator that produces both headers and source files. Having to specify all those files outside the usual source glob mechanism will be painful, to say the least. The code base is massive (over 10,000 source files) and I would like to keep the number of special cases down to a minimum.

There might be some middle of the road options here as well where a separate glob for files to generate could be done and then passed to some custom rule that converts all of them and somehow slips the generated source names into the list for the big MultiCppCompile invocation.

RE: How to generate source files? - Added by Joshua Jensen about 11 years ago

I should put together an example of this. The best I have right now is source:tests/generatedheader/simple and it doesn't show what you need.

Here's a quick example off the top of my head:

rule ProcessGenFiles PARENT
    # Scan for all *.gen files, returning them without $(SUBDIR) prepended.
    local sources = [ Glob $(SUBDIR) : *.gen : 0 ] ;
    sources = $(sources:G=$(PARENT)) ;
    SearchSource $(sources) : $(SUBDIR) ;

    # .gen files generate .cpp/.h files.  Put them in LOCATE_TARGET, so they
    # don't clutter the source tree.
    local targets = $(sources:S=.cpp) $(sources:S=.h) ;
    MakeLocate $(targets) : $(LOCATE_TARGET) ;

    # Dependency: PARENTS -> source.cpp source.h -> source.gen
    Depends $(PARENTS) : $(targets) ;
    Depends $(targets) : $(sources) ;

    # Of course, we want to clean up what we generate.
    Clean clean:$(PARENT) : $(targets) ;

    # ProcessGenToCppH in the action only converts one .gen to .cpp/.h at
    # a time.
    local genFile ;
    for genFile in $(sources)
        GenerateCppHFiles $(genFile:S=.cpp) $(genFile:S=.h) : $(genFile) ;

    # Return the list of .cpp and .h files we are making so they can
    # be inserted into the list of files we're going to build.
    return $(targets) ;

actions GenerateCppHFiles
    ProcessGenToCppH $(2) $(1)

# Later...
local SRCS =
        [ ProcessGenFiles CurrentProject ]

Application CurrentProject : $(SRCS) ;

Does this accomplish what you need, or is there more to it? If we can establish a good understanding of the problem, we can look into how we would build a UserObject-like rule.


RE: How to generate source files? - Added by Andreas Fredriksson about 11 years ago

Thanks Josua, this does indeed cover the case I was getting at. I just tried it (modified slightly) with a mock rule that generated dummy cpp and header files and they were generated and then batched together with the vanilla source files just as desired.

I will try to use this method for our generators and see how it works out.

I guess it would make sense to have Application, Library and so on filter their source lists through some UserObject factory to gather generated sources before passing them on to MultiCppCompile in this manner, but I don't know enough Jam to attempt to write that yet :)

RE: How to generate source files? - Added by Andreas Fredriksson about 11 years ago

I've run into an interesting snag in using these generated source files and headers. Both source files and headers are generated, and I can tell from Jam's dependency graph that it knows how to make all the files.

2 Name: Engine.Core
: Depends on <Engine.Core>engine/core/coresettings.cpp (update) (time:1231960628)
: Depends on <Engine.Core>engine/core/coresettings.h (update) (time:1231960628)
3 Name: <Engine.Core>engine/core/coresettings.cpp
: Depends on <Engine.Core>Engine/Core/CoreSettings.ddf (newer) (time:1231453738)
3 Name: <Engine.Core>engine/core/coresettings.h
Loc: J:/Dice/xxx/TnT/Local/Build/win32_debug_static/engine/core/coresettings.h
: Updating it
: Depends on <Engine.Core>Engine/Core/CoreSettings.ddf (newer) (time:1231453738)

Whenever a file in the Engine.Core library include CoreSettings.h, things work fine:

5 Name: <Engine.Core>Module.cpp
: Depends on <Engine.Core>engine/core/coresettings.h (update) (time:1231960628)

However, when other libraries include the generated files from other libraries, things break:

4 Name: <Engine.Pipeline>Driver/PipelineApp.cpp
: Depends on <Engine.Pipeline>engine/core/coresettings.h (stable) (time:0)

This last dependency is ignored due to the dependency scanning rules, so if PipelineApp.cpp is indeed
scheduled to compile before the library which generates the header, a compilation error occurs. This
happens almost instantly when enabling parallel compilation with Jam's -j flag.

The reason (AFAICT) is due to the header gristing feature which makes <Engine.Core>engine/core/coresettings.h and <Engine.Pipeline>engine/core/coresettings.h look like to different targets to Jam.

Is there a good workaround for this?

RE: How to generate source files? - Added by Joshua Jensen about 11 years ago

The HdrRule in JamPlus doesn't currently handle this, but I have done this before. The way around it is to override the HdrRule with a NO_HDRGRIST list and use it to not grist certain generated headers.

I'll knock out an example of it soon.