Includes + ScanContents bug
I believe I've discovered a bug in jamplus in my build environment, and after some trial & error I managed to reproduce it.
I'm attaching repro files - Jamfile.jam with build rules (no Jambase dependencies), and repro.bat control script (it creates source files for jamfile and launches jam several times with appropriate modifications in between to reproduce the bug). Near the top of repro.bat JAM environment variable is set; I have 'call jam' there since I have a rerouting jam.bat in my path instead of actual jam.exe.
The description of build is roughly as follows:
1. there are two resource types, mesh and texture
2. each one is build with the own tool in actual build, simplified to single action in test repro
3. each tool has a version which is stored in a .version file (in actual build .version files depend on tool .exe files and are generated automatically, in the test repro they're updated by repro.bat since this does not affect the bug)
4. .version file is marked as ScanContents (in actual build the intention is to rebuild mesh/texture files only if tool version changes, not on every tool exe change)
4. mesh and textures depend on their sources; mesh target file includes texture target file - I do not need to rebuild mesh if texture changes, but I do need to have texture file around for every target that builds mesh since I need texture file to load mesh file. SCons calls this type of dependency 'requires', which is a better name in this case.
Now on to the bug. Action sequence is described in repro.bat, but just in case:
1. I create mesh.version and texture.version files and launch first time build. It builds mesh/texture target files and updates depcache (as expected)
2. I launch another build to verify correctness - it builds nothing, as expected
3. I touch mesh.version file without changing its contents - subsequent builds build nothing (as expected), but print "updating 1 target" message - which is (I guess) an indicator that ScanContents rule is in effect, as per documentation
4. I touch texture.version file without changing its contents - next build builds mesh file (this is a bug), all subsequent builds build nothing (as expected)
1. Both include and scancontents parts are relevant, replacing Includes with Depends removes the bug (although perhaps it's still possible to reproduce it with a more complex tree) both in actual build and in test repro
2. repro.bat contains pauses - wait a couple of seconds on each one so that timestamps are actually incremented (without pauses mesh/texture versions timestamp can stay the same across the entire process)
3. I tried to understand what's going on using -dm, -dg and -df outputs and did not succeed; dependency graph looks weird though.
P.S. My initial implementation of tool versions used UseCommandLine and Shell rule, however several (6-7 total, one per tool) exe launches created a noticeable delay (half a second IIRC). Perhaps there is another solution for this problem, but this one is the most obvious that comes to mind...
#1 Updated by Arseny Kapoulkine about 9 years ago
Forgot to mention -
1. reproduces on latest nightly build (100120) and also on 091230, 090908 (so not version specific)
2. for some reason it behaves differently without default Jambase - the description above was for default Jambase, if I use Jamfile.jam as Jambase via -f Jambase.jam, the behavior is different - stage5 builds mesh in first build (which is still a bug but for some reason manifests in a different place).
#3 Updated by Joshua Jensen about 9 years ago
I do think there is more to this than this short response. I have a very limited amount of time right now, so hopefully, this will get you going. When time frees up, I'll look at the possible bug in greater detail.
What you want to use instead of
Needs will make sure the texture exists, but changes to the texture do not cause the mesh to update.
I'll go into reasons why
Includes is likely not correct in another response to this soon.
#4 Updated by Arseny Kapoulkine about 9 years ago
I feel ashamed. You're right, this should've been Needs. I can't track this change down, but IIRC it was intended to indeed add an include dependency to mesh so that the targets that depend on mesh (the level pack file) would be rebuilt on texture change but the mesh itself would not; the pack file was done another way anyway (it depends directly on all level files, including mesh and texture ones), so this is not needed (and is wrong anyway, it worked for me because I never referred to mesh via its file name but always via an alias made via NotFile + Depends).
Also with Needs I can understand dependency graph; the thing that confused me is that ScanContents is not reflected in -df or -dm output, and is mentioned in -dg briefly (i.e. it says that the node is ScanContents but does not say anything about the contents state, and also states that it's newer and it's parent is outdated; I assume this is because ScanContents evaluation happens in updating phase, and dependency information is printed before that).
I still do not understand why with Includes there are two mesh nodes in the dependency tree (both are called build/dragon.mesh, the second one is the child of the first one and is unbound).