Parallel building debian kernels ... and why its not working ... and how to make it work

So we build our own kernels. No great surprise, as we put our own patches in, our own drivers, etc. We have a nice build environment for RPMs and .debs. It works, quite well. Same source, same patches, same make file driving everything. We get shiny new and happy kernels out the back end, ready for regression/performance/stability testing. Works really well. But … but … parallel builds (e.g. leveraging more than 1 CPU) work only for the RPM builds. The .deb builds, not so much. Now the standard mechanism to build debian kernels involves some trickery including fakeroot, make-kpkg, and other things. These autogenerate Makefiles, targets, etc. based upon the rule sets. Fine, no problem with this. I like autogenerated things. Actually I often like programmatic generated things better than human generated things, as the latter invariably have crap you really don’t want in there. Not that the others don’t, but there is mysticism around the existence of some things in peoples build environment, versus empirical reality. The canonical mechanism is to use CONCURRENCY_LEVEL=N for N=some integer. Fine. Use it in the make file. And … We have a stubborn single threaded build. It will not change. Fine, lets capture output and make it verbose. Look for concurrency level in output. See if something is monkeying with it.

scalablekernel@build:~/kernel/3.18$ grep CONCUR out
export CONCURRENCY_LEVEL=8
cd linux-"3.18""" ; export CONCURRENCY_LEVEL=8 ; fakeroot make-kpkg -j8 --initrd --append-to-version=.scalable --added_modules=arcmsr,aacraid,igb,e1000,e1000e,ixgbe,virtio,virtio_blk,virtio_pci,virtio_net --overlay-dir=../ubuntu-package --verbose buildpackage --us
DEB_BUILD_OPTIONS="" CONCURRENCY_LEVEL=1     \

/sigh I look to see where that is coming from. Looks like debian/ruleset/targets/common.mk. Which is in turn, coming from /usr/share/kernel-package/ruleset/targets/common.mk . Look for concurrency and see this snippet

debian/stamp/build/buildpackage: debian/stamp/pre-config-common
        $(REASON)
        @test -d debian/stamp      || mkdir debian/stamp
        @test -d debian/stamp/build || mkdir debian/stamp/build
        @echo "This is kernel package version $(kpkg_version)."
ifneq ($(strip $(HAVE_VERSION_MISMATCH)),)
        @echo "The changelog says we are creating $(saved_version)"
        @echo "However, I thought the version is $(KERNELRELEASE)"
        exit 1
endif
        echo 'Building Package' > stamp-building
# work around idiocy in recent kernel versions
# However, this makes it harder to use git versions of the kernel
        $(save_upstream_debianization)
        DEB_BUILD_OPTIONS="$(SERIAL_BUILD_OPTIONS)" CONCURRENCY_LEVEL=1     \
          dpkg-buildpackage $(strip $(int_root_cmd)) $(strip $(int_us))     \
            $(strip $(int_uc)) -j1 -k"$(pgp)"  -m"$(maintainer) < $(email)>"
        rm -f stamp-building
        $(restore_upstream_debianization)
        echo done >  $@

[starts banging head against desk again] Force concurrency level to 1, AND then force -j1. Oh dear lord. Lets see if switching these back to 8 helps (8 core machine). Why … yes, yes it does … Grrr [Update] The deities of kernel building are not kind. It appears that parallel build in debian actually breaks other things that it should not break. I have a choice of a very slow (1+ hour) kernel + module build that works, or a fast (roughly 5-10 minute) kernel + module build that fails because of a causality violation (e.g. someone couldn’t figure out how to fix their code to run in parallel). So, if I have time in the next month or so, I am going to find that very annoying serializer, take it out behind the barn, and put it out of my misery.