Re: 6 DAY compile! Any volunteers for distributed build CPU sharing?

noreply@xxxxxxxxxxxx (Patrick Machielse) writes:

I remember a senior Apple engineer exclaiming: 'NO build should take
more than 2 minutes'... (this was after demoing a distributed build of
the Finder using a stack of XServes).

I've got some projects that would disprove his theory.

Where I work, our code consists of several millions of lines of code,
that are cross-compiled from various Linux and Solaris/SPARC host
workstations to our target hardware platform. The build environment is
distributed across a pool of 8 4-CPU Linux servers, 6 4-CPU UltraSPARC
III servers, and available CPU cycles on users' workstations (using a
screen-saver that adds the workstation to the compiler-pool whenever the
computer is idle.) There's also a company-wide cache, so different
developers never have to compile identical files.

Even with all this distribution, a change to a critical header file can
still cause a build to take two hours. (Without distribution, a full
build can easily take over 24 hours.)

I think that Apple engineer was speaking in hyperbole.

In practice distributed builds seem useful only if you can afford a
dedicated build farm and a high speed wired network. Trying to
leverage the unused capacity of regular machines on your local network
is a different story.

Without a dedicated compile farm, you won't see dramatic speedups. And
without a cache, multiple developers compiling the same code at once
will end up wasting all that distributed compiler power.

Using spare cycles of workstations does work, but you have to be careful
not to kill the desktop responsiveness of those computers when people
are running apps in the foreground.

-- David
.

Relevant Pages

  • Re: Which host distribution?
    ... What linux distribution uses a 2.4 kernel and XFree86 3.3.6 and is able to ... I want to compile some kind of minimalistic linux for my notebook ... My compiling distribution is Debian Woody. ... but BusyBox doesn't compile on this ...
    (comp.os.linux.embedded)
  • Win32 Platform SDK
    ... I'm currently studying/experimenting with Win32 native applications. ... I assume i can do this using my student edition VC++ 6 free of charges. ... Also can I develop a native application in Express 2005 and then compile it in VC++ 6 for free distribution? ...
    (microsoft.public.vsnet.general)
  • Re: large files: when ubiquitous?
    ... | There is a dependency because you need to compile all ... not a full distribution suite. ... | I have seen some bugs where the highmost bits were ... | complete distribution with 64 bit file offsets by ...
    (comp.os.linux.development.system)
  • Which host distribution?
    ... What linux distribution uses a 2.4 kernel and XFree86 3.3.6 and is able to ... I want to compile some kind of minimalistic linux for my notebook ... My compiling distribution is Debian Woody. ... XFree86 3.3.6, but BusyBox doesn't compile on this distribution (it uses gcc ...
    (comp.os.linux.embedded)
  • Re: Distributing frontend
    ... > How can I automate the distribution of the frontend to the workstations? ... > Make a batch file using xcopy? ...
    (microsoft.public.access.modulesdaovba)