This document will help prepare you to build the source code for illumos. It assumes some familiarity with development on Unix-like systems.
You need an illumos-based operating system to build illumos (cross-compilation is not supported). If you don't have one installed, the quickest way to get started is to download and install OpenIndiana from openindiana.org/download. If you have an existing system with OpenSolaris installed, see Updating OpenSolaris.
: All commands in this guide assume you run them as an unprivileged user.
sudo will be prepended to commands which need additional privileges; on some systems
pfexec can be used instead, if your user has the correct RBAC profiles. The generic commands below use a
$USER variable to define the current (unprivileged) user account's name which will ultimately be used to compile the project. If you are not using a stock shell or have manipulated the
$USER variable in your environment at a site, system, or user level, you are responsible for diagnosing and addressing any consequences.
Note: VirtualBox has proven unfit for purpose in building from source (builds take days instead of hours) and running debug kernels (systems have problems with clock discipline and are slow or become unresponsive). If you're interested in developing for illumos, consider running on bare metal or using an alternative virtualization product (KVM, VMware, Xen).
Update (2013-06-28): Some users have reported reasonable build times with recent (>= 4.2) versions of VirtualBox on recent Apple hardware. It is as yet unknown what specifically changed to resolve the issue, but presently it appears that recent VirtualBox may be more feasible. Until we know the specifics, consider VirtualBox unsuitable or experimental at best. More testing welcome! This only applies to building illumos, and not running debug kernels.
/code/illumos-gate/packages/i386/nightly/repo.redist) is not created and
onu script fails
Installing required packages
You need to install these packages in order to build illumos:
The source code was traditionally built using Sun's closed-source compiler suite, which was named Sun Studio at the time of the illumos fork. The specific versions of Studio required are no longer publicly available, it is possible to build illumos with GCC; this is the only option for "new" people in the project who do not have access to the distribution archives of required Sun Studio versions. You will still need some tools from Sun Studio even if you use GCC as your compilers, but those are available as packages in the legacy OpenSolaris repository.
It is possible to build illumos using GCC (the GNU Compiler Collection, specifically its C and C++ compilers), and you should generally prefer to do this, though it is convenient to still see build results from Studio in RTIs
The primary version of GCC necessary for development and integration 4.4.4 (4.4.4-il-2) should have been installed on your system when you installed the prerequisites above. If for some reason that didn't happen (old build system, or the like) tarballs are available as described in the initial heads-up notice 2012-06-15 Illumos will now build with GCC 4.4.4 + patches.
Note that the compiler for building
illumos-gate installs into
/opt/gcc/4.4.4 and is not available in the default
PATHfor building application programs. You can add
PATH="/opt/gcc/4.4.4/bin:$PATH"; export PATH to your profile to reference this compiler, or install one of the other GCC builds, destined as the current default compiler for application programs (versions ranging from 3.4.3 to 4.6.x, see
pkg info -r '*gcc*' for details du-jour).
Unfortunately, Sun Studio
dmake (Distributed Make) and
lint are currently required even for GCC-only builds. We currently deliver Lint libraries built using the closed Sun Studio version of
lint from the unpatched (but redistributable) version of Sun Studio 12.1 in the opensolaris.org repository can be used, regardless of its utility for actually checking the software.
You'll want the symlink so that
lint can be found during the build process.
Sun Studio compilers
If you wish to build with Sun Studio, you need specifically patched versions of both Studio 12 (used to compile) and 12.1 (used for lint) to get consistent results. If you do not have access to these versions, skip this section and use GCC.
illumos-gate does not build correctly with Sun Studio versions released after 12.0. Reportedly using a newer version of the compiler will produce a system containing grave, data-corrupting bugs. Patched compilers were previously available from Sun Microsystems, however not licensed for redistribution. After a recent Oracle site reorganization, the necessary compilers are no longer available.
If you have the patched versions of both Sun Studio 12 and 12.1, they should be installed in
/opt/SUNWspro (Studio 12), and
/opt/SUNWspro/sunstudio12.1 (Studio 12.1), respectively. Pay attention, the Studio 12.1 location is different from how it was in OpenSolaris, i.e. it's not
The meta-procedure for cleaning up optionally prepackaged Sun Studio installations and installing the ones you require for building illumos (assuming your stashed Sun Studio 12 distribution archives are now pre-copied to
/tmp) is as follows:
The first archive contains the product component sub-directories directly at the top-level, and the second has similar sub-directories under the top-level
sunstudio12.1 as its children. Thus you receive the structure outlined above.
You can check if you have the correct versions installed as follows. All of the following tests must succeed and return these specific product/patch versions:
Preparing the source-code directory
You can use any location, but for the purposes of this guide, we will assume that you have a directory
/code owned by your user ID, which you are using to obtain source and build.
Getting the source
You may use the Git or Mercurial source code management systems to retrieve the illumos source code. Both tools will be able to retrieve the same history. Ensure about 6 gigabytes of free space for the source and binaries combined.
Getting the source with Mercurial (hg)
illumos.org hosts a Mercurial repository containing the illumos source code.
illumos-gate URI is:
http://hg.illumos.org/illumos-gatedoes not work. Use ssh.
Additional Mercurial mirrors:
Mercurial pulls a complete replica (clone) of the repository, including all history, from a remote source, and then updates a working copy of the files from this clone. The working copy is the files you may edit and compile. You may update the working copy from any versions of files in the repository history.
When cloning an external source code repository, Mercurial starts by creating a new directory, and fails if one already exists and is not empty. The directory may be a mountpoint for an empty ZFS dataset. Mercurial creates a
The following example creates a clone of the "Mercurial repository" ssh://email@example.com/illumos-gate containing the illumos source code under
/code; this will create a sub-directory
illumos-gate under that.
Updating the source with Mercurial
If you want to get latest version of
illumos-gate merged with your private development code, you need to
hg pull new changes into your clone and
hg update the working copy. This can be done in one command as
hg pull -u. If you have committed changes prior to updating, you may be wish to
hg merge them with the new changes.
For more details see: http://mercurial.selenic.com/wiki/FAQ.
There is also a simple script to update the repository.
Getting the source with Git
The repository is also available from Github using the Git source code control system.
Install Git using this command:
The Git repository URI is:
Or, if the Git protocol is firewalled at your site:
illumos-gate source can then be cloned using:
Getting closed binaries
There are still closed components in illumos that have not yet been replaced. You must get the latest closed binaries from Sun/Oracle, since illumos currently needs those in order to build:
Note that if you had a
crypto tarball from an earlier build, do not use it here, as it is no longer required to build illumos. You should remove it from your tree if already present.
Preparing the build environment
General build configuration
The build is configured using a shell script with environment variables, a template for this file is included:
The settings we are going to focus on for now are the following:
NIGHTLY_OPTIONS- You may remove the "
L) option from the string, this will disable
lintchecking of the code to save some time during the build, but patches should be linted.
CODEMGR_WS- This should be the root of the directory with the code. If you followed the previous example, it will be
STAFFER- Change this to the name of the non-privileged user you use on the system
VERSION- Set this to
illumos-gateor whatever version string you want for the build
ONNV_BUILDNUM specifies the build number which is used in packaging. If you intend to upgrade a system whose build number is greater than the one in illumos-gate (currently 148), such as OpenIndiana, you need to specify this to allow upgrades until issue #1118 is addressed. Run:
If you see, for example, "
Branch: 0.151.1", you must choose a number greater than the part after the leading "
0.". For example, add
Building with GCC-4.4.4 (i.e. without Sun Studio)
If you are building with GCC 4.4.4 (i.e. you do NOT have Sun Studio with appropriate patches), you must append these lines to
Explanation of these settings:
__GNUC- Set this to "" (the empty string), and
exportit. This causes GCC to be used as the main compiler. Studio takes its place as the "shadow compiler".
CW_NO_SHADOW- Set this to 1 to prevent the shadow compiler from running (for instance, if you do not have Studio at all).
- ONLY_LINT_DEFS - This is needed so the
lintis able to find the proper
The last two lines make sure the patched version of GCC-4.4.4 is used (the default gcc version shipped with e.g. OpenIndiana, gcc-3.4.3, is not suitable for building illumos), as described here.
For the moment we still require that contributed changes build with Sun Studio as the compiler. We understand that the required version of Sun Studio is no longer publicly available and will accept contributions that were tested only with GCC.
If you are trying to build with GCC 4.4.4 and your build fails because of some invalid GCC warning options, try setting the variables GCC_ROOT and CW_GCC_DIR as shown above and described here.
Starting the build
You are now ready to start the illumos build.
Since this process may take up to several hours depending on your system (reference build times are in a chart below), it may be wise to start the build job in a VNC session, GNU Screen (
screen program can be installed with the
Run the following command to start the build:
Note that the command does not give any progress output. You can instead follow the log file at
log/nightly.log, which is updated (slowly) during the build process. In another terminal, run:
After the build is complete, the
nightly.log file and some other generated logs are moved into a uniquely named subdirectory
A dry overview of the build progress (hopefully small – about 2KB in size – with no errors) is saved in
log/log.$TIMESTAMP/mail_msg file intended for mailing to the build administrator after the build completes (this automation is out of the scope of the
nightly.sh script). If you do receive any errors in the
mail_msg report file, you can grep the error lines in the larger
Performing an incremental build
If you've made changes after completing a nightly build, you can perform an incremental build without discarding the already built files.
To make this the default, edit
illumos.sh and add the character
Before submitting a patch, we request that you perform a full, non-incremental build.
Building specific components, and other info
Installing built software
Installing the nightly build on local machine with
The build process will generate updated packages. Depending on the
NIGHTLY_OPTIONS variable in
illumos.sh script, the packages will be written either to
$PWD/packages/i386/nightly (with debug, the typical default) or to
$PWD/packages/i386/nightly-nd (without debug). Therefore there are two possible commands to install the software you you just built. You need choose the proper
onu command, typically the first one listed below.
Typical install: you can install with
onu for debug builds (the default, with '
F' and '
D' characters in
NIGHTLY_OPTIONS variable in
The non-typical installs are detailed in the following pages:
- Advanced - non-debug install describes different
onuoptions for the different set of built packages.
- Redistributing built packages and Installing built illumos packages into BEs on a remote host and non-onu updates describe how to install illumos using the package depot server over the network, perhaps on a machine different from your build host. This recipe bypasses the
onuscript automation and manipulates your boot environments with
illumos has entirely different localization infrastructure from Solaris, so your existing system locales will not work.
The English locales for illumos are in
pkg:/system/locale/en), and if you wish to use them, you'll need to install that package after you reboot. Likewise, other locales are in
pkg:/locale/<language code> where
<language code> is a two letter ISO 639-1 code, except for
If your locale is not installed, it will cause problems with software which behaves badly if
setlocale() fails. This includes, at least,
pkg(5) commands print a warning, but function normally.
If this is a problem, you can "
export LANG=C LC_ALL=C" as a workaround, and change the
LANG setting in
/etc/default/init to '
C' (and reboot) to make the change permanent.
Package repository (path
/code/illumos-gate/packages/i386/nightly/repo.redist) is not created and
onu script fails
This error has happened to me whenever I ran the default checkout and full-build procedure outlined above (in VirtualBox VMs); I don't yet know why (possibly due to a wrong setup of my SunStudio compiler stack, as was later discovered).
However, rerunning with an incremental build has created the package repository correctly (as well as full builds with proper SunStudio setup):
In any case, if you have build errors or other "inconsistencies", it would be wise to revise the
mail_msg logfile that can contain short error descriptions, and
grep for those errors in the longer
nightly.log file. If that doesn't help, you should also recreate the build in a "clean lab" environment, which may be set up as a local zone according to the Building in zones instructions and rule out the local environment's influence (conflicting libraries and binaries, perhaps from your own earlier builds, come to mind first).
As a convenience, we will try to assemble some build times for various hardware configurations here, so you can get an idea in advance of how long it will take:
|Processor(s)||No. of cores||Memory||Time (HH:MM)||Notes|
|2 x Xeon X5570 2.93GHz||2 x 4||36 GB||00:19||NIGHTLY_OPTIONS='-FnCDmprt'|
|4 x Opteron 6172 2.1 GHz||4 x 12||128 GB||00:21||NIGHTLY_OPTIONS='-FnCDmprt'|
|Xeon E3-1270 3.4 GHz||4||16 GB||00:23||NIGHTLY_OPTIONS='-FnCDmprt'|
|Xeon E3-1245V2 3.4 GHz||4||16 GB||00:31||NIGHTLY_OPTIONS='-FnCDmprt'|
|Core i7-2600K 3.4 GHz||4||8 GB||00:35||NIGHTLY_OPTIONS='-nClmprt'|
|2 x Xeon E5620 2.4 GHz||2 x 4||48 GB||00:38||None|
|1 x Xeon E5607 2.27GHz||1 x 4||12 GB||00:46||NIGHTLY_OPTIONS='-FnCDmprt'|
|Xeon E3-1245V2 3.4 GHz||4||16 GB||00:47||NIGHTLY_OPTIONS='-FnCDlmprt' (with lint)|
|2 x Xeon E5540 2.53 GHz||2 x 4||24 GB||00:50||With lint|
|2 x Xeon E5506 2.13 GHz||2 x 4||24 GB||00:50||No dmake check|
|1 x Xeon X5650 2.67 GHz||1 x 6||12 GB||00:51||With lint, default NIGHTLY_OPTIONS|
|2 x Xeon E5506 2.13 GHz||2 x 4||24 GB||01:03||None|
|Core i7-960 3.2 GHz||4||9 GB||01:03||With lint|
|2 x Xeon E5506 2.13 GHz||2 x 4||16 GB||01:06||VMware Workstation guest|
|Core i7-930 2.8 GHz||4||8 GB||01:07||VMware ESXi guest|
|Core 2 Quad Q6600 2.4 GHz||4||2 GB||01:16||None|
|2 x Xeon E5310 1.6 GHz||2 x 4||32 GB||01:24||None|
|1 x Athlon II X2 240 2.8 GHz||2||2 GB||01:27||NIGHTLY_OPTIONS='-FnCDmprt', on a low end SSD|
|Core 2 Quad Q9300 2.5 GHz||4||7 GB||01:32||None|
|2 x Xeon E5420 @ 2.50GHz||2 x 4||4 GB||01:33||Default NIGHTLY_OPTIONS; an incremental rebuild takes 13 min to walk all Makefiles|
|2 x Opteron 2218HE 2.6 GHz||2 x 2||16 GB||01:42||None|
|Core i5-540M 2.53 GHz||2||4 GB||01:54||None|
|Core i3-370M 2.40 GHz||2||2 GB||02:30||None|
|Core 2 Quad Q8200 2.33 GHz||4||5 GB||02:30||Default NIGHTLY_OPTIONS.|
|Core 2 Duo T8300 2.3 GHz (T61)||2||2 GB||02:58||VMware Workstation 2 CPUs|
|Core 2 Duo E6750 2.66 GHz||2||8 GB||02:59||Sun Ultra 24 (with lint)|
|Core 2 Duo Celeron E3200 3.2 GHz||2||4 GB||03:28||2.4 GHz CPU over clocked to 3.2. Everything running on a low end SSD. With lint.|
|Core 2 Duo T5600 1.8 GHz||2||3.3 GB||04:26||None|
|Opteron 146 2 GHz||1||4 GB||07:28||None|
|4 x Dual-Core Opteron 8218 2.6 GHz (8-vCPU VM)||16 GB (2GB VM)||35:47 + 9:05||VirtualBox 3.0.12 VM, with 8 vCPUs and 2GB vRAM; local vHDD; with lint and default NIGHTLY_OPTIONS='-FnCDlmprt'|
NOTE: Package repo did not get built at all with the first nightly run, only with the second, incremental nightly run (another 9 hours)
- "Bare Metal" just slightly faster than VMware guest. On the other hand, VirtualBox painfully slow.
- More cores faster but not linear increase.
- SSD use seems to provide nice speedup.
- As expected, lint checking results in longer build times.