org-build-system.org 24 KB

Introduction

Org can be run directly from sources, however usually it is byte-compiled and installed before use. Also, the documentation needs to be rebuilt from their respective sources, registered and put into a place where it can be found. Instead of you having to do all this by hand, a build system based on GNU make will do it for you. The Org build system makes extensive use of GNU make features and requires at least version 3.81 (use of the latest released version is recommended).

Build System Structure

The build system consists of these files:

/Makefile
/local.mk
/doc/Makefile
/etc/Makefile
/lisp/Makefile
/mk/default.mk
/mk/targets.mk
/mk/org-fixup.el

Additional files are present for the Orgmode server in the =/mk/= subdirectory. An additional file =/mk/version.mk= is present in the distribution archives to record the version information that gets used when building from these sources.

Top Level Makefile

This file provides only the help. Short help can be requested by =make help=, while make helpall will list all documented targets. Undocumented targets may exist for internal purposes and can be changed or removed at any time.

All other functionality of the build system is provided in separate files, which will either be explicitly included from the top level or implicitly read by GNU make.

Local Customization local.mk

Never push local.mk to any public branch!

The build system is an integral part of Org. To allow easy local customization, that customization must never become part of the official Git repository. On the other hand the build system must work even when no local customization has been applied. Therefore when the file local.mk does not exist, a local customization template will be created that copies some settings that most commonly would need to be changed from mk/default.mk. The template includes some comments that are hopefully self-documenting. As long as all necessary programs are available, the customization should in most cases be limited to setting prefix correctly.

If you need to do more extensive changes, copy the relevant parts from =mk/default.mk= to local.mk (or maybe even the whole file if that seems easier to you, but you'll have to remember to check local.mk= whenever =default.mk changes in the repository). Please look at some of the Customization Examples to get an idea of what is possible.

Second Level Makefile

The Makefile in subdirectories can be invoked from the top level only since they rely on the definitions that have been made there.

Defaults mk/default.mk

Never change anything in mk/default.mk — always do your changes in* local.mk !*

The defaults are designed to work on a typical GNU/Linux system. For system-wide installation as assumed by the build system the user must be able to obtain administrative privileges via sudo. If sudo is not available, the user running make must have the necessary privileges to do the install (i.e. full access to the build and install directories). Besides Emacs, the following programs must be available in PATH:

  • find
  • install
  • rm

The following programs are optional when loss of functionality can be tolerated:

  • install-info
  • makeinfo
  • texi2pdf
  • sudo
  • git

Make Targets mk/targets.mk

All targets that the Org build system can build are defined here. Most of the actual work will be handed off to the second level =Makefile= in their respective subdirectories. It is possible to override some parts of the build system by defining some variables. This is largely untested and consequently unsupported, but may still be useful in some situations. The following variables are considered stable:

TEST_NO_AUTOCLEAN
Define to a non-null value to keep the test
directory around for inspection. This is mostly useful for debugging the test suite.
ORGVERSION
Can be used to override the automatic determination
of the version strings. If you find a need to do this, it is likely a bug someplace else. *Never define this in a customization file!*
GITVERSION
See ORGVERSION. Please do not define it to a
value that could be confused with a real Git version string.
GITSTATUS
See GITVERSION. If this variable is defined
non-null, the string ".dirty" will be appended to the Git version string, which indicates a locally modified version.

Utilities mk/org-fixup.el

This is a collection of some Emacs Lisp routines that implement basic functionality of the build system. This mainly eliminates the need for some external programs and thus reduces the number of external dependencies.

A few of these functions have been designed to be used from the command line or even from within Emacs itself. This is an aid for manually building a working Org installation when the external dependencies of the build system cannot be met. See Support for Manual Build.

Make Targets

Each time you want GNU make to build something for you, you need to tell it what that is: this is called a target or goal. For each /target/, make determines the prerequisites and then goes on to build them in the order of dependence. A target can be an actual file that GNU make should build, but more commonly it is just a moniker (called a phony target) that has the files to build as /prerequisites/.

Help

help
Shows a brief list of the most commonly used targets
with a short description of what they do.
targets
This is an alias for help, mandated by GNU Makefile
conventions.
helpall
Shows (almost) all targets and a short description of
what they do.

Configuration Check

config-test
Show the test customization.
config-cmd
Show what commands will be used.
config-all
Show all customization.

Build

all
Build all of Org: byte-compile the source files and create
all documentation.[fn:1]
compile
Ensure a clean source directory and then byte-compile
the source files using the [[Compilation Methods][compilation method]] =dirall= by default.
compile-dirty
Byte-compile just those sources that haven't been
compiled already or are newer than their byte-compiled counterpart.[fn:2]
single
The same as compile, but uses the compilation method
=single= (unless overridden by defining =ORGCM= on the command line).
autoloads
Create just the autoload files, but do not
byte-compile anything.

Test

test
runs compile and then the full testsuite.1
check
An alias for test, to be compatible with GNU style.
test-dirty
Run the full testsuite on whatever currently is
available, compiled or not.

Installation

install
Build all of Org and install it.
install-etc
Install only the etc/ part of Org.
install-lisp
Install only the lisp/ part of Org.
install-info
Build the documentation and install only the info
documentation.

Documentation

doc
Create all documentation.2
docs
An alias for doc.
info
Create only GNU Info documentation (requires GNU Makeinfo).
pdf
Create only PDF documentation (requires PDFTeX).
card
Create only the reference card (requires PDFTeX).
refcard
An alias for card.

Cleaning

clean
Cleans in lisp/ and doc/.
cleanall
Cleans everything that can be cleaned, including
several types of backup files, so do not use this when you have active edit sessions!
clean-install
Removes a previous Org installation.3
cleantest
Removes a test directory if it exists.4

Compatibility and Convenience

up0
Updates the current Git branch from upstream by doing a
=git pull=.
up1
Does up0 and then builds and checks Org.
up2
Does up1 and installs Org if there was no test error.
update
Does up0 and then builds Org. Does not test.
update2
Does update and then installs. This is not
recommended, since there is no way of telling whether the just built Org has errors.
uncompiled
Removes any byte-compiled files and then creates the
autoload files. You can then use the Git worktree almost like an installed version of Org. Not recommended for normal use of Org.
local.mk
Create a customization template. If one already
exists, you need to rename or remove it first.

Customization

Changing the behaviour of the build system to conform to your local system rules is done by editing the file local.mk. The standard template that is created when this file does not exist offers only the most common customization variables, but you are free to customize anything that mk/default.mk offers (but you really have to know what you are doing for some of this). Remember to only change local.mk, please.

Simple Customization

Default target

The default target is what make tries to build when you don't give it anything else to do. For compatibility with the old build system, a freshly created local.mk will have oldorg defined as the default target. If you remove that line entirely from local.mk, all will become the default target. But you can put any other target there that you want to become the default target or even define a new one (OK, that isn't simple customization anymore).

Including sources from contrib/

If you just want to try out some of the things in contrib/, you can simply add the directory to load-path. But if you want to include some files in an installed version of Org: simply specify in the customization variable ORG_ADD_CONTRIB which files you want included, then build and install in the usual way. Your =local.mk= default customization template has a commented out example for including the new exporter, you just need to remove the comment marker:


  # Define if you want to include some (or all) files from contrib/lisp
  # just the filename please (no path prefix, no .el suffix), maybe with globbing
  ORG_ADD_CONTRIB = ox-* # e.g. the contributed exporter

You just give the base name of the file to include (much like you do in a require form), only that you can use a shell globbing pattern to specify many similar names at ones. You do not need to specify the path prefix contrib/lisp/ nor the file suffix =.el=, these are added by the build system. To include all of contrib/lisp/ (some of these aren't really meant to be used together, so this isn't recommended) you'd say:


  ORG_ADD_CONTRIB = org* ox*

Or if that was just a one-time install (with quoting for a POSIX shell):

make ORG_ADD_CONTRIB="ox-*" install

Note: A simple * would also include htmlize.el, which is currently bundled in contrib. It is recommended to install that seperately, it is available for instance in GNU ELPA.

Non-standard Emacs location

Customization for using a self-compiled Emacs 24 installed in =/usr/local= and the default target changed to up2. Additional customization to enable htmlize installed from ELPA in users' home directory and ESS (for R) in the system =/usr/share/emacs/site-lisp/= and all Babel languages activated for testing.


  up2::   # default target
  EMACS   = /usr/local/bin/emacs-24.3
  prefix  = /usr/local/share
  lispdir = $(prefix)/emacs/site-lisp/org
  datadir = $(prefix)/emacs/etc/org
  infodir = $(prefix)/info

  BTEST_EXTRA = ess-site 
  BTEST_OB_LANGUAGES = awk C fortran maxima lilypond octave python sh R
  BTEST_POST = -L ~/.emacs.d/elpa/htmlize-1.39 \
               -L /usr/share/emacs/site-lisp/emacs-ess-12.04.4

XEmacs

Customization for using XEmacs 21.5, since there seems to be no ERT for XEmacs testing will not work and has been disabled. The default target is set to up0 doc uncompiled (pull from Git and update documentation and autoload files).


  .PHONY: xemacs
  xemacs: up0 doc uncompiled
  EMACS   = xemacs
  prefix  = /usr/local/share
  lispdir = $(prefix)/xemacs/site-lisp/org
  datadir = $(prefix)/xemacs/etc/org
  infodir = $(prefix)/info

  BTEST = /bin/true
  BATCH = $(EMACS) -batch -q -vanilla # XEmacs
  # How to byte-compile the whole source directory
  ELCDIR  = $(BATCH) \
                  --eval '(add-to-list '"'"'load-path ".")' \
                  --eval '(byte-recompile-directory "." 0)'

Emacs on Windows

Customization for using vanilla Emacs 24 on Windows, with GNU make and other binaries provided by Cygwin. Make sure the installation path(s) contain no spaces! Use the 8.3 compatible names, e.g. PROGRA~1= instead of "Program Files", if you already installed the applications in such a location. Babel languages have been stripped down for testing and the default target is again set to =up2.


  up2::
  CYGWIN += nodosfilewarning
  prefix  = C:/Freeware/Emacs-24.3
  EMACS   = SHELL=sh $(prefix)/bin/emacs
  lispdir = $(prefix)/site-lisp/org
  datadir = $(prefix)/etc/org
  infodir = $(prefix)/info

  BTEST_OB_LANGUAGES = octave
  SUDO =

Advanced Customization

Compilation Methods

The default compilation method compiles all source files within the same Emacs process, simply because that is the fastest method. Unfortunately, Emacs does not isolate the side-effects of compilations from each other, so the byte compiler may not issue some errors or warnings (mostly about missing declarations or requires). To enable developers to catch these errors, different compilation methods can be configured by defining ORGCM to one of these values (either permanently in local.mk or for a single invocation of make):

dirall
The default compilation method, invoked via ELCDIR.
single
Uses a separate Emacs for each compilation, invoked via
=ELC=.
source
Uses a separate Emacs for each compilation invoked via
=ELC=. Removes all =*.elc= files before and each =*.elc= file directly after compilation so that all requires are also loaded from source. Recompiles via =dirall= at the end so that all the =*.elc= files exist.
slint1
First compiles using dirall, then compiles each file
again using =single=.
slint2
First compiles via source and then again via slint1.

Both ELCDIR and ELC are also customizable, but changing their definitions must not alter the semantics of compilation. You have been warned.

Multiple Emacsen

Method 1

If you're a developer (or a system administrator that serves a diverse set of users) you'll likely have to deal with the need to build and test Org for different versions of Emacs. Having to copy or link the correct customization file to local.mk quickly loses the appeal and is error prone. Here's a (hopefully better) suggestion: put each customization into a file named local-<pattern>.mk and create a =local.mk= that looks like this:


  ifdef LOCALMK
    include local-$(LOCALMK).mk
  else
    include local-emacs24.mk
  endif

Now switching between your different customizations is as easy as

make LOCALMK=emacs23

(which assumes that there is a customization file =local-emacs23.mk= available).

Method 2

If you need a more flexible structure, say to test different versions of Emacs with different configurations, then you quickly end up with lots of customization files. Worse, if you then need to change a customization that has been copied into many such files, you'll have to remember to change it in all of them. The idea from Method 1 can be extended to split the name into different parts and then use these parts seperately to customize a single aspect only. To keep this tidy, you can put all of these files into a directory localmk under control of Git and maybe even register it as a submodule in your local branch of the Org repository.


  LOCALMK ?= loc
  # split into words
  _LMK_:=$(subst -, ,$(LOCALMK))
  # what's the current trunk?
  _LMK_:=$(subst trunk,24.3.50,$(_LMK_))
  # specific Emacs version requested?
  _VER_:=$(filter 2%,$(_LMK_))
  ifdef _VER_
    _LMK_:=$(subst $(_VER_),loc,$(_LMK_))
    _MAJ_:=$(word 1,$(subst ., ,$(_VER_)))
    _VER_:=$(_VER_:2%=-2%)
  else
    _MAJ_:=24
  endif
  # lets us just specify which emacs to use
  ifeq ($(words $(_LMK_)),1)
    _LMK_+=testall tmp extra
  endif
  ifneq ($(origin DEBUG),undefined)
    $(info LMK[$(words $(_LMK_))]: $(_LMK_))
    $(info VER: $(_VER_) major $(_MAJ_))
  endif
  # remember this file is used from the org-mode directory via link,
  # so we have to prepend localmk/
  $(foreach part,$(subst -, ,$(_LMK_)),$(eval include localmk/$(part).mk))

So, what does this do? First, it splits LOCALMK on - characters. It then checks if something looks like an Emacs version number and tries to make sense of that, otherwise it's using a default version. If it only got an Emacs version to use, it tacks on a default set of standard customizations: testall, tmp, extra. Finally, it includes each of the customization files it inferred from LOCALMK to arrive at the final customization.


  # Name of your emacs binary
  EMACS   := /usr/local/bin/emacs$(_VER_)

  # Where local software is found
  prefix  := /usr/local/share

  # Where local lisp files go.
  lispdir := $(prefix)/emacs/site-lisp/org

  # Where local data files go.
  datadir := $(prefix)/emacs/etc/org

  # Where info files go.
  infodir := $(prefix)/info

  BTEST_OB_LANGUAGES = perl awk C fortran maxima lilypond octave perl python sh
  BTEST_POST = --eval '(add-to-list '"'"'load-path "~/.emacs.d/elpa/htmlize-1.39")'
  ifeq ($(_MAJ_),24)
    BTEST_EXTRA += ess-site
    BTEST_OB_LANGUAGES += R
    BTEST_POST += --eval '(add-to-list '"'"'load-path "/usr/share/emacs/site-lisp/emacs-ess-12.04.4")'
  else
    BTEST_PRE += --eval '(add-to-list '"'"'load-path "testing/ert")'
  endif

  # where to create temporary files for the testsuite
  TMPDIR ?= /tmp
  testdir = $(TMPDIR)/tmp-orgtest

  # extra targets
  .PHONY: testclean
  testclean:      test clean

  BTEST_OB_LANGUAGES = 
  BTEST_POST = --eval '(add-to-list '"'"'load-path "~/.emacs.d/elpa/htmlize-1.39")'
  ifneq ($(_MAJ_),24)
    BTEST_PRE += --eval '(add-to-list '"'"'load-path "testing/ert")'
  endif

Selective Testing

Sometimes you only want to run a set of specific tests instead of all. This is especially useful if you are trying to bisect a large range of commits with a run script. Instead of checking if the failing test was perhaps a different one than the one you wanted to check for, running just these test (or tests) makes that task much easier. For instance,

make BTEST_RE='^test-org/forward-element$' test-dirty

would run only the test for forward-element and nothing else, which is also much faster than running the almost 500 other tests as well. Keep in mind that the test selector is a regular expression, the default value of \(org\|ob\) matches all tests.

Support for Installers

The Org build system supports staged installs via DESTDIR. If =DESTDIR= is defined as a non-empty string (it really should be a leading path and end with a path separator), the actual installation paths are all prepended by the expansion of DESTDIR. Except for install and testing, the build does not write outside the build directory and both of these can be customized to stay within the build directory also.

Support for Manual Build

Since GNU make or some programs used by the build system might not be available on some systems, the core functionality has been implemented or replicated in Emacs Lisp with no dependencies on external tools. The supported functions are: org-make-autoloads, =org-make-autoloads-compile= and org-make-autoloads-compile-force. All other interfaces should be considered private and are subject to change without notice. The commands assume that the current working directory is at the toplevel of the Org build directory (i.e. where you'll find mk/default.mk).

To make the autoloads file:

emacs -batch -Q -L lisp -l ../mk/org-fixup -f org-make-autoloads

To make the autoloads file and byte-compile Org:

emacs -batch -Q -L lisp -l ../mk/org-fixup -f org-make-autoloads-compile

To make the autoloads file and byte-compile all of Org again:

emacs -batch -Q -L lisp -l ../mk/org-fixup -f org-make-autoloads-compile-force

If git is also unavailable, fake version strings need to be provided.

emacs -batch -Q -L lisp -l ../mk/org-fixup \
--eval '(let ((org-fake-release "8.0.1")(org-fake-git-version "8.0.1-fake"))\
(org-make-autoloads))'

The above assumes a POSIX shell for its quoting, Windows CMD.exe has quite different quoting rules and this won't work. Also, users of Aquamacs have reported that the command line examples aren't working for them. Your other option is to start Emacs like this

emacs -Q -L lisp -l ../mk/org-fixup

then paste the following into the *scratch* buffer:


  ; replace the version strings with something sensible that can't be
  ; confused with a real Git version
  (let ((org-fake-release     "8.0.1")
        (org-fake-git-version "8.0.1-fake"))
    (org-make-autoloads))

Execute each form by placing the cursor after the closing paren on the last line and press C-j or C-x C-e.

If the command line above is still spooking you: start Emacs like you normally do, then add <orgmode>/lisp to your load-path, then issue =M-x load-library=, answer the prompt with =../mk/org-fixup.el=, then do the same things in the scratch buffer as outlined above.

Footnotes

influenced via the configuration variable ORG_MAKE_DOC.

anything else, you can run make cleanelc compile-dirty.

with BTEST_OB_LANGUAGES.

may not happen when there are test errors. Also, the automatic removal of the test directory can be prevented (for debugging purposes) by defining a variable TEST_NO_AUTOCLEAN.

installation is, of course: it tries to remove Org from where it would install it, based on the current customization. So if you want to move your Org installation, you need to first uninstall it using the old customization, then change the costomization and then do a fresh install.


  1. The build systems' notion of "full testsuite" can be configured
  2. The build systems' notion of "all documentation" can be
  3. The build system doesn't really know where your previous
  4. The test directory is generally removed after testing, but this