Nicolas Goaziou 9a8506b7af org-footnote: Fix footnote deletion 3 years ago
examples d262ae53c9 org-agenda: Fix agenda standard name when going unsticky 4 years ago
jump @ 820bb7d81b a9f3c9fe11 now using newer version of jump.el -- run $ git submodule update 10 years ago
lisp 9a8506b7af org-footnote: Fix footnote deletion 3 years ago
.gitignore 1b6de8fad5 ignore org-id file generated during testing 9 years ago
README 994e9ebf0b Update testing/README 4 years ago
org-batch-test-init.el 1b0fb1a484 testing: allow to select tests 7 years ago
org-test.el de24694f05 Turn org-mode into Org or Org mode 4 years ago


# -*- mode:org -*-
#+TITLE: Org-mode Testing
#+PROPERTY: results silent

* Dependencies

The only dependency is [[][ERT]] the Emacs testing library which ships with
Emacs24. If you are running an older version of Emacs and don't
already have ERT installed it can be installed from its old [[][git

* Non-interactive batch testing from the command line

The simplest way to run the Org-mode test suite is from the command
line with the following invocation. Note that the paths below are
relative to the base of the Org-mode directory.

Also note that many of the current tests uses babel evaluation...

#+BEGIN_SRC sh :dir (expand-file-name "..")
# For Emacs earlier than 24, add -L /path/to/ert
emacs -Q --batch \
-L lisp/ -L testing/ -L testing/lisp -l lisp/org.el \
-l lisp/org-id.el -l testing/org-test.el \
--eval "(progn (org-reload) (setq org-confirm-babel-evaluate nil) \
(org-babel-do-load-languages 'org-babel-load-languages \
'((emacs-lisp . t) (shell . t) (org . t))))" \
-f org-test-run-batch-tests

The options in the above command are explained below.

| -Q | ignores any personal configuration ensuring a vanilla Emacs instance is used |
| --batch | runs Emacs in "batch" mode with no gui and termination after execution |
| -l | loads Org-mode and the org mode test suite defined in testing/org-test.el |
| --eval | reloads Org-mode and allows evaluation of code blocks by the tests |
| -f | actually runs the tests using the `org-test-run-batch-tests' function |

* Trigger testing with 'make test'

Target ~test~ can be used to trigger a test run.

#+BEGIN_SRC sh :dir (expand-file-name "..")
make test

See ../mk/ for details.

* Interactive testing from within Emacs

To run the Org-mode test suite from a current Emacs instance simply
load and run the test suite with the following commands.

1) First load the test suite.
#+BEGIN_SRC emacs-lisp :var here=(buffer-file-name)
(add-to-list 'load-path (file-name-directory here))
(require 'org-test)

2) Load required Babel languages
#+BEGIN_SRC emacs-lisp
(mapc (lambda (lang) (add-to-list 'org-babel-load-languages (cons lang t)))
'(emacs-lisp shell org))

3) Then run the test suite. Babel evaluation confirmation is disabled
and ~C-c C-c~ is enabled while running the tests.
#+BEGIN_SRC emacs-lisp
(let (org-babel-no-eval-on-ctrl-c-ctrl-c

When a test fails, run it interactively and investigate the problem
in the ERT results buffer.

To run one test: Use this as a demo example of a failing test
#+BEGIN_SRC emacs-lisp
(ert-deftest test-org/org-link-escape-ascii-character-demo-of-fail ()
(should (string= "%5B" ; Expecting %5B is correct.
(org-link-escape "[")))
(should (string= "%5C" ; Expecting %5C is wrong, %5D correct.
(org-link-escape "]"))))
or evaluate the ~ert-deftest form~ of the test you want to run.
Then ~M-x ert RET
test-org/org-link-escape-ascii-character-demo-of-fail RET~. When
not visible yet switch to the ERT results buffer named ~*ert*~.
When a test failed the ERT results buffer shows the details of the
first ~should~ that failed. See ~(info "(ert)Running Tests
Interactively")~ on how to re-run, start the debugger etc.

To run several tests: ~M-x ert RET "" RET~.

To run all tests of a single test file: ~M-x ert-delete-all-tests
RET~ and confirm. ~M-x load-file RET testing/lisp/.el RET
M-x ert RET t RET~.

Consider to set
#+BEGIN_SRC emacs-lisp
(setq pp-escape-newlines nil)
before running the test when looking at ~should~ in the ERT results
buffer. Especially when using ~l~ to look at passed test results
and possibly missing an appropriate setting of ~pp-escape-newlines~
made only temporarily for the running time of the test as
e. g. tests using ~org-test-table-target-expect-tblfm~ do.

* Troubleshooting

- If the variable ~org-babel-no-eval-on-ctrl-c-ctrl-c~ is non-nil then
it will result in some test failure, as there are tests which rely
on this behavior.