1. # -*- mode:org -*-
  2. #+TITLE: Org-mode Testing
  3. #+PROPERTY: results silent
  4. * Dependencies
  5. The only dependency is [[][ERT]] the Emacs testing library which ships with
  6. Emacs24. If you are running an older version of Emacs and don't
  7. already have ERT installed it can be installed from its old [[][git
  8. repository]].
  9. * Non-interactive batch testing from the command line
  10. The simplest way to run the Org-mode test suite is from the command
  11. line with the following invocation. Note that the paths below are
  12. relative to the base of the Org-mode directory.
  13. Also note that many of the current tests uses babel evaluation...
  14. #+BEGIN_SRC sh :dir (expand-file-name "..")
  15. # For Emacs earlier than 24, add -L /path/to/ert
  16. emacs -Q --batch \
  17. -L lisp/ -L testing/ -L testing/lisp -l lisp/org.el \
  18. -l lisp/org-id.el -l testing/org-test.el \
  19. --eval "(progn (org-reload) (setq org-confirm-babel-evaluate nil) \
  20. (org-babel-do-load-languages 'org-babel-load-languages \
  21. '((emacs-lisp . t) (shell . t) (org . t))))" \
  22. -f org-test-run-batch-tests
  23. #+END_SRC
  24. The options in the above command are explained below.
  25. | -Q | ignores any personal configuration ensuring a vanilla Emacs instance is used |
  26. | --batch | runs Emacs in "batch" mode with no gui and termination after execution |
  27. | -l | loads Org-mode and the org mode test suite defined in testing/org-test.el |
  28. | --eval | reloads Org-mode and allows evaluation of code blocks by the tests |
  29. | -f | actually runs the tests using the `org-test-run-batch-tests' function |
  30. * Interactive testing from within Emacs
  31. To run the Org-mode test suite from a current Emacs instance simply
  32. load and run the test suite with the following commands.
  33. 1) First load the test suite.
  34. #+BEGIN_SRC emacs-lisp :var here=(buffer-file-name)
  35. (add-to-list 'load-path (file-name-directory here))
  36. (require 'org-test)
  37. #+END_SRC
  38. 2) Load required Babel languages
  39. #+BEGIN_SRC emacs-lisp
  40. (org-babel-do-load-languages
  41. 'org-babel-load-languages
  42. (and
  43. (mapc (lambda (lang) (add-to-list 'org-babel-load-languages (cons lang t)))
  44. '(emacs-lisp shell org))
  45. org-babel-load-languages))
  46. #+END_SRC
  47. 3) Then run the test suite. Babel evaluation confirmation is disabled
  48. and ~C-c C-c~ is enabled while running the tests.
  49. #+BEGIN_SRC emacs-lisp
  50. (let (org-babel-no-eval-on-ctrl-c-ctrl-c
  51. org-confirm-babel-evaluate)
  52. (org-test-run-all-tests))
  53. #+END_SRC
  54. When a test fails, run it interactively and investigate the problem
  55. in the ERT results buffer.
  56. How to run one test:
  57. Use this as a demo example of a failing test
  58. #+BEGIN_SRC emacs-lisp
  59. (ert-deftest test-org/org-link-escape-ascii-character-demo-of-fail ()
  60. (should (string= "%5B" ;; expecting %5B is right
  61. (org-link-escape "[")))
  62. (should (string= "%5C" ;; expecting %5C is wrong, %5D right
  63. (org-link-escape "]"))))
  64. #+END_SRC
  65. or evaluate the ert-deftest form of the test you want to run. Then
  66. "M-x ert RET test-org/org-link-escape-ascii-character-demo-of-fail RET"
  67. When not visible yet switch to the ERT results buffer named
  68. "\*ert\*". When a test failed the ERT results buffer shows the
  69. details of the first "should" that failed. See
  70. (info "(ert)Running Tests Interactively") on how to re-run, start
  71. the debugger etc.
  72. How to run all tests of a single test file:
  73. "M-x ert-delete-all-tests RET", confirm. Open the file
  74. ./lisp/test-*.el, "M-x eval-buffer RET", "M-x ert RET t RET"
  75. Consider to set pp-escape-newlines nil before running the test when
  76. looking at "should" in the ERT results buffer. Especially when
  77. using "l" to look at passed test results and possibly missing an
  78. appropriate setting of pp-escape-newlines made only temporarily for
  79. the running time of the test as e. g. tests using
  80. org-test-table-target-expect-tblfm do.
  81. * Troubleshooting
  82. - If the value of the =org-babel-no-eval-on-ctrl-c-ctrl-c= is non-nil
  83. then it will result in some test failure, as there are tests which
  84. rely on this behavior.