README 5.4 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158
  1. # -*- mode:org -*-
  2. #+TITLE: Org mode Testing
  3. #+PROPERTY: header-args:emacs-lisp :results silent
  4. * Dependencies
  5. The only dependency is [[http://www.emacswiki.org/emacs/ErtTestLibrary][ERT]] the Emacs testing library which ships with
  6. Emacs24. If you are running an older version of Emacs and don't
  7. already have ERT installed it can be installed from its old [[https://github.com/ohler/ert][git
  8. repository]].
  9. * Non-interactive batch testing from the command line
  10. The simplest way to run the Org mode test suite is from the command
  11. line with the following invocation. Note that the paths below are
  12. relative to the base of the Org mode directory.
  13. Also note that many of the current tests uses babel evaluation...
  14. #+BEGIN_SRC sh :dir (expand-file-name "..")
  15. # For Emacs earlier than 24, add -L /path/to/ert
  16. emacs -Q --batch \
  17. -L lisp/ -L testing/ -L testing/lisp -l lisp/org.el \
  18. -l lisp/org-id.el -l testing/org-test.el \
  19. --eval "(progn (org-reload) (setq org-confirm-babel-evaluate nil) \
  20. (org-babel-do-load-languages 'org-babel-load-languages \
  21. '((emacs-lisp . t) (shell . t) (org . t))))" \
  22. -f org-test-run-batch-tests
  23. #+END_SRC
  24. The options in the above command are explained below.
  25. | -Q | ignores any personal configuration ensuring a vanilla Emacs instance is used |
  26. | --batch | runs Emacs in "batch" mode with no gui and termination after execution |
  27. | -l | loads Org mode and the Org mode test suite defined in testing/org-test.el |
  28. | --eval | reloads Org mode and allows evaluation of code blocks by the tests |
  29. | -f | actually runs the tests using the `org-test-run-batch-tests' function |
  30. * Trigger the tests with 'make'
  31. ** Recompile all
  32. Target ~test~ can be used to trigger a test run. The tests start
  33. after cleaning up and recompilation.
  34. #+BEGIN_SRC sh :dir (expand-file-name "..") :results silent
  35. make test
  36. #+END_SRC
  37. See ../mk/default.mk for details.
  38. ** Test dirty
  39. The 'dirty' targets are for recompiling without cleaning and
  40. rebuilding everything. This usually speeds up the recompilation
  41. considerably. Note that this speed up comes to the price of possibly
  42. weird errors due to the unclean build.
  43. The dirty target for testing is called ~test-dirty~.
  44. #+BEGIN_SRC sh :dir (expand-file-name "..") :results silent
  45. make test-dirty
  46. #+END_SRC
  47. ** Select tests by regexp
  48. Variable ~BTEST_RE~ can be set to limit the tests which are performed.
  49. ~BTEST_RE~ is interpreted as regexp.
  50. Example:
  51. #+begin_src shell
  52. make BTEST_RE='test-.*-inlinetask' test-dirty
  53. #+end_src
  54. yields
  55. #+begin_example
  56. ...
  57. selected tests: test-.*-inlinetask
  58. Running 2 tests (2017-12-28 15:04:45+0100)
  59. passed 1/2 test-org-export/handle-inlinetasks
  60. passed 2/2 test-org-inlinetask/goto-end
  61. Ran 2 tests, 2 results as expected (2017-12-28 15:04:45+0100)
  62. ...
  63. #+end_example
  64. * Interactive testing from within Emacs
  65. To run the Org mode test suite from a current Emacs instance simply
  66. load and run the test suite with the following commands.
  67. 1) First load the test suite.
  68. #+BEGIN_SRC emacs-lisp :var here=(buffer-file-name)
  69. (add-to-list 'load-path (file-name-directory here))
  70. (require 'org-test)
  71. #+END_SRC
  72. 2) Load required Babel languages
  73. #+BEGIN_SRC emacs-lisp
  74. (org-babel-do-load-languages
  75. 'org-babel-load-languages
  76. (and
  77. (mapc (lambda (lang) (add-to-list 'org-babel-load-languages (cons lang t)))
  78. '(emacs-lisp shell org))
  79. org-babel-load-languages))
  80. #+END_SRC
  81. 3) Then run the test suite. Babel evaluation confirmation is disabled
  82. and ~C-c C-c~ is enabled while running the tests.
  83. #+BEGIN_SRC emacs-lisp
  84. (let (org-babel-no-eval-on-ctrl-c-ctrl-c
  85. org-confirm-babel-evaluate)
  86. (org-test-run-all-tests))
  87. #+END_SRC
  88. When a test fails, run it interactively and investigate the problem
  89. in the ERT results buffer.
  90. To run one test: Use this as a demo example of a failing test
  91. #+BEGIN_SRC emacs-lisp
  92. (ert-deftest test-org/org-link-escape-ascii-character-demo-of-fail ()
  93. (should (string= "%5B" ; Expecting %5B is correct.
  94. (org-link-escape "[")))
  95. (should (string= "%5C" ; Expecting %5C is wrong, %5D correct.
  96. (org-link-escape "]"))))
  97. #+END_SRC
  98. or evaluate the ~ert-deftest form~ of the test you want to run.
  99. Then ~M-x ert RET
  100. test-org/org-link-escape-ascii-character-demo-of-fail RET~. When
  101. not visible yet switch to the ERT results buffer named ~*ert*~.
  102. When a test failed the ERT results buffer shows the details of the
  103. first ~should~ that failed. See ~(info "(ert)Running Tests
  104. Interactively")~ on how to re-run, start the debugger etc.
  105. To run several tests: ~M-x ert RET "<your regexp here>" RET~.
  106. To run all tests of a single test file: ~M-x ert-delete-all-tests
  107. RET~ and confirm. ~M-x load-file RET testing/lisp/<file>.el RET
  108. M-x ert RET t RET~.
  109. Consider to set
  110. #+BEGIN_SRC emacs-lisp
  111. (setq pp-escape-newlines nil)
  112. #+END_SRC
  113. before running the test when looking at ~should~ in the ERT results
  114. buffer. Especially when using ~l~ to look at passed test results
  115. and possibly missing an appropriate setting of ~pp-escape-newlines~
  116. made only temporarily for the running time of the test as
  117. e. g. tests using ~org-test-table-target-expect-tblfm~ do.
  118. * Troubleshooting
  119. - If the variable ~org-babel-no-eval-on-ctrl-c-ctrl-c~ is non-nil then
  120. it will result in some test failure, as there are tests which rely
  121. on this behavior.