Browse Source

Initial testing documentation
* org-tests/
Installing ERT
Running Tests
Preliminary Test Utilities

Martyn Jago 9 years ago
1 changed files with 116 additions and 36 deletions
  1. 116 36

+ 116 - 36

@@ -15,18 +15,125 @@
 [[file:../][{Back to Worg's index}]]
-Org-mode has grown a big and feature rich application. Help is always needed and
-highly appreciated. If you're not a lisp programmer, you still might help to
-prevent regressions and greatly improve the stability of Org-mode.
+* Why test?
-This page is meant to provide help with this kind of helping. We hope to provide
-all the information needed here, and hope to create tests for non-programmers
+Org-mode has grown into a popular large and feature-rich application,
+on which development often happens at a great pace. Indeed, feature
+requests posted on the [[][Org mailing list]] are frequently developed and
+published within hours of the request. 
-Currently we are in a very early planning phase, so will find only what we
-_want_ to do. If you're not a worger yet, you may as well post ideas or code to
-the great [[][Org mailing list]].
+The down-side of such rapid development is that on some occasions, one
+of the many patches, features, or [[][refactorings]] being applied to the
+code-base may unintentionally cause unexpected behavior within
+Org-mode. This is known as a [[][software regression]]. The larger and more
+complex Org-mode becomes, the more you may expect to see such software
+One concern with introducing software regressions is that the
+regression may not be immediately obvious to users. Indeed it can be
+hours, days, or even months before an Org-mode user moves to export an
+important schedule, when "bang" - the regression has reared its ugly
+The solution to avoiding software regressions? [[][Regression Testing!]]
+With a regression test suite the developer can run the tests prior to
+committing their change, to ensure their additions do not break the
+regression tests.
+Of course a regression test suite is only as good as the number of
+regression tests within it, and this is where Org-mode users can help.
+* Running the current test suite
+So long as you are on an up-to-date release of Org-mode you may run
+the tests on your local setup. To update Org-mode check out the 
+[[][org FAQ page]]. Note, the test suite will run on versions of Emacs back
+to version 22.
+A pre-requisite for running the tests is the [[][ERT test library]]. This
+may be obtained easily as follows.
+** Emacs-22 / Emacs-23
+In Emacs-22 and Emacs-23 you will need to download the test files
+_ert.el_ and _ert-x.el_ to your testing directory. This may be
+accomplished with the following commands entered on the command line.
+cd /path/to/org-mode/testing
+curl -O
+curl -O
+Alternatively you may download the files within your browser.
+ - browse to [[][]]
+ - right click ert.el link and select _download linked file_ (or
+   equivalent) and save to org-mode/testing/ert.el
+ - right click ert-x.el link and select _download linked file_ (or
+   equivalent) and save to org-mode/testing/ert-x.el
+Thats it - you may now run the tests.
+** Emacs-24
+Emacs-24 includes ERT, so nothing to do.
+** Run tests from the command line
+To run the tests from the command line (without loading your personal
+configurations) use the following commands (Note, if you have more
+than one version of Emacs on your system, you may need to include the
+full path to the required Emacs executable).
+cd /path/to/org-mode/testing
+Emacs -Q -batch -l org-test.el -eval "(setq org-confirm-babel-evaluate nil)" -f org-test-run-batch-tests'
+** Run tests from within Emacs
+To run the tests from within Emacs itself.
+M-x load-file RET /path/to/org-mode/testing/org-test.el
+M-x org-test-run-all-tests
+* Can I write tests?
+Most certainly! Even if you're not a lisp programmer, you may still
+help prevent regressions and greatly improve the stability of
+Org-mode. In fact, thanks to the growing number of test-macros that
+are available you can write regression tests with very minimal lisp
+knowledge, and you are also likely to greatly improve your knowledge
+of lisp as a result. And the more tests you write, the more confident
+you may be that your personal use of the many Org-mode features
+available has not been affected by regression.
+And of course, any help toward improving the Org-mode test-suite is
+highly appreciated.
+* Preparing to write tests
+Coming soon...
+* Tutorial
+To come...
+* Testing Utilities
+Several useful utilities exist to make the practice of writing tests
+even easier. These are available from the Org-mode repository as a
+seperate git sub-module. Assuming you have cloned Org-mode using git,
+you may install these extra utilities with the following git commands.
+cd /path/to/org-mode
+git submodule init
+git submodule update
 * Aims
@@ -80,25 +187,6 @@ could copy and paste to do the following
     between them.  That way different people who  run the same test on different
     hosts can get the same result.
-** Testing tools
-Many of the above questions (how to we setup and tear down tests, and
-how can we make and check assertions) will depend heavily on the
-chosen testing tool.  Two options posited to the list so far are...
-- ERT :: (from ert is
-         a tool for automated testing, reporting results, as well as
-         debugging errors
-- EmacsLispExpectations :: (from
- is
-     a minimalist elisp testing framework based on 
-Both of these elisp files have been included in the =tools= directory
-inside this directory.
-I've never used EmacsLispExpectations, so I can't make any reasonable
-judgment between the two, perhaps if someone can implement some tests
-in both we could come to some sort of agreement.
 *** ERT Example
@@ -107,14 +195,6 @@ See [[file:ert-publish-test.el]] for the implementation.  To run the test
 open up org-publish-test.el, and =M-x eval-buffer=.  This should load
 ert, and run the simple html export test.
-* Clipboard
-Running a minimal emacs should suppress custom config files:
-: emacs -q -l yourtest.el
-Getting a list of all variables (incomplete):
-: grep -r defvar lisp/*.el
 * COMMENT buffer dictionary
  LocalWords:  ert el EmacsLispExpectations org's