invasion of special variables in Emacs Lisp

This post is part of Living with Emacs Lisp. This post is very long and you probably don’t need read sections after the - - - part.

1. Invasion of special variables

Invasion of special variables is something that might happen when the following two conditions are met

  • an emacs lisp file uses defvar, defcustom or defconst to declare a global special variable
  • another emacs lisp file happens to use a lexically scoped local variable of the same name as the global special variable from the other emacs lisp file.

What happens is that the behavior of the latter file’s code may become unreliable, although not always. It is as if the latter file is being invaded by the global special variable. The other way around is invasion of local variables.

This happens because of a property of defvar that Erann Gat described as “pervasiveness of defvar”.1 The code (defvar abc) declares the name abc special, which makes abc to be dynamically bound rather than lexically bound, in all emacs lisp files, including files which just happen to name a local variable as abc, even the files that are loaded before execution of (defvar abc), even the ones that are written by other teams, and sometimes even the ones that are byte-compiled.

For example, one day, Alice installs emacs-html-server.el for running an html server on Emacs. (packages mentioned in examples here are all made up, unless stated otherwise.) Let’s suppose emacs-html-server.el defines the command ehs-start-server which happens to use a local variable with name python-path. Then another day, Alice installs python-mode.el. Let’s suppose python-mode.el contains this line:

(defvar python-path "python" "Path to Python executible.")

After she installs it, she finds that the ehs-start-server command becomes broken as soon as she opens a python mode buffer.

Another example. One day, Alice installs print-to-pdf.el for printing buffers to pdf files. Let’s suppose print-to-pdf.el defines the command ptp-print-and-open which takes an optional parameter print-newline. Then another day, a new version of Emacs is released. Let’s suppose this version happens to introduce print-newline as a global special variable which specifies how the function print should behave. After she installs this version of Emacs, she finds that the command ptp-print-and-open is not working as before.

We will come to more detailed examples later to see precise conditions in which invasion causes serious collision. There are measures you can take to reduce collision. First, let me tell you a story about Bob. Bob was making something for dinner. The dinner was rice burgers with chicken, tomatoes, lettuce, etc. His cat was observing his cooking and found something funny about the way Bob was cooking. It was that Bob was using two cutting boards. He was using a yellow board for cutting raw chicken, and a green board for cutting tomatoes and lettuce. His cat asked “why two cutting boards?” He answered “My father told me that using two boards is a good habit to have because it reduces chance of cross contamination”2. Yellow board for (global) special variables. Green board for (lexically scoped) local variables.

2. yellow naming convention for special variables

Make sure that special variables have names with a hyphen in them. For example, if you use defvar to create a global variable in your emacs init file, you should name the variable with a common prefix (with a hyphen) such as my- or jh- if jh is your initials.

;; in emacs init file

(defvar my-abc nil "blah.")

(defvar my-xyz nil "blah.")

(defconst my-in-ms-windows (eq system-type 'windows-nt)
  "Non-nil if this is on MS Windows.")

If you are a package author, you would have a common prefix for names of all special variables and functions in your package. Even special variables for internal use should be named with that common prefix. For example, python.el chooses python- as its common prefix.

;; in python.el

(defvar python-mode-map ...)

(defcustom python-indent-offset ...)

(defvar python--timer ...) ; for internal use

3. green naming convention for (lexical) local variables

You can choose one of the following three possible rules, one of which you may choose to subscribe to:

Rule L1: In any emacs lisp file with lexical-binding set to t, all variables should be declared with hyphenless names, except when defining global special variables. For example, names such as pythonpath and python_path and path are OK, but names like python-path are not.

Rule L2: In any emacs lisp file with lexical-binding set to t, all variables mentioned within an anonymous function body (or a local function body if you use cl-flet), except for global special variables defined in that file or in other required files, should have hyphenless names. In other words, unlike Rule L1, you can declare a (lexical) local variable with names with hyphens, as long as you don’t use that variable within the body of an anonymous function.

Rule L3: In any emacs lisp file with lexical-binding set to t, all non-local variables mentioned within an anonymous function body or a local function body, except for global special variables, should have hyphenless names.

Rule L1 is the least permissive of names with hyphens, and Rule L3 the most permissive. Rule L1 requires the least amount of thinking, and Rule L3 the most.

Whichever rule you choose, don’t forget that function parameters are local variables too and therefore you must follow that rule when you are choosing names for function parameters.

There are some built-in special variables with hyphenless names, you should avoid those names as well.

(Update: Also, while any of these rules is enough to defend against invasion of special variables, none of them is enough to prevent invasion of local variables. See Invasion of local variables in Emacs Lisp. One of the approaches from that article can be named Rule L0.

Rule L0: In any emacs lisp file, all local variables should be declared with hyphenless names.)

If you are going with Rule L2 or L3, you should know that it is very easy to end up introducing anonymous functions, sometimes without you noticing it. For example, you are very likely to use anonymous functions if you use mapcar a lot. You might be using some fancy looping macro from some library which is implemented using closures, that is, the macro might be writing anonymous functions for you without you noticing. Macros such as with-process-shell-command macro from Nic Ferrier also make a closure from code you pass, for example.

4. what about casual global variables?

Many articles on Emacs Lisp use setq (rather than defvar) to introduce global variables with simple names like list, list1, x, y in short code examples. No hyphens there. For example, the following code is from an official article on marks and uses setq to create a global variable m.

(setq m (mark-marker))
(set-marker m 100)
(mark-marker)

The global variable m is a casual global variable: it is only being used for tutorial purposes, for testing out things, it is not being used as part of code for packages or your init file.

Global variables created by setq are not special variables and therefore do not cause invasion of special variables (true as of Emacs 24). This means that the yellow naming convention does not apply to globals from setq. But this behavior of setq is never explicitly mentioned in the manuals, as far as I know. So it is possible that future emacs may change the behavior of setq so that it creates special variables. In that case, you should start naming all global variables (whether created by setq or defvar) with hyphens in names.

5. - - -

6. does byte compilation eliminate collision?

Byte compilation does help reduce chance of special variable invasion, as it tends to remove mentions of local variable names, but one should still rely on variable naming conventions.

We cannot rely on byte compilation alone, because for example, even if the author of emacs-html-server.el only distributes byte compiled files, he still needs run emacs-html-server.el without byte compilation during his interactive development of the package. Also, when Alice installs python-mode.el from a package archive, and then opens a python mode buffer, and then installs emacs-html-server.el from a package archive and byte compiles it, she may end up with a byte compiled code for ehs-start-server which still mentions the name python-path.

Also, what if using defadvice or debugging something somehow uncompiles some compiled functions? Does that happen? I don’t know.

Another thing is, people rarely compile their emacs init files.

Yet another thing, I am not sure that byte compilation eliminating mentions of local variable names is an explicitly documented behavior.

7. detailed examples of collision

All emacs lisp file names or all emacs lisp package names in these examples are hypothetical unless stated otherwise.

7.1. passing an asynchronous callback function (which involves a nonlocal variable)

Suppose Alice is a user of example.el and some.el. Suppose example.el has a function that makes xhr requests and gets responses using the xhr-get function which is defined in xhr.el.

;;; example.el --- example stuff -*- lexical-binding: t -*-

(require 'xhr)

...

(defun example-something ()
  (dolist (some-query (list "x" "y" "z"))
    (xhr-get
     (format "http://www.example.com/%s" some-query)
     (lambda (response status)
       (message "%s => %s" some-query response)))))

...

Notice that some-query is the name of a variable which is a nonlocal variable to the anonymous callback function. This violates Rule L3.

And here is contents of some.el:

;;; some.el --- some stuff

...

(defvar some-query "query" "some.el query command")
(defvar some-version 1.2 "some.el version")

...

Notice that this file declares some-query special.

If example.el and some.el are loaded within same Emacs session, the function example-something will not work as intended because the name some-query is resolved by dynamic binding even within example.el and therefore won’t refer to intended values like “x”, “y” or “z” at the time the anonymous callback function is called, which is after the dolist form has finished executing.

7.2. passing an asynchronous callback code

This example is a slight modification of the previous example. Suppose that xhr.el provides a Lisp macro xhr-with-get which is just like xhr-get except it takes a callback code instead of taking a callback function and that the macro is implemented using xhr-get in the obvious way. Suppose example.el defines example-something-2 which uses that macro:

;;; example.el --- example stuff -*- lexical-binding: t -*-

(require 'xhr)

...

(defun example-something-2 ()
  (dolist (some-query (list "x" "y" "z"))
    (xhr-with-get (format "http://www.example.com/%s" some-query)
        (response status)
      (message "%s => %s" some-query response))))

...

That still sort of violates Rule L3.

If example.el and some.el are loaded within same Emacs session, the function example-something-2 will not work as intended.

xhr-with-get is written like this:

(defmacro xhr-with-get (url vars &rest body)
  "Note: use this macro only in lexical bound Emacs Lisp files"
  (declare (indent 2))
  `(xhr-get ,url (lambda ,vars ,@body)))

7.3. passing a synchronous callback function

Suppose Alice is a user of hello.el which defines the hello-insert-stuff command which in turn relies on the fp-repeat function defined in fp.el (which we suppose is a library providing lots of functions for functional programming written by Bob) and suppose fp-repeat is for repeatedly calling a function many times.

;;; hello.el --- hello stuff -*- lexical-binding: t; -*-

(require 'fp)

(defun hello-insert-stuff ()
  (interactive)
  ;; inserts "1111111111\n2222222222\n3333333333" to current buffer
  (dolist (i (list "1" "2" "3"))
    (fp-repeat 10
               (lambda ()
                 (insert i)))
    (insert "\n")))

...

Suppose fp-repeat is implemented like this:

;;; fp.el --- fp stuff -*- lexical-binding: t; -*-

...

(defun fp-repeat (n func)
  "Calls FUNC repeatedly, N times."
  (dotimes (i n)
    (funcall func)))

...

Now suppose Alice has a habit of using defvar with hyphenless names for her casual global variables, thereby violating yellow naming convention for special variables. Today Alice got curious about arithmetic operators and ran the following code in scratch buffer to see how they work with more than two arguments.

(defvar i 100)
(defvar j 200)
(defvar k 300)

(print (+ i j k))
(print (* i j k))
(print (- i j k))
(print (/ i j k))

Running that code has a nasty side effect of breaking the intended behavior of hello-insert-stuff. That is because it declares i special, and when Alice runs hello-insert-stuff later, it will insert3 “0123456789\n0123456789\n0123456789″ rather than the intended “1111111111\n2222222222\n3333333333″, because the dynamic binding of i established from within the fp-repeat calls will shadow other bindings.

7.4. passing a synchronous callback code

This example is a slight modification of the previous example. Suppose hello.el defines hello-insert-stuff-2 which is like hello-insert-stuff except it uses fp-repeat-do. Suppose fp-repeat-do is a Lisp macro defined in fp.el and is simply a macro version of fp-repeat.

(defun hello-insert-stuff-2 ()
  (interactive)
  ;; inserts "1111111111\n2222222222\n3333333333" to current buffer
  (dolist (i (list "1" "2" "3"))
    (fp-repeat-do 10
      (insert i))
    (insert "\n")))

We also suppose that fp-repeat-do is implemented using fp-repeat:

(defmacro fp-repeat-do (n &rest body)
  "Note: use this macro only in lexical bound Emacs Lisp files"
  (declare (indent 1))
  `(fp-repeat ,n
              (lambda () ,@body)))

Alice running her code for testing arithmetic operations has an effect of also breaking the intended behavior of hello-insert-stuff-2.

7.5. mid wrap-up

Someone deviating from naming conventions mentioned in this post causes collision which then breaks the intended behavior of someone else’s code that involves passing callbacks.

7.6. returning an anonymous function (which is a closure)

Suppose fp.el defines fp-counter which returns a sort of generator of numbers that start from a certain value and with a certain step. For example, (fp-counter 100 2) returns a function that returns 100, 102, 104, … on repeated calls.

(defun fp-counter (count-start count-step)
  (let ((current-number count-start))
    (lambda ()
      (prog1
          current-number
        (setq current-number (+ current-number count-step))))))

Notice that the author of fp.el violated Rule L3 twice in that code: current-number and count-step.

Now suppose Alice installs current.el which we suppose is an emacs lisp package for reporting the amount of electric current going through Emacs. What does that mean. I don’t know. Anyway, suppose current.el happens to include this line:

(defvar current-number nil
  "The amount of electric current in teaspoons.
This number is updated every `current-update-interval' seconds")

When Alice turns on reporting of electric current, everything that relies on fp-counter will break.

7.7. local function example

I cannot come up with a good example combining violation of naming conventions and use of local functions leading to trouble.

8. Common Lisp note

Common Lisp programmers use a different way of eliminating invasion: the earmuffs convention. They put asterisks around special variable names and those asterisks are called earmuffs. There is a guy4 who has not been using the earmuffs convention for years though. I don’t know about Common Lisp, but with Emacs Lisp, if you don’t stick to the naming conventions, your code will invade others code and vice versa.

Another difference. setq-ing on an undefined variable name at top level may do one of the following three things depending on Common Lisp implementations:

  1. error
  2. creates a lexical global, that is, declares a global variable (or in Lisp speak, sets the global value of the variable) without making it special (or in Common Lisp speak, without proclaiming it special).
  3. creates a global special variable.

Footnotes:

1

he used that phrase in The Idiot’s Guide to Special Variables and Lexical Closures

2

bacteria may move from raw chicken to the cutting board or knife and then from there to lettuce which goes to the rice burgers. see http://cooking.stackexchange.com/questions/2209/how-do-you-properly-clean-a-cutting-board-and-knife-to-prevent-cross-contaminati

3

actually not true. It will insert the character corresponding to ASCII code 0, then that of 1, and so on

4

Doug Hoyte who wrote Let Over Lambda

This entry was posted in Emacs, Lisp and tagged . Bookmark the permalink.

3 Responses to invasion of special variables in Emacs Lisp

  1. Pingback: Living with Emacs Lisp | Yoo Box

  2. Pingback: Differences between Common Lisp and Emacs Lisp | Yoo Box

  3. Pingback: lexical scoping and dynamic scoping in Emacs Lisp | Yoo Box

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s