`sumlist_2b`

as a function that would sum lists of length 0, 1, or more. (defun sumlist_2b (numbers) (apply plus 0 0 numbers))

Unfortunately `sumlist_2b`

cannot handle extremely long lists. In this posting, I will introduce `sumlist_6`

which does not suffer from this limitation.

This posting will not introduce any new SKILL++ primitives. Instead, it will use several primitives which have been introduced in the previous several postings to implement an algorithm which you may not have seen before. If any of these are unknown or mysterious to you, please consult previous postings of *SKILL for the Skilled* or the Virtuoso on-line documentation.

`apply`

used to call a function indirectly given the function object and a list of arguments. `sstatus`

modify a status variable of the SKILL virtual machine. `optimizeTailCall`

makes function calls in tail position be stack-space neutral. `labels`

defines local function functions which may be mutually recursive `unwindProtect`

defines a clean-up procedure which is executed even if an error is triggered. #### Very long lists

If you attempt to use `sumlist_2b`

on extremely long lists it will fail. You can generate a very long list as follows:

data = (list 1.1 -2.1 2.3 -.04) (for i 1 15 data = (append data data))

This produces a list of length 131072. If you attempt to sum this list with `sumlist_2b`

you get the following error.

*Error* plus: too many arguments (at most 65535 expected, 131074 given) - (0 0 1.1 -2.1 2.3 ... ) <<< Stack Trace >>> apply(plus 0 0 numbers) sumlist_2b(data)

The error message complains about a list of length `131074`

. This is because `data`

has length `131072`

, but `sumlist_2b`

prepends two additional `0`

's to the argument list. `131072 + 2 = 131074`

.

#### An esoteric issue?

You may very well consider this an esoteric issue. And you would be right. If you are in charge of generating the data, and you know the number of elements in your list is less than `16K`

, then there is no need to worry about this corner case. I suspect very few of the readers of this blog have ever, or will ever encounter such a situation. Even with that being the case, I discuss this here in the hopes that some of the techniques may prove useful, even if the exact problem never occurs.

Another reason to present a solution to this particular problem is that in so doing we can combine several other concepts which have been discussed in the previous *SKILL for the Skilled* blog postings.

#### The applyReduce function

The following implementation of `applyReduce`

takes advantage of `(sstatus optimizeTailCall t)`

and `unwindProtect`

as seen in a previous post of *SKILL for the Skilled*.

(defun applyReduce (fun args @key (maxArgs 8192) identity) (labels ((apply_head (fun args tail) (let ((save (cdr tail))) (setcdr tail nil) (unwindProtect (apply fun args) (setcdr tail save)))) (apply_reduce (args) (let ((tail (nthcdr (sub1 maxArgs) args))) (if (cdr tail) (apply_reduce (cons (apply_head fun args tail) (cdr tail))) (apply fun args))))) (cond ((cdr args);; if args has more the two elements(let ((save_optimizeTailCall (status optimizeTailCall))) (sstatus optimizeTailCall t) (unwindProtect (apply_reduce args) (sstatus optimizeTailCall save_optimizeTailCall)))) (args;; if args has exactly one element(car args)) (t;; if args is nilidentity))))

#### How does it work?

The `applyReduce`

function above calls the given function multiple times, but each time with a maximum of `maxArgs`

number of arguments. The `maxArgs`

parameter defaults to `8192`

but you may override it to something smaller, especially to understand better how the function works.

(trace plus) (applyReduce plus '(.1 .2 .3 .4 .5 .6 .7 .8 .9 .11 .12 .13 .14 .15 .16 .17 .18 .19) ?maxArgs 5) (untrace plus)

In the trace output you can see that the `plus`

function is called several times, but never with more than 5 arguments. After the initial call, the first argument to `plus`

is the return value of the previous call.

||||||||||||||(0.1 + 0.2 + 0.3 + 0.4 + 0.5) ||||||||||||||plus --> 1.5 ||||||||||||||||(1.5 + 0.6 + 0.7 + 0.8 + 0.9) ||||||||||||||||plus --> 4.5 ||||||||||||||||||(4.5 + 0.11 + 0.12 + 0.13 + 0.14) ||||||||||||||||||plus --> 5.0 ||||||||||||||||||||(5.0 + 0.15 + 0.16 + 0.17 + 0.18) ||||||||||||||||||||plus --> 5.66 ||||||||||||||||||(5.66 + 0.19) ||||||||||||||||||plus --> 5.85

Now we can use `applyReduce`

to create an efficient `sumlist_6`

function which is able to add up the elements of an arbitrarily long list. We can use `sumlist_6`

to add up the 131072 elements of the list `data`

created above.

(defun sumlist_6 (numbers) (applyReduce plus numbers ?identity 0))

Testing it, we can see that it works and returns the correct result. We need to test `sumlist_6`

for a very long list, as well as for the nil list and a singleton list.

(sumlist_6 data) --> 41287.68 (sumlist_6 nil) --> 0 (sumlist_6 '(42)) --> 42

#### What about the performance?

Recall the algorithm implemented as `sumlist_1a`

.

(defun sumlist_1a (numbers) (let ((sum 0)) (foreach number numbers sum = sum + number) sum))

On the Linux machine which I normally use, evaluating `(sumlist_1a data)`

takes approximately `0.02`

seconds, while evaluating `(sumlist_6 data)`

takes about `0.003`

. I.e., `sumlist_6`

is about 6 times faster for large lists.

#### Summary

The technique shown in `sumlist_2b`

is very simple, and requires very little code. Nevertheless, it has the caveat that it will fail for arbitrarily long lists. This usually does not matter, and because this corner case is relatively unlikely to occur, the technique is tempting to use, and in fact is a good solution if the data is under your control and you know the length of the list in question is not extremely large.

The technique shown in `sumlist_6`

is more correct, at the sacrifice of being more complicated code. That being said, because the complication of `sumlist_6`

is factored into the implementation of the function `applyReduce`

, the actual implementation of `sumlist_6`

is as simple as `sumlist_2b`

. I would suggest putting a copy of `applyReduce`

in your personal SKILL function library, with the appropriate customer prefix of course.

#### More to come

In upcoming posts we'll continue to survey the SKILL++ language using the example of summing a list.

Jim Newton

**Previous blog posts in this series**

SKILL for the Skilled: Many Ways to Sum a List (Part 5)

SKILL for the Skilled: Many Ways to Sum a List (Part 4)

SKILL for the Skilled: Many Ways to Sum a List (Part 3)